UNICEF Child Protection Meta-Evaluation
Final 15 May 2008
Commissioned by
Child Protection Section, Programme Division
United Nations Children’s Fund, New York USA
3 UN Plaza, NY New York 10017
Focal Points
Ms. Shirin Nayernouri
Child Protection Specialist, Child Protection Section
Telephone: 212-326-7680
Facsimile: 212-735-4413
E-mail: [email protected]
Mr. Sam Bickel
Senior Advisor, Evaluation Office
Telephone: 212-326-7504
Facsimile: 212-824-6492
E-mail: [email protected]
Evaluator
Anne Sheeran, Ph.D., M.R.P.
Sheeran Consulting, Inc
4203 South Findlay Street
Seattle, WA 98118 USA
E-mail: [email protected]
Tel/Fax (206) 322 0817 PST
Sheeran Consulting, Inc.
Executive Summary
Background
Child Protection (CP) is one of five core UNICEF programming focus areas. Its mandate, to
protect children from violence, exploitation and abuse, directs CP to address issues including
violence against children, child labor, juvenile justice reform, trafficking, discrimination, support
to children deprived of parental care, and child protection in emergencies.
1. The long range aim of the UNICEF CP strategy is to facilitate systemic changes in national
and societal support for protective laws, polices and social conventions. The blueprint for
this is the “Protective Environment Framework” (PEF), a web of eight interconnected
programming areas that have steered CP investments since 2002.
Purpose/Objectives of the Meta-evaluation
2. The purpose of the meta-evaluation is to take the measure of how the PEF is understood
and addressed in the field by UNICEF and others. The primary objective is to provide UNICEF
CP with an authoritative evidence base for global lessons sharing with partners and for
global, regional, and national programming and advocacy efforts. An important subsidiary
objective is to explore knowledge management software options to help index evidence
based good practices.
Research Methods and Constraints
3. The meta-evaluation analyzes 59 CP evaluation reports that were produced between 2002
and 2007, or 40% of all CP evaluations in that period. Selection was based upon availability
and language (English). Scales were established to assign values to characteristics of interest
addressing programming quality, evidence quality, and PEF outputs and outcomes.
4. Each report’s catalogue of PEF investments was coded for delivery into the askSam
database, which allows text searching of materials from a variety of document types.
5. Limitations: in most cases, the depth of the database for identifying best practices is
constrained by (a) insufficient analysis of institutional contexts (e.g. donor requirements,
host government sensitivities), (b) scarce time series information (e.g. 64% of the reports
have no longitudinal data), and (c) an alarming over-reliance on Key Informant Interviews
with an under-emphasis on sampling and scale (e.g. 80% lack a sampling plan).
Findings – Protective Environment Framework and Cross-cutting Issues
6. Regarding the eight interconnected elements of the PEF:
a. UNICEF’s best work is at the national level, Strengthening Government
Commitments through national plans/policies and building a Legislative Framework.
b. Conversely, Monitoring and Oversight ranks as the most deficient area of the entire
PEF. Investments that align to centralized political power –e.g. relating to
enforcement of a national plan--are among the least likely to be monitored for
quality assurance.
c. The quality of programming addressing Access to Necessary and Child Friendly
Services is second only to “Strengthening Government Commitments.” In five cases,
investments in child friendly services have arguably laid the groundwork for
broader, systemic changes. Nevertheless, evaluators do not assess systemization
with any depth or consistency: only 15 reports (one-quarter of the meta-evaluation
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
ii
data universe) had relevant information on this topic and of these only 10 had a
reliable evidence base.
d. Building Capacities of Those Closest to the Child is the third most highly rated area
of the PEF for programming design and execution quality. UNICEF appears to be
skilled at dealing with professionals: capacity building programmes for professionals
were significantly better in quality over capacity building programs for families and
communities. And, strengthening of capacities of professionals is linked to
systematization and/or integration of CP programming. Further,
i. The meta-evaluation tracks 31% more activity in the “response” category
than in the “recognition” category
ii. Community involvement in program planning, design and implementation
appears to be a facilitating factor for recognition of and response to
protection needs by families and communities.
iii. Yet six studies suggest that over-reliance on volunteer workers impedes CP
recognition and response. And, 27% of the reports say that a lack of basic
needs (e.g. water, food) makes families, communities and governments less
willing and able to engage in CP issues.
iv. 20% of the reports do not touch upon systemization of capacity building;
29% do so but without a credible evidence base.
e. Neither Life Skills nor Child Participation are robustly represented in the reports that
were available for the meta-evaluation, however:
i. Outcomes are good for CP in at least four instances where service providers
had steady contacts with young people’s networks, which allowed for a sort
of rolling needs assessment.
ii. Life Skills investments were described in terms of activities or outputs in 17
reports; of these, 16 could not deliver reliable information.
f. Open Discussion/Civic Engagement is meagerly represented. Neither programmes
nor evaluations distinguish among knowledge, awareness, and advocacy.
g. Changing Attitudes, Customs, and Practices, another meagerly represented category
of investments, shows a validated, replicable, methodology that packages peer-topeer counseling, participatory monitoring, and training in rights awareness, with
community education.
7. Regarding the Cross-Cutting Issues and Emergencies:
a. Gender should have been a cross-cutting issue; seniority levels of CP partners are
un/under-reported; cost-analysis is virtually non-existent but a consistent theme is
“do-more-for-less-itus.”
b. CP in Emergencies shows good overall coherence though situation analyses are not
always asking the right questions.
8. A theme and gap analysis showed:
a. Substantial missed opportunities for strategic and programme communications in
support of CP investments; only six reports discussed media-related investments and
mostly at a descriptive level (e.g. activities or outputs).
b. A persistent concern in 71% of the reports that girls’ and boys’ different
opportunities, roles, and needs, are only superficially accounted for in the
programmes and in some cases missing altogether.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
iii
Conclusions
9. Leadership role: Technical assistance supports to government and civil society partners are
possibly the premier deliverable of UNICEF to CP globally. This follows from findings showing
that closest to central political power, UNICEF is making regular good use of its status as a
convening agency with outputs including situation analyses and coalition development.
10. Context analysis: Without information about upstream and institutional factors, such as
UNICEF global managerial priorities and donor commitments/expectations, the frame of
reference for interpreting programming decision making (e.g. regarding monitoring or
baselines) is narrow and un-illuminating.
11. Enforcement gap: While UNICEF finds and uses its “clout” most effectively with national
level government partners, the meta-evaluation finds consistent gaps in enforcement and
monitoring. These may reflect an undercurrent of instability in government commitments to
the CP agenda, but the meta-evaluation is not conclusive on this point. It raises the question
of whether enforcement weaknesses are basically the price of doing business on CP, or
whether UNICEF could better leverage international agreements on aid effectiveness (e.g.
Paris Declaration) in forging these agendas with host governments.
12. Internalizing the PEF: The central emphasis of the PEF, systematization of national and
societal protections, is not clearly articulated in the evaluation reports; three findings
indicate that programming is not systematic about systemization either: no evidence of
contingency planning to address the difficulties of obtaining baseline data, insufficient
analysis of pervasive inequalities such as gender, and ad hoc planning addressing linkages
between poverty alleviation and community level buy in.
13. Strategic communications: The absence of investments on this topic could reflect sampling
biases, or, alternatively, a weak emphasis on strategic communications within CP with a
corollary need for technical support in design, monitoring, and implementation.
14. Trainings and volunteerism: Findings concerning community engagement and capacities
development/training suggest that when these work well, they are powerful tools. A
consistent standard about how best to utilize volunteers, however, is missing, and it is
unclear why trainings –when they can be so powerful and additive—might ever be under
resourced or poorly orchestrated.
15. Knowledge Management: The askSam database has been indispensible; used synergistically
with Excel, askSam allowed for statistical and textual evidence to be contrasted side by side.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
iv
Acknowledgements
The author expresses appreciation to the meta-evaluation managers, Shirin Nayernouri (Child
Protection Section) and Sam Bickel (Evaluation Office) for technical vision and collegiality on this
project. Howard Dale (Knowledge Management) consulted with us on the incorporation of the
askSam database into the meta-evaluation. The author also wishes to thank Dnyanada Sharma
(Child Protection) for administrative support.
Participants in the one-day workshop on the meta-evaluation at UNICEF Global Headquarters
(April 2008) provided depth and interpretation to the conclusions and generated an initial set of
recommendations.
Tyler Blake Davis, Ph.C ., oversaw the statistical components of the meta-evaluation and the
askSam database development. Claire Jones, Ph.D., acted as a second reader and contributed
particularly to the capacities development contents of the report. Ratna Menon, M.A., assisted
in coding data for the askSam database and on the annotations of the Protective Environment
Framework Classification System. Lisa Maynard proofread the final document. dzMedia
supervised document layout and graphics. The author expresses great appreciation for all of
these contributions.
This study was commissioned by UNICEF Child Protection, New York, USA and remains the
property of UNICEF. The views expressed in this report are those of the author and do not
necessarily reflect those of UNICEF.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
v
Table of Contents
Executive Summary................................................................................................................ii
Acknowledgements................................................................................................................ v
List of Figures........................................................................................................................ ix
Findings Guide ...................................................................................................................... xi
Chapter 1:
Introduction ....................................................................................................1
Overview ...................................................................................................................................... 1
Tools and Methods ...................................................................................................................... 3
Limitations and Biases ................................................................................................................. 7
Profile of the Meta-evaluation Reports ....................................................................................... 9
Chapter 2:
Government Commitment to Fulfilling Protection Rights of Children ............... 18
Introduction and Overview ........................................................................................................ 18
National Plans and Policies ........................................................................................................ 20
Implementation ......................................................................................................................... 25
Chapter 3:
Legislation and Enforcement .......................................................................... 28
Introduction and Overview ........................................................................................................ 28
Legislation .................................................................................................................................. 29
Enforcement .............................................................................................................................. 32
Chapter 4:
Attitudes, Traditions, Customs, Behavior, and Practices .................................. 36
Introduction and Overview ........................................................................................................ 36
Chapter 5:
Open Discussion and Engagement .................................................................. 41
Introduction and Overview ........................................................................................................ 41
Increase in Public Knowledge About the Issue .......................................................................... 42
Open Discussion on Sensitive Issues.......................................................................................... 45
Use of Media for Advocacy Work .............................................................................................. 46
Formal Linkages to CRC Are Visible, Part of the Program ......................................................... 48
Children Feel Able to Speak and/or Act More Freely About Protection Concerns ................... 50
Chapter 6:
Children’s Life Skills, Knowledge, and Participation ......................................... 51
Introduction and Overview ........................................................................................................ 51
Insights into What Children and Young People Know and Need to Know ................................ 52
Relevant Information and Services Are Provided ...................................................................... 54
Children and Young People Get Involved .................................................................................. 58
Chapter 7:
Building Capacities of Those Closest to the Child ............................................. 62
Introduction and Overview ........................................................................................................ 62
Major Actors Are Facilitated in Recognizing Child Protection Needs ........................................ 64
Strengthened Capacities for Responding to Child Protection Needs ........................................ 68
Capacity Development Is Becoming Systematized and Integrated, not Ad Hoc ....................... 77
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
vi
Chapter 8:
Services Delivery ............................................................................................ 86
Introduction and Overview ........................................................................................................ 86
Provision of Necessary and Child-Friendly Services .................................................................. 88
Strengthened Social Welfare Systems ....................................................................................... 96
Chapter 9:
Monitoring and Oversight ............................................................................ 102
Introduction and Overview ...................................................................................................... 102
Monitoring Systems, CP Data Collection ................................................................................. 104
Participatory and Locally Based Mechanisms.......................................................................... 109
Reliable Reporting Systems ..................................................................................................... 110
Community Surveillance Systems ............................................................................................ 111
Chapter 10: Child Protection in Emergencies ................................................................... 113
Introduction and Overview ...................................................................................................... 113
Ability to Execute Rapid Assessments ..................................................................................... 114
Overall Management to Deliver a Holistic Response .............................................................. 116
Protection and Care of Separated/Unaccompanied Children and Children Associated with
Armed Groups.......................................................................................................................... 120
Psychosocial Support Interventions Are Targeted and Quickly Established ........................... 121
Prevention/Response Capacities for Gender-Based Violence Are Quickly in Place ................ 124
Gender-Differentiated Needs Are Addressed in Emergencies ................................................ 125
Chapter 11: Cross-Cutting Issues ..................................................................................... 128
Introduction and Overview ...................................................................................................... 128
Costs......................................................................................................................................... 129
Partnership Arrangements ...................................................................................................... 131
Child Protection Themes, Sectoral Good Practices ................................................................. 133
Scale ......................................................................................................................................... 136
Prevention or Response........................................................................................................... 138
Chapter 12: Conclusions ................................................................................................. 140
Overview .................................................................................................................................. 140
Conclusions .............................................................................................................................. 142
Appendix One: Terms of Reference .................................................................................... 150
Appendix Two: Protective Environment Classification System ............................................. 153
Introduction ............................................................................................................................. 153
Appendix Three: Evidence Quality Reporting Framework .................................................... 159
Introduction ............................................................................................................................. 159
Evidence Quality Findings ........................................................................................................ 161
Limitations and Biases ............................................................................................................. 162
Appendix Four: Programming Practices Reporting Framework ............................................ 164
Introduction ............................................................................................................................. 164
Programming Practice Findings ............................................................................................... 165
Limitations and Biases ............................................................................................................. 166
Appendix Five: Meta-Eval Reports/Evidence Ratings ........................................................... 167
Appendix Six: Frequency of Evidence, Protective Environment Elements ............................. 179
Appendix Seven: Thematic Overview, Meta-Evaluation Reports .......................................... 181
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
vii
Appendix Eight: How Large Are the Programs and the Evaluations? .................................... 185
Introduction ............................................................................................................................. 185
Target Population and Sampled Population ............................................................................ 185
Appendix Nine: Stakeholders’ Initial Recommendations ..................................................... 188
Introduction ............................................................................................................................. 188
Recommendations ................................................................................................................... 188
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
viii
List of Figures
Figure 1: Geographic Distribution of Meta-evaluation Reports .............................................. 10
Figure 2: Average Quality, Protective Environment Elements (Aggregates) ............................ 12
Figure 3: Programme Practices: Leaders and Laggards ........................................................... 13
Figure 4: Average Quality, Protective Environment Elements (Disaggregated) ..................... 157
Figure 5: Evidence Quality Reporting Matrix ....................................................................... 160
Figure 6: Evidence Quality, Average Per Variable ................................................................ 161
Figure 7: Programme Practices Reporting Framework ......................................................... 164
Figure 9: Frequency of Evidence, Protective Environment elements .................................... 180
Figure 10: Scale of Evaluations versus Scale of Programmes/Projects .................................. 187
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
ix
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
x
Findings Guide
Because the meta-evaluation tracked at least 64 distinct characteristics over a relatively small
number of reports (59), “findings” for any one of these characteristics can consist of a single
item in a single report. Critical mass, in other words, was not a determinant in applying the label
“findings.” As such, this guide could be equally appropriately titled “Guide to investment
strategies.” Further:

For the most part, findings that form the basis of the Conclusions chapter are those
which have the largest degree of recurrence, and

The applicability of findings to Child Protection thematic areas, for instance, juvenile
justice or trafficking, must be determined. The meta-evaluation focused on the
Protective Environment Framework in practice, not on thematic sub-sections per se.
Executive Summary................................................................................................................ii
List of Figures........................................................................................................................ ix
Findings Guide ...................................................................................................................... xi
Chapter 1:
Introduction ....................................................................................................1
1.1 Most/least frequent Protective Environment investments ............................................. 11
1.1.2 Average Quality, Protective Environment investments ................................................ 12
1.2 Programming Practices: Leaders and Laggards ................................................................... 12
1.2.1 Strategic planning ......................................................................................................... 13
1.2.2 Context-appropriateness .............................................................................................. 14
1.2.3 Community ownership .................................................................................................. 14
1.2.4 Facilitative role (UNICEF’s practical and moral persuasion) ......................................... 14
1.3 Theme and Gap Analysis ...................................................................................................... 15
1.3.1 de Facto gender bias ..................................................................................................... 15
1.3.2 Limited Communication for Development (C4D) investments ..................................... 16
1.3.3. Missing quality assurance accountabilities .................................................................. 16
1.3.4 Non-systematic sampling plans and small scale evaluations........................................ 16
1.3.5 Broad donor-political contexts are under-analyzed ..................................................... 17
Chapter 2:
Government Commitment to Fulfilling Protection Rights of Children ............... 18
2.1 Investment characteristics that strengthen government commitments ............................ 20
2.1.1 Technical assistance, coalition building ........................................................................ 20
2.1.2 Technical assistance, policy development .................................................................... 22
2.1.3 Technical assistance, pilot projects ............................................................................... 23
2.2 Constraints to strengthening government commitments ................................................... 25
2.2.1 Limited follow through mechanisms and accountabilities ........................................... 25
2.2.2 Lessons lost due to a weak evidence base .................................................................... 27
Chapter 3:
Legislation and Enforcement .......................................................................... 28
3.1 Investment characteristics that support legal reform ......................................................... 30
3.1.1 Technical supports that grasp the big picture .............................................................. 30
3.2 Constraints to legal reform .................................................................................................. 30
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
xi
3.2.1 Legislative efforts are not addressing accountabilities for enforcement ..................... 30
3.3 Investment characteristics that support enforcement ........................................................ 33
3.3.1 Community stakeholders build on the threat of enforcement..................................... 33
3.4 Constraints to enforcement................................................................................................. 34
3.4.1 Enforcement is largely at the level of moral persuasion .............................................. 34
Chapter 4:
Attitudes, Traditions, Customs, Behavior, and Practices .................................. 36
4.1 Investment characteristics that facilitate changed attitudes, practices.............................. 37
4.1.1 Intergenerational alliances, with peer counseling and health education..................... 37
4.2 Constraints to changing attitudes, practices ....................................................................... 38
4.2.1 Lessons lost due to a weak evidence base .................................................................... 38
Chapter 5:
Open Discussion and Engagement .................................................................. 41
5.1 Supports to increased public knowledge ............................................................................. 43
5.1.1 Leveraging already mature networks/administrative mechanisms ............................. 43
5.2 Constraints to expanding public knowledge........................................................................ 44
5.2.1 Investments in awareness-raising without service or policy follow up ........................ 44
5.3 Supports to facilitating open discussion .............................................................................. 45
5.3.1 Intergenerational alliances, with peer counseling and health education..................... 45
5.4 Constraints to facilitating open discussion .......................................................................... 45
5.4.1 Lessons lost due to a weak evidence base .................................................................... 45
5.5 Constraints to media/advocacy ........................................................................................... 46
5.5.1 Lessons lost due to a weak evidence base .................................................................... 46
5.6 Supports to facilitating rights awareness ............................................................................ 48
5.6.1 Using the CRC to a programming lingua franca ............................................................ 48
5.7 Constraints to facilitating rights awareness ........................................................................ 49
5.7.1 Investments are framed without reference to the CRC................................................ 49
5.7.2 Lessons lost due to a weak evidence base .................................................................... 50
Chapter 6:
Children’s Life Skills, Knowledge, and Participation ......................................... 51
6.1 Finding out what young people know and need to know ................................................... 52
6.1.1 Regular contact acts as a rolling needs assessment ..................................................... 53
6.2 Constraints to finding out what young people know and need to know ............................ 53
6.2.1 Top-down programming missed vital details................................................................ 53
6.2.2 Lessons lost due to a weak evidence base .................................................................... 54
6.3 Supports to delivering relevant life skills ............................................................................. 55
6.3.1 Regular contact acts as a rolling needs assessment ..................................................... 55
6.4 Constraints to delivering relevant life skills ......................................................................... 55
6.4.1 Life skills component is missing .................................................................................... 56
6.4.2 Training deficiencies constrain delivery ........................................................................ 56
6.4.3 No impact monitoring ................................................................................................... 56
6.4.4 Not taken to scale ......................................................................................................... 57
6.4.5 Lessons lost due to a weak evidence base .................................................................... 57
6.5 Investment characteristics that support child participation ............................................... 58
6.5.1 Close proximity of service provides and young people ................................................ 58
6.6 Constraints to child participation ........................................................................................ 59
6.6.2 Investments are not framed with respect to participation .......................................... 60
6.6.3 Lessons lost due to a weak evidence base .................................................................... 60
Chapter 7:
Building Capacities of Those Closest to the Child ............................................. 62
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
xii
7.1 Investment characteristics that facilitate recognition of child protection needs ............... 64
7.1.1 Reaching families via adolescent livelihoods interventions.......................................... 64
7.1.2 Reaching families/communities via involvement in programme planning .................. 65
7.1.3 Reaching communities via capacity building programming ......................................... 65
7.1.4 Reaching professionals via training and agent-partner linkages .................................. 65
7.1.5 Reaching all stakeholders via rights awareness training .............................................. 66
7.2 Constraints to greater recognition of child protection needs ............................................. 66
7.2.1 Abusive relatives not being engaged by community-based project ............................. 66
7.2.2 Communities/CSOs need more intense training........................................................... 66
7.2.3 Over-reliance on volunteers ......................................................................................... 66
7.2.4 Weak front-end needs assessments ............................................................................. 67
7.2.5 Lack of community involvement in programming ........................................................ 67
7.2.6 Professionals’ training is inadequate ........................................................................... 67
7.2.7 Poor coordination among institutions and agencies .................................................... 67
7.2.8 Lessons lost due to a weak evidence base .................................................................... 67
7.3 Investment characteristics that facilitate response capacities............................................ 69
7.3.1 Providing relevant supports (e.g., crèches) to families ................................................. 69
7.3.2 Rooting child protection efforts in the community ...................................................... 69
7.3.3 Community outreach makes a difference ..................................................................... 70
7.3.4 “Indigenized” models are effective ............................................................................... 70
7.3.5 Facilitating agency-community cooperation................................................................. 70
7.3.6 Training is a vital investment for building professionals’ response capacities ............. 70
7.3.7 Specialist capacity building has clear value................................................................... 71
7.3.8 Advocacy and mobilization supports professionals’ response capacities .................... 71
7.3.9 Giving the same trainings to all stakeholders: .............................................................. 72
7.4 Constraints to strengthening response capacities .............................................................. 72
7.4.1 For young people, unsupportive parents are a constraint ........................................... 72
7.4.2 Families’ economic needs are unmet ........................................................................... 72
7.4.3 For communities/CSOs, an over-reliance on unpaid volunteers .................................. 72
7.4.4 For communities/CSOs, an over-reliance on non-professionals................................... 73
7.4.5 For communities/CSOs, material resource constraints ................................................ 73
7.4.6 For communities/CSOs, inadequate training ................................................................ 73
7.4.7 For communities/CSOs, vertical communication breakdowns ..................................... 73
7.4.8 For communities/CSOs, failure to build public awareness ........................................... 74
7.4.9 Other constraints for communities/CSOs: unmet basic needs, ownership confusion . 74
7.4.10 For professionals, myriad training inadequacies ........................................................ 75
7.4.11 For professionals, too much staff turnover................................................................. 75
7.4.12 For professionals, lack of human resources and expertise ........................................ 76
7.4.13 For professionals, lack of material resources/transport ............................................ 76
7.4.14 For professionals, failure to integrate approaches .................................................... 76
7.4. 15 Lessons lost due to a weak evidence base ................................................................. 76
7.5 Investment characteristics that facilitate systematic capacity development ..................... 78
7.5.1 Integrating capacities development with awareness raising ........................................ 78
7.5.2. Professionals’ capacity building integrated into long term CP programming ............. 78
7.5.3 Establishing a cohort of trained child protection professionals ................................... 79
7.5.4 Developing local ownership .......................................................................................... 79
7.5.5 Effective cooperation and division of labor between institutions ................................ 80
7.6 Constraints to systematizing capacities development ........................................................ 81
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
xiii
7.6.1 Training is limited to a one-time event (“one off”) ....................................................... 81
7.6.2 No quality control on trainings ..................................................................................... 81
7.6.3 Trainees have no practical work experience................................................................. 81
7.6.4 Unmet needs and ownership confusion are not addressed ......................................... 82
7.6.5 Dependency syndrome ................................................................................................. 82
7.6.6 Confusion among partners............................................................................................ 83
7.6.7 Inadequate material support ........................................................................................ 83
7.6.8 Unclear project aims and strategies ............................................................................. 83
7.6.9 Need to grapple with conditions in conflict-affected areas.......................................... 84
7.6.10 Lessons lost due to a weak evidence base .................................................................. 84
Chapter 8:
Services Delivery ............................................................................................ 86
8.1 Investment characteristics that facilitate service delivery .................................................. 89
8.1.1 Specific services for specific demographic, with community outreach ........................ 89
8.2 Constraints to service delivery............................................................................................. 90
8.2.1 No substantive accountabilities .................................................................................... 90
8.2.2 Unassertive management ............................................................................................. 92
8.2.3 Misunderstanding programme contexts ...................................................................... 93
8.2.4 Inattention to girls’ needs, opportunities ..................................................................... 94
8.2.5 Overestimating volunteer capacities ............................................................................ 94
8.3 Investment characteristics that contribute to stronger social safety nets .......................... 97
8.3.1 Context-driven, with long-term outlooks ..................................................................... 97
8.4 Constraints to strengthening social safety nets .................................................................. 98
8.4.1 Skills training in a vacuum ............................................................................................. 98
8.4.2 Implementation pushed ahead of institutional capacity building ................................ 98
8.4.3 Small scale investments and limited time horizons ...................................................... 99
Chapter 9:
Monitoring and Oversight ............................................................................ 102
9.1 Investment characteristics that support systematic monitoring, data collection ............. 105
9.1.1 Management or (high level) government commitment ............................................. 105
9.2 Constraints to systematic monitoring, data collection ...................................................... 106
9.2.1 Call for an “off the shelf” system ................................................................................ 106
9.2.2 Management commitments (not capacities) are not evident .................................... 106
9.2.3 The “why” of monitoring—not just the “how”—needs reinforcing ........................... 107
9.2.4 Lessons lost due to a weak evidence base .................................................................. 108
9.3 Investment characteristics of functional participatory, locally based mechanisms .......... 109
9.3.1 Strong intergenerational alliance at the community level ......................................... 109
9.4. Constraints to participatory, locally based mechanisms .................................................. 109
9.4.1 Failing to recognize and make good use of obvious opportunities ............................ 109
9.5 Investment characteristics of reliable reporting systems.................................................. 110
9.5.1 High level government involvement ........................................................................... 110
9.6 Constraints to reliable reporting systems.......................................................................... 110
9.6.1 Programmes implemented without reporting plans .................................................. 110
9.6.2 Lessons lost due to a weak evidence base .................................................................. 111
9.7 Investment characteristics of working community surveillance systems ......................... 111
9.7.1 Strong intergenerational alliance at the community level ......................................... 111
9.8 Constraints to mobilizing community surveillance systems .............................................. 112
9.8.1 Lessons lost due to a weak evidence base .................................................................. 112
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
xiv
Chapter 10: Child Protection in Emergencies ................................................................... 113
10.1 Investment characteristics of good rapid assessment capacities.................................... 114
10.1.1 Situation analysis has to ask the right questions ...................................................... 114
10.2 Constraints to rapid assessment capacities ..................................................................... 115
10.2.1 Situation analysis ignored or missing ........................................................................ 115
10.3 Investment characteristics of well managed, holistic responses .................................... 117
10.3.1 Essential: partnership-building in non-emergency times ......................................... 117
10.4 Constraints to management response capacities............................................................ 118
10.4.1 Limited quality assurance mechanisms .................................................................... 118
10.4.2 Unresolved administrative, fiscal, coordination Issues............................................. 118
10.5 Supports to separated/unaccompanied children ............................................................ 120
10.5.1 Pre-planning for family based care ........................................................................... 120
10.6 Investment characteristics of targeted psychosocial interventions ................................ 121
10.6.1 Ability to mobilize quickly ......................................................................................... 121
10.6.2 Providing a safe haven .............................................................................................. 122
10.6.3 Training develops intergenerational cadre ............................................................... 122
10.7 Constraints to psychosocial interventions ....................................................................... 122
10.7.1 Inadvertently excluding out-of-school children ........................................................ 122
10.7.2 Haphazard and non-integrative service delivery ...................................................... 123
10.7.3 Confusion on service provision to adolescents ......................................................... 123
10.7.4 Inconsistent application of the psychosocial framework ......................................... 123
10.7.5 Missing needs assessment ........................................................................................ 123
10.7.6 Over-reliance on teachers ......................................................................................... 124
10.8 Investment characteristics of poorly gender-differentiated needs in emergencies ....... 125
10.8.1 Ignorance of basic socio-cultural definitions ............................................................ 125
10.8.2 Programmatic gender apartheid, no impact analyses .............................................. 125
10.8.3 Failure to advocate for girls’ dignity and equity ....................................................... 126
Chapter 11: Cross-Cutting Issues ..................................................................................... 128
11.1 Effective cost analyses ..................................................................................................... 129
11.1.1 Cost-benefit analysis is extremely limited ................................................................ 129
11.1.2 Two evaluation discuss connections between costs and sustainability ................... 129
11.2 Constraints to costs analyses ........................................................................................... 130
11.2.1 Rigid budget requirements constrain the program .................................................. 130
11.2.2 “Do-more-for-less-itus”............................................................................................. 130
11.2.3 Lessons lost due to a weak evidence base ................................................................ 130
11.3 Investment characteristics of partnerships arrangements.............................................. 132
11.3.1 Who: Civil Society Organizations and Government Ministries ................................. 132
11.3.2 What: Implementation and advocacy, not enforcement ......................................... 132
11.3.3 Where: Low to middle-level seniority ....................................................................... 132
11.4 Inter-sectorality blind spots ............................................................................................. 133
11.4.1 Child Protection needs to consider socio-economic contexts .................................. 133
11.4.2 Protection must address economic vulnerabilities ................................................... 134
11.4.3 Child Protection should increase synergies with other areas ................................... 135
11.4.4 Poverty negatively affects community participation ................................................ 135
Chapter 12: Conclusions ................................................................................................. 140
Programming and Evaluation Practices ................................................................................... 142
12.1.1 Crippling programming and evaluation practices are taken for granted ................. 142
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
xv
12.1.2 De facto operations assumptions may be limiting the ambition of Child Protection
evaluations ........................................................................................................................... 142
12.1.3 Improving analysis of institutional contexts is a necessity ....................................... 142
12.1.4 Need to re-examine quality control, leverage, at the at the sub-programme level . 142
12.1.5 Difficult to reconcile gender findings with corporate commitments ....................... 143
12.1.6 The askSam database has been an indispensible tracking tool ............................... 143
Strengthening Government Commitments ............................................................................. 143
12.2.1 How the convening organization convenes .............................................................. 143
12.2.2 Disparity in provision of technical information inputs .............................................. 143
12.2.3 Need to clarify the reasons for dis-investments in enforcement ............................. 144
Legislation and Enforcement ................................................................................................... 144
12.3.1 Evaluators do not address institutional contexts, re constraints to enforcement ... 144
12.3.2 A de facto “backdoor” enforcement strategy is in operation................................... 144
12.3.3 Lopsided supports to duty bearers versus rights holders ......................................... 144
Attitudes, Traditions, Customs, Behaviors, Practices .............................................................. 144
12.4.1 A wholly replicable methodology is not seeing much marketing ............................. 145
12.4.2 Models/theories of societal change may be missing? .............................................. 145
Open discussion/engagement with Child Protection issues.................................................... 145
12.5.1 Need to clarify practical/theoretical differences: “information” and “awareness” . 145
12.5.2 Rational and place for programme communications (“C4D”) are unclear ............... 145
12.5.3 Rights based approaches have an unclear status ..................................................... 145
Children’s Life Skills, Knowledge, and Participation ................................................................ 146
12.6.1 Rights based approaches are not a central emphasis .............................................. 146
12.6.2 Evaluating Life Skills investments has not been a priority ........................................ 146
Building capacities of those closest to the child ...................................................................... 146
12.7.1 Professionals get more professional inputs from UNICEF? ...................................... 146
12.7.2 There is no consistent standard for working with volunteers .................................. 146
12.7.3 The front end information needs of civil society partners/stakeholders are taken for
granted ................................................................................................................................. 147
12.7.4 Protective Environment programming theory is unclear about the status of building
capacities of families ............................................................................................................ 147
12.7.5 Evaluators do not address the systematization of capacities building ..................... 147
12.7.6 Best investment towards sustainability: training professionals ............................... 147
Delivery of necessary, child friendly, services ......................................................................... 147
12.8.1 Evaluation terms of reference: out of sync with the PEF.......................................... 147
12.8.2 Child Protection vastly understates its successes ..................................................... 148
Monitoring and Oversight........................................................................................................ 148
12.9.1 No explanation for monitoring findings, they are totally inexplicable .................... 148
Child Protection in Emergencies .............................................................................................. 148
12.10.1 Gender and Psychosocial findings merit further study ........................................... 148
Cross-cutting issues ................................................................................................................. 149
12.11.1 Lack of clarity on what makes a cross-cutting issue cross-cutting.......................... 149
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
xvi
Chapter 1:
Introduction
Overview
Meta-evaluation purpose and context
This meta-evaluation covers UNICEF Child Protection investments between 2002 and 2007. The purpose
is to develop “an authoritative basis for confirming effective protection interventions.” Child Protection
is one of the five UNICEF programme focus areas and represents 10.2% of the global programme
assistance expenditure.1
In 2002, in the context of the Mid-Term Strategic Plan, UNICEF introduced a new strategy for Child
Protection, the Protective Environment Framework (PEF). This defines eight elements that are critical to
good protection and that allow UNICEF to work toward systemic changes in
national and societal support for protective laws, policies, and social
OVERVIEW
conventions. The introduction of the PEF coincided with a management concern
The Protective Environment
to consolidate the evidence base for what works and what does not work. As
Framework defines eight
articulated in the Terms of Reference:
elements of child protection
All but one of the 157 UNICEF-supported countries, areas, and territories
address child protection issues programmatically. At the same time, not all
country offices work confidently in child protection, and the evidence base
remains small, including in verifying to what extent and how the elements of
the protective environment are understood and addressed in the field by
UNICEF and others. 2
that allow UNICEF to work
toward systemic changes in
national and societal support
for protective laws, policies,
and social conventions.
In developing an authoritative basis for confirming effective protection investments, the metaevaluation will (a) provide UNICEF Child Protection with the basis for global lessons sharing with
partners and for global, regional, and national programming and advocacy efforts, (b) support
management accountability for capturing lessons learned and examining the impact of UNICEF’s work in
child protection, and (c) inform any revisions of UNICEF’s Child Protection goals, especially those
contained in the Medium Term Strategic Plan 2006-2009.3
1
Source: 2006 UNICEF Annual Report. The other four programme areas and budgets are: Child Survival and Development
(51%), Basic Education and Gender Equality (21.3%), HIV/AIDS and Children (5.5%), and Policy Advocacy and Partnerships for
Children’s Rights (11%).
2
Terms of Reference may be found in Appendix One. In 2006, UNICEF cooperated with 155 countries, areas, and territories: 44
in sub-Saharan Africa (ESARO and WCARO); 35 in Latin America and the Caribbean (TACRO); 35 in Asia (EAPRO and ROSA); 20 in
the Middle East and North Africa (MENARO); and 21 in Central and Eastern Europe and the Commonwealth of Independent
States (CEE/CIS). The scope of child protection programmes and interventions vary according to the magnitude of protection
issues in these countries and the set priorities.
3
The Mid-Term Strategic Plan serves as UNICEF’s guiding framework over a four year period. It is developed through a
consultative process with Executive Board Members, United Nations agencies, non-governmental organizations, and UNICEF
staff. A copy of this document is available at this web address: http://www.unicef.org/childsurvival/files/05-11_MTSP.pdf
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
1
Objectives and scope
To address the overall purposes of the meta-evaluation, two basic objectives were set: to identify viable
evidence based programming strategies, and to examine programming drivers and constraints. An
important subsidiary objective was to explore knowledge-management software options to help index
evidence based good practices. Summarizing from the Terms of Reference:
1. “Determine which programming strategies and project interventions have been proven to work.
Proven to work means that the combination of good design, monitoring, and quality evaluation
allows UNICEF to state that certain approaches—if well executed—are likely to result in
measureable improvement in resisting violations of children’s rights to protection, and to
enhancing the resiliency and social networks that will foster strong child and community
development.”
2. Examine the evidence “against the goals contained in the Protective Environment Framework” to
allow Child Protection to contribute to UNICEF and the child protection community broadly on
matters such as management accountabilities, training priorities, knowledge management
strategies.4
3. Determine the value added of knowledge management software options to help index evidence
based good practices.
PRIMARY OBJECTIVE
Although the scope of the meta-evaluation is global, covering all regions and
in principle all countries, only English language reports were slated for
inclusion. Of 149 Child Protection evaluation reports that were completed
during the time period designated for the meta-evaluation, 2002 to 2007,
English language reports comprised 117 or 78% of the total.5
The meta-evaluation is based upon 59 evaluation reports. This is 50% of the
total in English and 40% of the total for all languages for the period under
consideration.
4
Determine programming
strategies that are proven to
work … through the
combination of good design,
monitoring, and quality
evaluation … in building the
Protective Environment for
children.
The terms of reference for this assignment specify that the meta-evaluation should allow Child Protection to contribute to
UNICEF and the child protection community broadly on matters such as management accountabilities, training priorities, and
knowledge management strategies (pp. 2–3 Terms of Reference). These are characterized as “Other direct and indirect
expected benefits” (p.3).
5
These figures are estimates provided by the Evaluation Office (January 2008). The time period for the meta-evaluation, 20022007, was selected in light of the coterminous introduction, during those years, of the Protective Environment Framework as a
new programming strategy.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
2
Tools and Methods
Reporting Framework
Overview: reporting tools
To realize the objectives described above, UNICEF Child Protection and UNICEF Evaluation Office
developed two reporting tools:


To report Child Protection activities relating to the eight components of the Protective
Environment Framework, emergency child protection, and five cross-cutting issues.
To report “programming evidence of a high quality” according to guidelines established by the
UNICEF Evaluation Office for this meta-evaluation.
Both tools contained numerous sub-categories. These provided the possibility
of tracking as many as 64 distinct variables across an expected 70 evaluation
reports.6
Neither tool was set up to allow for corporate self reflection per se. That is, the
tools could collect information about child protection activities and evidence
quality, but they could not collect information about elements of UNICEF’s
own programming practice such as programme design and execution.7 To fill
this gap, the evaluator built a third reporting tool:

TOOLS
The meta-analysis
catalogued Protective
Environment strategies and
cross-referenced them to
(a) evidence quality and
(b) programme design and
execution quality.
To report on programming practice as it may impinge upon or support protective environment
investments, such as strategic planning, knowledge management, stakeholder engagement, and
context sensitivity.
askSam database
The askSam database is a free-form database solution that allows text searching of materials from a
variety of source document types. The UNICEF Evaluation Office selected this database and steered its
incorporation into the meta-evaluation as an experiment (successful, in the opinion of the evaluator) in
knowledge management. Each report was coded according to the Protective Environment Classification
System and delivered into askSam.
Creating the meta-evaluation data set
This section provides further detail on the reporting tools and summarizes the methodology that united
them to produce a statistical dataset for the meta-evaluation.
6
The original terms of reference specified that 70 evaluation reports would form the basis of the meta-evaluation. Only 59
th
were identified as suitable in time to be included. This turned out to be more than a matter of expediency, however: by the 40
report the level of redundancy across the reports was clear. The additional reports up to 59 helped to confirm these trends.
7
As this report unfolds, readers will see that evaluation report authors linger on programming practices to the detriment of
evaluating outputs. A paucity of longitudinal data constrained the analysis of outcomes, and impacts.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
3
1. Protective Environment Classification System
This classification system contains nine primary categories—the eight components of the Protective
Environment Framework, plus Child Protection in Emergencies—and an additional five cross-cutting
categories that are “critical for successful Child Protection strategies”.8
The 14 categories each contained numerous sub-categories, 53 in all, and many contained numerous
sub-sub-categories. These various levels of differentiation describe strategies and outcome areas for
protective environment investments.
The Classification System in its entirety may be found in Appendix Two. The primary categories of the
Protective Environment Framework are:








Strengthening Government Commitments
Legislation and Enforcement
Attitudes, Traditions, Customs, and Behaviors
Open Discussion/Engagement on Child Protection Issues
Children’s Knowledge, Life Skills, and Participation
Building Capacities of Those Closest to the Child
Delivery of Necessary and Child Friendly Services
Monitoring and Oversight
Each category was filtered through any of five cross-cutting issues: Costs, Partnerships, Intersectorality,
Scale, and Prevention or Response. For example, when an investment in, say, delivering necessary and
child friendly services, worked well, what were the partnerships in play? As another example, which, if
any, of the successful Protective Environment categories seemed to map to larger or smaller scale of
investment?
2. Programme Practice Classification System
This system tracks eight elements of programming practice that, when in play in a Protective
Environment investment strategy, contribute to or constrain success. In other words, this is a system
that allows the meta-evaluation to probe more deeply into causation than is possible within the
framework of the Protective Environment Classification System. The Programme Practice Classification
system asks “why,” for instance:


Did a protective environment investment pay enough attention to community buy-in in
developing a community-based child protection network?
Did management drop the ball in putting a monitoring system into place or in facilitating
partners to ensure proper quality assurance and oversight?
Consulting Appendix Four, readers will see that the Programming Practice system is organized
alphanumerically with letter codes for the programming practice elements and numerics for how well or
how poorly they were put into play. The eight components are:
a.
8
Context (In practical terms, how well does the programme design fit the political, cultural,
and economic climate?)
Excerpted from the Meta-Evaluation Terms of Reference, p. 5.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
4
b.
c.
d.
e.
f.
g.
h.
Intersectorality (Reasonable coordination with other related needs?)
Programme communications
UNICEF’s use of its facilitative role
Achieved level of community ownership
Child participation
Strategic planning (How the work will get done, by whom, with what resources, etc.)
Knowledge management
How was the programming practice system developed?
During the inception phase of the meta-evaluation:

Step one
- The evaluator used the Protective Environment Classification System to conduct a content
analysis of the first 47 reports and was able to identify a recurring pattern of themes and issues,
which mostly pointed to programming practice as making or breaking Protective Environment
investments.

Step two
- On the basis of the recurring thematic patterns, the evaluator converted the Protective
Environment Classification System into a programming practice grid. This grid describes a
continuum of programming practices ranging from unacceptable to excellent.

Step three
- Based on this continuum, the evaluator then designed a simple numerical ratings scale to
address the appropriateness/effectiveness of the Protective Environment strategies discussed in
each report, so that if it appeared (on the subjective basis of the report) that a particular
strategy had worked well or was poorly executed or not sufficiently thought out, it would be
rated high or low accordingly. The system operated on a scale of zero (unacceptable) to three
(excellent).
How was this tool applied?
During the in-depth analysis phase, the evaluator:

Step four
- Returned to each report in depth, categorized investments according to their best fit within the
Protective Environment Classification System, filtered them back through the programming
practices grid, and assigned a quality rating. This process allowed the meta-evaluation to make
judgments about quality and effectiveness of investment strategies.
-
Meaningful evidence addressing any component of the Protective Environment Classification
System was assigned a code to correspond to the programme practice that was most obviously
in play. More than one letter code could be assigned as necessary.
Appendix Four provides more details about the programming practices classification system and where
the more highly rated investments were tending to cluster in relation to any of its eight elements.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
5
How does the programming practices system relate to the matter of evidence quality?
The evidence quality tool acted as a sort of “backseat driver” to the programming practice classification
and measuring system. That is, if a particular finding was unjustified or contradictory on the basis of the
evaluator's own evidence, it was ignored. However, the two tools pursued different lines of emphasis.
The evidence quality tool and the programming practices system proved to be mutually reinforcing:
good quality ratings on the Protective Environment strategies correlated well with good evidence
ratings, and vice versa. Moreover, a significant pattern in programming practices also emerged. Among
other things, this showed that poorly rated Protective Environment investments shared similar
weaknesses in programming practice—for example, neglecting the findings of a situation analysis or not
having had a situation analysis to begin with. Ultimately, the system helps to identify a failed or
successful investment while simultaneously shedding light on programming practices that were most or
least in play.
Further information on this system and many more findings than can be summarize in the main report
may be found in Appendix Four.
3. Meta-evaluation Evidence Quality Tool
This tool was designed by the UNICEF Evaluation Office to determine evidence quality. It uses seven
variables, each rated on a scale of zero to three (zero = unacceptable, three = excellent).







Results based management levels
- Whether the evaluator reports activities or goes forward to report outcomes or impacts
Use of comparison groups
- Selected at the time of evaluation or random assignment
Incorporation of longitudinal data
RESULT
- End evaluation only or baseline, middle, and end
Excellent evidence quality
Acknowledgement and reflection on international standards
indicators included: use of
- None or uses globally accepted standards
base-middle-end-line time
Analysis of intended and unintended consequences
series, random selection of
- None or considers intended and unintended consequences
comparison groups, and
Data analysis/Disaggregation
complex statistical analysis.
- None or use of compound or single variables
Statistical analysis
- None or complex analysis of survey data
Appendix Three provides details on the ratings averages for individual variables. It also comments
on areas of similarity and divergence between these meta-evaluation ratings and the ratings given
to UNICEF Evaluations generally for the period under scrutiny.
4. askSam database
The evaluation reports were coded for content that shed light on the various elements of the Protective
Environment Classification System. These reports were then imported into askSam. The coding followed
basic text searching protocols for askSam to allow flexible key word and alpha-numeric search queries
based on the reporting variables. The real strength of AskSam is the ability to examine actual text from
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
6
the 59 documents that pertain to the same element at the same time. Without the ability to use coded
documents the evaluative process would be reduced to more general comparison.
askSam reports were generated for each of the Protective Environment Framework characteristics and
according to the level of evidence quality (low, medium, high). In Excel, the quantitative assessments of
programming practices were captured as scores. These were accompanied by notations on the metaevaluator’s reasons, giving the project’s subjective evidence base complete transparency. While Excel
and AskSam were used for two different ends of the project, they were both structured by the
Protective Framework elements. As a result, the two data stores worked synergistically, with the
statistical and textual evidence contrasted side by side.
A “ReadMe” file (available on request from UNICEF Child Protection) guides first time users in searching
the database.
Limitations and Biases
The evaluation uses non-probability purposive sampling based on this criteria: materials were included
if they were available, if they were in English, and if they were completed between 2002 and 2007.9 This
meta-evaluation is comprised of English-language only evaluation reports. As explained earlier, this
excludes about 21% of evaluation reports (mostly in Spanish and French) for the period under
consideration.
This evaluator notes three limitations of the meta-evaluation process:
1. The meta-evaluation tracks about 64 unique characteristics of the
Protective Environment Framework (including Child Protection in
Emergencies and Cross-cutting Issues) across a relatively small number of
reports. As such, in some instances a "finding" for any one of these
characteristics consists of a single report. Put another way, there is enough
redundancy in some evaluation findings to allow a firm conclusion to be
drawn, for instance, with respect to monitoring or to the quality of certain
programming practices. But the analysis of findings is necessarily
constrained by the high level of itemization in the Protective Environment
Classification System.
LIMITATION
With some exceptions,
evaluation reports tend to
operate along the lines of
customer satisfaction
surveys, where the likeability
of the investment (from the
point of view of key
informants) stands in for
outcomes analyses.
In what sense, then, is the meta-evaluation a baseline for Child Protection? Inasmuch as the
fundamental requirement of a “baseline” is to show a specific performance measure, findings for
the programming practices and evidence quality meet this requirement. The meta-evaluation also
provides lessons about how the Protective Environment Framework operates in practice, for
instance, that technical work such as situation analyses cluster around governments more than
communities.
9
As explained in the Research Methods Knowledge Base http://www.socialresearchmethods.net/kb/sampnon.php , “With a
purposive sample, you are likely to get the opinions of your target population, but you are also likely to overweight subgroups
in your population that are more readily accessible.” Readers are reminded that the meta-evaluation tracks information on the
basis of the Protective Environment Framework categories, not child protection thematic categories.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
7
In terms of individual thematic areas of Child Protection, for instance, child labor or early marriage,
findings about what works and what is not working must be scrutinized for their applicability.
2. The level of detail addressing any topic in the Protective Environment Framework Classification
System varied greatly from one report to another. For example, one report may contain in-depth
information on partnerships, while another may have only touched up this in favor of focus on other
issues. This kind of disparity is a consequence of the meta-evaluation’s retrospective application of
the Protective Environment Framework, which for the most part had not yet begun to inform
evaluation reporting emphases for the time frame that was selected for the meta-evaluation. Yet,



While the reports are too varied in the level of detail to allow for much comparative depth,
and
While the actual universe of reports for any particular characteristic of the Protective
Framework Classification System may be as large as 20 or as small as two,
Common themes emerged and provide Child Protection with (1) insights into its
investments; (2) some evidence-based good practices; (3) a storehouse of information on
programming design and execution lessons learned (and a few lessons lost); and (4) a solid
basis to regularize Child Protection evaluation reporting, i.e., “Model Terms of Reference.”
2. The reports that were available for the meta-evaluation provide (a) insufficient analysis of
institutional contexts (e.g. donor financing limitations, host government sensitivities), (b) scarce
time series information (e.g. 64% of the reports have no longitudinal data), and (c) an alarming overreliance on Key Informant Interviews with an under-emphasis on sampling and scale (e.g. 80% lack a
sampling plan). With some important exceptions the evaluations tend to operate along the lines of
customer satisfaction surveys where the likeability of the investment too often stands in for impact.
This situation, together with the high level of itemization in the reporting framework (see Point One,
above), means that in many cases the data for identifying good practices and significant trends is
largely inferential.
The meta-evaluation compensates for Points One and Four by (a) indicating sample sizes for all themes,
and (b) presenting themes and issues in the context of verbatim extracts from the reports.
Limitations and biases of the three reporting matrices (Protective Environment Framework Classification
System, Programme Practice Classification System, and Evidence Ratings Tool) may be found in their
respective appendices.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
8
Profile of the Meta-evaluation Reports
Evidence Quality and Regional Draw
Ratings and rankings
In 2004, UNICEF implemented an evaluation quality ratings system that reflects and is complementary
to international evaluation standards. This system rates evaluations on 22 variables such as the clarity of
the description of stakeholder contributions, the sensitivity to evaluation ethics, concordance with rights
based approaches, transparency of methodology, and depth of interpretation of findings. All metaevaluation reports were initially assessed with this ratings system. A comparison between all UNICEF
evaluations versus those for the Child Protection Meta-Evaluation showed no significant differences:

Average UNICEF evaluation quality, 2000–2006, is 2.3410
o 58% of the reports received a “Satisfactory” rating; 26% rated “Poor”

Average Child Protection Meta-Evaluation quality rating is 2.43
o 63% of the reports rated as “Satisfactory”; 22% rated “Poor”
The Meta-evaluation Evidence Quality Ratings focuses exclusively on how the evidence base for each
report was generated. According to this tool:

The more highly rated reports utilized longitudinal data, comparison groups, and complex
statistical analysis. These reports were also consistently attentive on matters pertaining to
results based tools and methods. Approximately 41% of the reports (24) carried good ratings
and four were excellent.

The average overall quality was poor with nearly 40% scoring in that range (24); two were rated
as unacceptable.
o 64% of the reports have no longitudinal data; 80% had no sampling plan
Appendix Five lists all evidence quality ratings alongside each report’s title, country, and date.
Regional distribution of meta-evaluation reports
The following chart illustrates that the meta-evaluation’s regional draw is relatively equal for the four of
seven regions where UNICEF works: Central and Eastern Europe and the Commonwealth of Independent
States (CEE/CIS), East Asia and the Pacific (EAPRO), Eastern and Southern Africa (ESARO), and the Middle
10
http://www.unicef.org/evaldatabase/files/UNICEF_Evaluation_Report_Quality_Review_2006.pdf The 2006 review sample
was nearly five times the size of the Child Protection meta-evaluation sample (313 reports to 59). This probably accounts for
the small percentage difference in the quality rating averages between the two groups of evaluations. A detailed comparison
of Child Protection ratings for individual standards with ratings of the 2006 Quality Review would be expected to map to the
Review’s trend analysis which shows, among other findings, that the quality of Child Protection evaluations is increasing
annually.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
9
East and North Africa (MENA). It is quite low for the remaining three regions: Latin America and the
Caribbean (TACRO), West and Central Africa (WCARO), and South Asia (ROSA). This difference is largely
reflective of the meta-evaluation’s concentration on English language materials for TACRO and WCARO
and the relatively smaller number of countries in ROSA.
Figure 1: Geographic Distribution of Meta-evaluation Reports
Thematic Overview
The UNICEF Evaluation Database lists all reports by primary and secondary theme codes. For the reports
that were available for the meta-evaluation:

The largest two thematic categories are “Multi-thematic Child Protection evaluation” (18
reports) and “Armed conflict/Child soldiers” (10 reports).
o Of the “Multi-thematic” evaluation reports, the mean rating was slightly lower than the
overall mean for the meta-evaluation.11 In terms of evidence quality, the reports were
split evenly with half having evidence ratings of satisfactory or higher and half scoring
low to unacceptable.
11
The overall mean for the meta-evaluation for evidence quality was 1.45; for the “multi-thematic” category, the mean was
1.37.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
10

The next largest thematic categories are “Juvenile Justice” (7 reports) and “Sexual
exploitation/trafficking” (6 reports).

“Social protection; social safety nets; conditional cash transfers (CCT)” is a secondary theme in
17 reports. 14 of these come under the category “Multi-thematic Child Protection.”
 “Child rights legal and policy/administrative reform; Convention on the Rights of the Child (CRC)
CRC compliance; policy and law analysis” is a secondary theme in 10 reports.
 Gender based violence and sexual exploitation is a secondary theme in only five reports. In
terms of evidence quality, the lowest rated report of the metaMAJOR THEMES
evaluation is one of these.
Multi-thematic CP
 No report takes child participation as a secondary theme and there is
evaluations are the largest
no category in the coding classification system for Child Protection
primary category of the
reports to address.
reports (18); "Social
 The “programme communications/media analysis” category for Child
Protection" is the largest
Protection is not represented in the meta-evaluation. Undoubtedly
secondary thematic category
this reflects what was available for the meta-evaluation; however, a
(17); Programme
call for information on investments of this nature would help to clarify
Communications
the status of programme communications (“Communication for
(“Communication For
Development”, or C4D) in child protection investments to build a
Development”) has no
protective environment.
showing.
12
Appendix Seven lists all reports by primary and secondary themes.
1.1 Characteristics of the Protective Environment Investments
1.1 Most/least frequent Protective Environment investments
Figure 9, Appendix Six, shows the frequency of evidence that address characteristics of the Protective
Environment Framework and Emergencies in the 59 reports.13 The following points may be made:




Within the universe of this meta-evaluation, evidence addressing Open Discussion on Sensitive
Issues and Information on Sensitive Topics is limited.
The highest evidence areas are: Strengthening Capacities (of governments and
community/professional stakeholders) and Providing Child Friendly Services.
Monitoring and Oversight is also an area of very high evidence area; however, this reflects the
frequency with which evaluators flagged its absence or incompleteness.
Legislation-related activities are less well represented in this meta-evaluation than investments
dedicated to developing national plans of action and partnerships.
12
The Evaluation Office classification system for Child Protection has certain constraints. For example, while a very standard
Child Protection investment is the care of children who may have been orphaned by war, the taxonomy breaks this into two
separate categories: Armed Conflict, and Children without Caregivers. The meta-evaluation author has systematized these and
other classifications through a careful analysis of each report’s primary and secondary areas of emphasis.
13
Fig 9 does not include cross-cutting issues because, as shown in Chapter 11, “Cross-Cutting Issues,” the evidentiary scope for
these was minimal.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
11
1.1.2 Average Quality, Protective Environment investments
In this Average Quality Ratings Table (see Fig. 2), the horizontal axis shows the eight components of the
Protective Environment plus Emergencies. The vertical axis shows the average quality rating of the
programming practices (their appropriateness and effectiveness) for these nine components.
Again, these quality ratings are not measures of evidence quality: they are measures of the quality and
effectiveness of the investment strategies, as facilitated or constrained by programming practices.
Figure 2: Average Quality, Protective Environment Elements (Aggregates)
The following observations may be made regarding this chart and its disaggregated version in Appendix
Two (Figure 4):
 The highest ratings are attached to the central axis of the Protective Environment: Government
commitment to fulfilling protection rights of children. The disaggregated version of this chart
(Appendix Two, Fig. 4) shows that within this category, efforts to strengthen partnerships,
advocacy, and implementation are receiving the highest values for appropriateness and
effectiveness.
 The lowest quality ratings were seen in the area of monitoring and oversight. This is the weakest
link of Child Protection investments as represented in the meta-evaluation reports.
1.2 Programming Practices: Leaders and Laggards
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
12
The following chart (Fig. 3) shows the frequency with which Protective Environment investment
strategies pointed meaningfully to any of the eight characteristics of effective programming.
Figure 3: Programme Practices: Leaders and Laggards
*Vertical Axis = Number of occurrences (frequency) when the programming variable was flagged
as having significant evidence
*Horizontal Axis = Eight characteristics of effective programming
Frequency of evidence is shown on the vertical axis against eight programming practices on the
horizontal axis.
Appendix Four carries further detail on programming practices leaders and laggards. Anticipating indepth discussion of these characteristics in relation to Protective Environment elements as presented in
the main chapters of this report, four points can be flagged now:
1.2.1 Strategic planning
As the chart in Figure 3 makes clear, strategic planning has been flagged more than any other
programming practice. Strategic planning is defined as how the work will get done, by whom, and with
what resources. The density of evidence around this topic, which is four and one-half times higher than
that for child participation, conveys a great deal of information about the professionalization of
development practice. How is this so?
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
13


Strategic planning-related practices were flagged most often in relation to the weakest link in
the Protective Environment Framework: monitoring and oversight. Only 26% of the strategic
planning practices in this area had a satisfactory or higher quality rating.14
This is as opposed to good or better quality ratings for 62% of the strategic planning-related
practices for service delivery, 72% for building capacities of those closest to the child, and 75%
for child participation.
1.2.2 Context-appropriateness
Context-appropriateness, that is, the “fit” between programming design and concept and the
programming environment (e.g., political, cultural, ethnic, economic), was the second most populated
programming practice for the meta-evaluation, as seen in Figure 3. This is no surprise at all: with the
bulk of Child Protection investments focusing on building capacities of those closest to the child (within
the context of this meta-evaluation, at any rate), the topic of how well or poorly UNICEF understood the
environment of potential allies came up regularly.
Good or better ratings were given to investments that appear to:


Tap into relevant formal and informal decision-making networks.
Activate new, unusual partnerships.
About 58% of the time, the Protective Environment investment’s resonance with its context was rated
good or better.
1.2.3 Community ownership
Behind programme design and execution issues relating to strategic planning and contextappropriateness, community ownership was the third most populated programming practice area of
the meta-evaluation. If the investment helped to facilitate buy-in and/or was community-defined or
driven, it was rated good or better. If expectations were unrealistic—for example, evaluators found
volunteer fatigue that had not been addressed—this programming practice was rated poorly.
“Building capacities of those closest to the child” was the component of the Protective Environment
where practices relating to community ownership had the most evidence. About 45% of the times, this
programming practice was rated good or better.
On a related note, as stated earlier, the “Multi-thematic Child Protection” theme is the largest category
of Child Protection investments with an even split, nine for nine, in reports with satisfactory versus
unsatisfactory evidence ratings. The findings are not conclusive as to whether these –as opposed to
more narrowly defined—investments stand out in any way. Multi-themed programs are successful in
some instances, and must be evaluated on a case-by-case basis. The community-based programs are
most likely to succeed when there is maximum input from the community, i.e. ownership (stakeholders
and target population). Yet these are the same cases that caution against over-use of (community)
volunteers.
1.2.4 Facilitative role (UNICEF’s practical and moral persuasion)
14
All disaggregated values may be found in Fig. 8 (Appendix Four)
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
14
As shown in Figure 3, above, UNICEF’s use of its role as a convening organization (“facilitative role”) is
the fourth most populated programming practice in the meta-evaluation. Essentially, did UNICEF know
who to lobby and how to end programming bottlenecks, and proceed accordingly?
Significantly, when the answer is “yes” to the above question, it is usually with respect to UNICEF’s work
with government partners at the national level:



The closer to centralized political power, the more “clout” UNICEF appears to find and utilize (at
least in the universe of reports that were available for the meta-evaluation).
With good results: strengthening government commitments and Legislation/Enforcement had
good or better ratings in this area of programming practice in 68% and 64% of the instances,
respectively.
Alternatively, there are practically no instances of UNICEF using its comparative “clout” to press
for monitoring and oversight, for civic engagement with Child Protection issues, or with respect
to service delivery.
Further information may be found in Appendix Four, Programme Practices.
1.3 Theme and Gap Analysis
The Terms of Reference for this project call for a gap and thematic coverage analysis. Three topics carry
the weight here:



Is there a de facto orientation in programming and evaluation that gives gender analysis a
secondary emphasis?
GENDER BIAS?
Where are the effective instances of programme communications
Most (42 of 59) reports
(Communication For Development, C4D) supporting Protective
find gender analysis to
Environment efforts?
have been insufficient at
How does evaluation practice limit what’s known and knowable?
the level of program
Does it shed light on practices that could be specific to Child
design and
Protection?
implementation.
1.3.1 de Facto gender bias
Since the universe for this evaluation report addresses only the available reports that are in English, it is
not feasible to make an overall assessment of efforts to strengthen the protective environment for girls.
Within the scope of these evaluation reports, however, the situation appears worrisome:

Six of 59 reports carry gender as a secondary thematic emphasis. These come under four
general categories: Sexual Exploitation and Trafficking (2), Violence (2), Street Children (1), and
Harmful Traditional Practice (1). One of the smaller and worst rated reports addresses harmful
tradition practices.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
15

Supporting the impression of a secondary status for gender analysis, the meta-evaluation finds
that gender analysis is insufficient at the level of programme design including with respect to
emergencies.
1.3.2 Limited Communication for Development (C4D) investments
The overall place of advocacy and the media as tools in strengthening the protective environment
cannot be determined from the reports that were available for the meta-evaluation. Although this is
likely a robust area of investment, it is not well represented in this context.
In fact, no reports carry a secondary thematic emphasis addressing any aspect of programme
communications or advocacy. This is in spite of the fact that the biggest category of investments—
training and capacity building—would seem naturally to align with efforts to nurture a broad public
dialogue.
Indeed, the analysis shows that for capacity building there is a heavier relative degree of emphasis on
capacity building to respond rather than to recognize child protection needs. Where programme
communications were most frequently present (Developing Open
Discussion/Engagement), in 41% of the instances (39
EVIDENCE BASE
times) the ratings were satisfactory or higher.
The meta-evaluation
evidence base is largely
1.3.3. Missing quality assurance accountabilities
atemporal and ad hoc:
Monitoring capabilities as represented in these reports do not differ significantly
approximately 64% of
from the global norm in that they are processes in need of support. 90% of the
the reports have no
reports that were available for the meta-evaluation contained complaints about
longitudinal data and
monitoring having been insufficient or absent altogether. The meta-evaluation
80% lack sampling
finds this area of programming practice to be critically sub-standard.
plans.
The situation regarding baselines also reflects the global norm in that in the
majority of the reports, evaluators say that there were no baselines for the programmes being
evaluated. Yet, the meta-evaluation also notes a critical absence of dialogue by evaluators about the
institutional drivers of this situation, such as donor financing requirements. 15
1.3.4 Non-systematic sampling plans and small scale evaluations
The validity of evaluation evidence is typically measured in relation to the evaluation’s plan for surveying
a subset of the programme’s target population. This is the sampling plan: of the universe targeted for an
intervention, what portion of that universe can be tapped for research purposes to get a generally
accurate picture of what is going on?
In the reports for the meta-evaluation, evaluators did not provide guidance on which findings could be
generalized beyond the stakeholders that were consulted for the evaluations. For example:
15
The status of impact evaluation in global development aid has been addressed in a recent study by the Center for Global
Development Evaluation Gap Working Group, When Will We Ever Learn? Improving Lives through Impact Evaluation (May,
2006) http://www.cgdev.org/content/calendar/detail/7829/ . According to this report, “An “evaluation gap” has emerged
because governments, official donors, and other funders do not demand or produce enough impact evaluations . . .” (p.2).
The NONIE network (Network of Networks Impact Evaluation Initiative) also lists resources on the problems and issues relating
to impact evaluations and development aid.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
16


Only 50% of the reports reflect on the number of stakeholders consulted for the evaluation in
relation to the number of stakeholders targeted by the programme.
Of this subset, 20% have a stated criteria for which respondents were selected for the
evaluation (sampling plan).
The overall absence of stated criterion for how respondents were selected for the evaluations also
created difficulties in interpreting the scale of the evaluations in which Child Protection is investing:


Whereas in four (known) instances, the evaluators consulted with more than one thousand
stakeholders,
In as many as twenty (known) instances, the number of stakeholders consulted for the
evaluation is negligible. For example,
o 52 stakeholders consulted for a programme targeting 11,000 (Bosnia and Herzegovina
2003).
o 28 . . . for 1,733 (Zambia 2002).
o 109 . . . for 8,000 (Eritrea, 2004).
It is unnecessary, of course, to have a huge sample size to get relevant statistical data. A sample size of a
thousand can give high confidence over a population of several million (say 95% confidence level and +/5% confidence interval). The point here, however, is not about sampling plans or confidence intervals
per se, but to ask whether Child Protection evaluations are conceived and funded at too small a scale.
Further information on the matters of programme and evaluation scale may be found in Appendix Eight.
Fig. 10, Appendix Eight, is a simple bar graph showing the number of stakeholders that have been
targeted by a programme (when known) versus the number of stakeholders consulted for the evaluation
(when known).
1.3.5 Broad donor-political contexts are under-analyzed
Even in the stronger evidence-based reports, there is no analysis of donor
requirements as a potential factor in programming design and evaluation
emphases and timelines. There is no information, for instance, as to whether
donors pressed the case for baseline studies, or made ample funding and time
available to ensure that these were properly developed. There is likewise a
signal drop off in analyses of political considerations between UNICEF and host
governments that may constrain or otherwise shape programming and
evaluation. How is the balance of power between UNICEF and host governments
(or among UNICEF, governments, and donors) a factor in the disposition by Child
Protection to place a relative de-emphasis on the mechanics of enforcement?
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
17
EVIDENCE BASE
How might the balance
of power between
UNICEF and host
governments be a factor
in Child Protection’s deemphasis on
enforcement-related
investments? Evaluation
reporting does not
address this question.
Government Commitment
to Fulfilling Protection Rights of
Children
Chapter 2:
UNICEF tends to be chosen as a preferred partner by governmental counterparts. Such
credibility is based, among other things, on the organisation’s capacity of coordinating work
with numerous partners at the same time, as well as its proven ability to act as a bridge
between governments and civil society.
Evaluation of the UNICEF Impact on the Juvenile Justice System Reform (Serbia, Romania, Tajikistan 2006)
Introduction and Overview
Government commitment to fulfilling protection rights of children is the cornerstone of the Protective
Environment. Government interest in, recognition of, and commitment to child protection is an essential
element for a protective environment. This includes ensuring that adequate resources are made
available for child protection. It also includes political leaders being proactive in raising protection on
the agenda and acting as advocates for protection.
Within the universe of this meta-evaluation, the “Government Commitments” component of the
Protective Environment is not the largest direct category of investment; however, it is the most highly
rated for the quality of its programming practices.16 Evaluation report contents were analyzed on the
basis of two general sub-categories17:

National plans, policies, and budgets

Partnerships, advocacy, and implementation
Investments to strengthen partnerships, advocacy, and implementation were twice as common among
the reports used in this meta-evaluation as efforts addressing national plans, policies, and budgets per
se. However, the latter was 8.5% stronger in terms of the quality of its programming practices than the
former.
16
Reminder: Appendix Two contains disaggregated investment quality ratings for the Protective Environment Framework.
The askSam database was set up to track a third category, “Commitments emphasize prevention, response, or remediation;”
however, the level of detail to ensure accuracy on this point was not available in the majority of the reports.
17
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
18
Twenty-one reports contain evidence on this area of the Protective Environment. In the aggregate:

UNICEF advocates for child protection at the national level through three technical outputs:
o Bringing partners to the table.
o Providing a good situation analysis to sensitize partners to the issues.
o Investing in pilot projects.

UNICEF does not advocate for outputs to address:
o Results-based tools and methods.
o Technical details of implementation.
Essentially, efficacy flows from UNICEF’s use of its status as a convening organization. On matters with
high visibility—HIV and AIDS, juvenile justice reform, trafficking—reports show that UNICEF Child
Protection is strong in facilitating policy development. This indeed was described as an area of “best
investment” for the cause of juvenile justice reform in Serbia, Montenegro, Romania, and Tajikistan.
Consistent with this strong facilitative role, the mine risk education programme evaluation found that
UNICEF’s comparative strengths were more salient at the level of advocacy, not implementation.
The Programming Practice Leaders and Laggards table (Fig. 3, previous chapter) shows that UNICEF’s use
of its facilitative role was the fourth most common element of programming practice to come out in the
meta-evaluation reports, running behind strategic planning, community ownership, and context
relevance. However, with the exception of the Protective Environment component addressing building
capacities of those closest to the child, use of UNICEF’s leveraging capacities was almost ten times more
dominant in “Government Commitments” and “Legislation” than in any other area of the Protective
Environment.
What works — supports to strengthening government commitments

Technical assistance in (a) establishing inter-ministerial bodies, (b) policy development, and (c)
developing experimental or pilot projects.
What is not working — constraints to strengthening government commitments
 De-emphasis on monitoring and oversight, weaknesses in the area of sub-programme
coordination, and a tendency to underplay follow through.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
19
National Plans and Policies
2.1 Investment characteristics that strengthen government commitments
’best investments’…: Advocacy and support for the establishment of inter-ministerial,
inter-sectoral bodies, which have been established in Serbia, Montenegro, and Tajikistan
and provide UNICEF with opportunities for a dialogue with all its partners in juvenile justice
reform. (emphasis in original)
Evaluation of the UNICEF Impact on the Juvenile Justice System Reform (Serbia, Romania, Tajikistan) (2006)
As this epigram suggests, a “best investment” for building national political will is establishing interministerial bodies, which provide stakeholders with an otherwise non-existent context for dialogue and
planning. Within the universe of materials that was available for the meta-evaluation, this is one of
three areas of Child Protection practice that show efficacy. The other two are: Policy Development, and
Investments in Pilot Projects.
2.1.1 Technical assistance, coalition building
As indicated already, the meta-evaluation finds efficacy where the Child
Protection focus was on mobilizing partners’ political will through strategies
including building national plans of action and ministerial-level monitoring
bodies (seven reports; first six with good evidence base ratings).
FINDINGS
Activities like coalition
building show good outcomes
such as the formation of
sustainable inter-ministerial
working groups and
programme monitoring
groups (six reports).

Evaluation of the Street Children and Street Mother, Partnership
Programme (Ethiopia 2004)
Creating the national political will:
o Of utmost importance in terms of contribution has been the huge strides made in
consciountization of the plight of street children and mothers among the different strata
of Ethiopian society. Although it has yet to reach the critical mass whereby it translates
into the enactment of policies and the allocation of public budget, the awareness
created has been instrumental in drawing national attention to the phenomenon. The
government has formulated a national action plan and the media has begun publicizing
the plight of street children and mothers.18

Evaluation of the Protecting Street Girls and Ending Child Migration Programme (Ghana 2003)
Identifying significant decision makers from the top on down to the community level:
o The RPP [Rights Protection and Promotion] programme also has a strong advocacy
component at the national level. ‘Operation End Child Kayayee’ was a campaign
initiated at the beginning of 2002 and engages advocates from among Ministers of
State, Parliamentarians, traditional rulers, and opinion leaders at community level to
fight child migration. Beneficiaries themselves are also actively engaged in the planning
and implementation of the programme and managers pay special attention to issues of
18
Unless otherwise noted in the main text, all narratives attached to the evaluation reports are direct excerpts. This holds for
all content in all chapters of the meta-evaluation.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
20
sustainability in building linkages and synergies between the government and NGO
sectors.

Evaluation of the Mahzel Project for Social Reintegration and Protection of Disadvantaged
Children (Eritrea 2004)
Bringing partners to the table:
o The Programme Review and Monitoring Group (PROREMO) meetings were a managerial
tool to periodically allow all partners to monitor progress in execution and
implementation and to regulate project implementation. UNICEF played a crucial role in
the overall coordination of the project through PROREMO meetings.

Evaluation of UNICEF’s Support to Mine Action (Global 2005)
Creating a framework or context:
o What UNICEF has done better is create platforms of dialogue, raise standards, and
engage others in an inclusive collaborative forum at an international level.

Evaluation of the Psychosocial Care for Children Traumatized by Terrorist Violence Programme
(Algeria 2003)
Bringing partners to the table:
o In response to the needs, UNICEF has launched a project
for psychosocial rehabilitation of children traumatized by
terrorist violence. The project has developed through
several stages from an initial response to the crises to a
partnership in development with public sectors and
non-governmental organization.
"The Programme Review and
Monitoring Group (PROREMO)
meetings were a managerial
tool to periodically allow all
partners to monitor progress in
execution and implementation
and to regulate project
implementation. UNICEF played
a crucial role in the overall
coordination of the project
through PROREMO meetings."

Evaluation of the UNICEF Impact on the Juvenile Justice System
Reform (Serbia, Romania, Tajikistan) (2006)
Bringing partners to the table:
o …’best investments’…: Advocacy and support for the
establishment of inter-ministerial, inter-sectoral bodies,
which have been established in Serbia, Montenegro and
Tajikistan and provide UNICEF with opportunities for a
Eritrea 2004
dialogue with all its partners in juvenile justice reform.
(emphasis in original)
EXCERPT
o UNICEF tends to be chosen as a preferred partner by
governmental counterparts. Such credibility is based, among other things, on the
organisation’s capacity of coordinating work with numerous partners at the same time,
as well as its proven ability to act as a bridge between governments and civil society.

Mid-Term Evaluation of the “Children’s Chance for Change” Project, a Juvenile Justice Initiative
in Serbia and Montenegro (2006) — This evaluation is not highly rated for evidence quality, but
it shows at least one evaluator’s sense of the magnitude of UNICEF’s contributions:
 … the project attracted attention of professional and wider community . . . The
most reputable experts and social and political figures of GO and NGO sector were
included in the project activities through the work of Juvenile Justice Committee of
the Ministry of Justice of the Government of Republic of Serbia.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
21
2.1.2 Technical assistance, policy development
Seven reports describe UNICEF’s technical supports to national-level policy development. The evidence
quality is viable in only two and these offer slightly differing findings on the matter of technical
assistance.


Education as a Preventive Strategy Against Child Labour (Global 2003)
— UNICEF was found to be technically deficient, but still effective in
terms of advocacy:
o Policy advice and advocacy require UNICEF to have a more
comprehensive understanding of and play an active role in
existing and planned policies and strategies of governments and
other partners.
o In general, the Programme was considered relevant by
participating countries in a variety of ways: activating their
political commitment to the CRC; increasing awareness and
understanding of child labour in relation to education …
FINDINGS
Technical assistance for
policy development was
found to be insufficient for
Child Labor; for Juvenile
Justice Reform, UNICEF
showed technical vision and
leadership.
Because of poor evidence
quality there are more
lessons lost here than lessons
learned.
Evaluation of the UNICEF Impact on the Juvenile Justice System
Reform (Serbia, Romania, Tajikistan) (2006) — Here, technical
proficiency and vision projected UNICEF into a leadership role:
o On the whole [regarding Tajikistan], it would seem that the need for a comprehensive
approach was not being met by any other international actor, except UNICEF.
The following five reports have a less certain evidence base. Together with the two cited just now, these
reports show UNICEF working on the ground floor, so to speak, providing technical assistance in areas
that are new and unfamiliar. This is a feature of UNICEF’s work as a convening organization that is well
leveraged in the service of Child Protection at the national level.19

Mid Term Review UNICEF Romania Report on Child Protection Programme (Romania 2002)
Good credibility in defining the course of action:
o Support for Child Protection Reform: … This sub-project is possibly one of the most
important parts of the UNICEF Child Protection Programme. There are other actors
providing technical assistance and training but it is difficult to think of another actor
with the authority concerning policy development that UNICEF has. For the following
years the policy part of the project will be of high relevance when the revised legislative
framework shall be implemented.

Evaluation of UNICEF Support to Juvenile Justice System (Yemen 2004)
A responsible partnership:
o The main purpose of this programme is to assist the Government in developing a
comprehensive national polices and strategies on all child protection issues …

Evaluation of UNICEF Project on Preventing Trafficking of Women and Children, Gender-Based
Violence, and HIV/AIDS in Bosnia and Herzegovina (2003)
19
Two additional reports imply that UNICEF has been a central “mover and shaker” in supporting the developing of national
plans and policies, but the information is not enough to go on:
Institution Building & Mainstreaming Child Protection in Indonesia UNICEF supported Child Protection Bodies (LPAs),
(Indonesia 2003)
Preliminary Assessment of the Use of Family and Community Conference Group as an Alternative Approach in Juvenile
Justice, Evaluation Report (Thailand 2007)
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
22
Leadership in brokering inter-sectoral coordination:
o UNICEF BiH has been working to support the Ministry of
Human Rights and Refugees in the development and
implementation of their National Plan of Action on Trafficking
… UNICEF was a bridge from government organisations to
NGOs and now the door was open to mutual understanding
and a different relationship based on a multi-sectoral
approach. UNICEF ‘s role had also opened the possibility of
cooperation with other international agencies…
FINDINGS
Because of poor evidence
quality there are more
lessons lost regarding
technical assistance for policy
development than lessons
learned.

Lessons Learned Review on the Law Enforcement Against Sexual Exploitation and Trafficking of
Children Project (Cambodia 2006) — A partner agency (World Vision) steered this effort:
o LEASETC has very effectively harnessed high level commitment and support, ensuring
that LEASETC’s efforts are regarded as priority activities in the MoI. Policy development
at the national level has been slow. However, this was to be expected with such a new
concept, the benefits of which needed to be demonstrated through the successful
implementation of the pilot projects.

Evaluation of the Deinstitutionalisation Process in the Republic of Tajikistan (2005) — This is an
example of evaluation findings that are difficult to harmonize:
o On the macro level, this project has been a catalyst for change. The development of the
new Children’s Rights Departments has sparked the beginnings of a national debate on
the need for replication of these departments across the country and the need for
systemic reform of the child protection system both at local and national level.
o The lack of a clear direction by the State has resulted in a similar lack of an integrated
response from the international donor community …
2.1.3 Technical assistance, pilot projects
Four reports describe foundational, preliminary, or experimental investments, including pilot projects
and situation analyses, which have helped to define or elaborate national policies. The first three have
good evidence quality ratings.

FINDINGS
An Assessment of the Impact of Implementing the MVC Programme
Using pilot projects to
and the Operation of Funds and the Potential for Scaling Up (Tanzania
complement work toward
2007)
national plans and policies
Good stewardship of pilot project lessons learned:
shows good outcomes (three
o … the national guidelines and monitoring and evaluation
reports).
system which resulted from the pilot phase of the programme
are now used by collaborating partners who support programmes for most vulnerable
children with resources from the Global Fund and from PEPFAR [President’s Emergency
Plan for AIDS Relief]. A national implementing partners’ forum meets regularly, chaired
by the Department of Social Welfare, to review progress and discuss any problems
which may arise.20
20
The author also writes that a “national framework for social protection is under development” with an inference that UNICEF
is involved in that process.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
23

Evaluation of Lihlombe Lekukhalela Child Protectors (Swaziland 2005)
Good stewardship of a situation analysis:
o The DPM’s (Deputy Prime Minister’s) office coordinated the survey involving community
youth assessments …
UNICEF in collaboration with Save the Children, Deputy Prime Minister’s Office, National
Emergency Response Council on HIV and AIDS (NERCHA), Care Nakekela, Swaziland Action
Group Against Abuse (SWAGAA) has been able to set up child protection committees in 188
communities. A total of 5,690 child protectors have been trained on child protection.

Evaluation of the UNICEF Impact on the Juvenile Justice System
Reform (Serbia, Romania, Tajikistan) (2006)
More good stewardship of a situation analysis:
o …’best investments’…: A good situation analysis. The situation
analysis not only can be, through its dissemination and
presentation to government’s bodies, an effective instrument of
sensitization on the social importance of juvenile justice reform,
but can also, more specifically, contribute to a keen appreciation
on the part of governmental counterparts of the value of data
collection, evidence based planning, and monitoring. (emphasis
in original)
o Last but not least, one of the main strengths of UNICEF’s
programmes in support of juvenile justice reform is represented
by the important focus given to the creation of community
based alternatives. These initiatives vary from country to
country, but all of them have an important element of
innovation and are not only having a positive effect on the lives
of children, but also by the force of example strengthening
support for reform amongst professionals and decision makers.

"… one of the main strengths of
UNICEF’s programmes in
support of juvenile justice
reform is represented by the
important focus given to the
creation of community based
alternatives … all of them have
an important element of
innovation and are not only
having a positive effect on the
lives of children, but also by the
force of example strengthening
support for reform amongst
professionals and decision
makers."
CEE/CIS 2006
EXCERPT
External Evaluation of the Day Care Center “TISA” for Children with
Disabilities (Montenegro 2005)
Working in a non-traditional area to deliver a pilot project that is designed for replication:
o Tisa is an innovative project … *it+ represents the first community setting for children with
disabilities in Montenegro … designed to be replicated in Montenegro, through a network
of day care centres, for children and teenagers with disabilities. …The results and lessons
learned in the first year of functioning will contribute also to the analysis and propositions
related with the quality standards for this type of service in Montenegro.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
24
Implementation
2.2 Constraints to strengthening government commitments
Activities supporting national plans and policies are not strongly oriented toward delivery. This finding is
exemplified by a de-emphasis on monitoring and oversight, weaknesses in the area of sub-programme
coordination, and a tendency to underplay follow through. National plans may be in place, as one
evaluator found, without the requisite policy frameworks to see them through, or basic financial
commitments may not have been mobilized.
2.2.1 Limited follow through mechanisms and accountabilities
Constraints of this nature are described in the ten reports excerpted here. The first six have good
evidence quality ratings.

FINDINGS
Rehabilitation/Re-integration of Mine Victims and People with
Investments in national planDisabilities (Cambodia 2005) — Suggests deficits in attention to follow
and policy-building are not
through and missed advocacy opportunities:
strongly oriented toward
o There is a definite need for more strategic level support to
follow through (six reports).
partners to work through programming issues with
government services implementing other UNICEF funded
responses. Equally there is a need for UNICEF to undertake
high-level advocacy to government on issues affecting impact of Child Protection
Programme supported projects.
o There seems to be a need across UNICEF to examine inter-programme and subprogramme coordination.

External Evaluation of the Child Protection Network (Cambodia 2004) — Suggests that the
programme was put into place ahead of the necessary and enabling policy and fiscal measures:
o There is a lack of policy development from the part of the ministry. Monitoring and
evaluation conducted by the national level are superficial and lack analysis. The national
trainers who were supposed to take the role of linking the provincial and national level
have not been able to fill this role.
o From a structural point of view, the (CBCP) Community Based Child Protection
programme lacks a policy framework that would provide it with the necessary
legitimacy and pressure to mobilise stakeholders from national to community level.
o There is a lack of long term, sustained support from communities and government. …
As long as the government does not commit itself to contribute to the safety nets by
allocating appropriate budgets to provincial social affairs, there is little hope that the
situation of children improves substantially. Virtually all longer term solutions are
provided through service providers outside the community, notably NGOs.

Education as a Preventive Strategy Against Child Labour (Global 2003) — Suggests a potentially
missed advocacy and fundraising context to assure the investments’ future sustainability:
o The fact that the Programme has triggered the use of regular
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
25

and other resources for child labour-related activities in several Country Programmes is
a remarkable sign of growing commitment, but needs to be
complemented by a more massive mobilisation of resources
"The creation of networks and
for the fight against child labour by governments, civil
structures
is necessary, but not
society (including the private sector), and external support
sufficient
to ensure improved
agencies.
child protection. Systems need
Evaluation of the UNICEF Impact on the Juvenile Justice system
to be functional, supported by
Reform in Three Countries in the Regions (Serbia, Romania,
the actors involved, and
Tajikistan) (2006) — The findings here speak to general findings of
oriented
towards achieving
the meta-evaluation itself:
results. This requires in addition
o … main challenges … Reinforcement of results-based
to a planning framework, a
management within the UNICEF Country Offices, with
commitment from government
greater efforts to be deployed in order to develop:
to develop and apply policy and
well-defined long term strategies; more precise and
legislation."
concrete objectives; relevant and quantifiable indicators;
Cambodia 2004
tools for data collection and monitoring systems. More
emphasis should then be given to the evaluation of impact
EXCERPT
or outcomes, rather than outputs. Last but not least, greater
synergy and links between UNICEF’s work on juvenile justice and its work on other
areas should be created. (emphasis in original)

An Assessment of the Impact of Implementing the MVC Programme and the Operation of Funds
and the Potential for Scaling Up (Tanzania 2007) 21 — Suggests a failure to create ownership to
facilitate implementation:
o … the problems plaguing the implementation of the pilot programme persist: in many
areas, the programme continues to be viewed as an externally driven programme with
consequent lack of local commitment among stakeholders and facilitators at all levels
and mobilisation of community participation is limited

Impact Assessment of the Most Vulnerable Children (MVC) Community Based Care, Support,
and Protection in Musoma Rural (Tanzania 2004) — Shows more evidence of inattention to
implementation:
o The programme implementation process at all levels is constrained by poor coordination;
lack of commitment by the government leaders; lack of integration of the programme to
other development efforts; under-resourcing the programme; poor MVC data management
and lack of follow-up and lastly, lack of integration of the non-state actors in the
implementation of the programme (i.e., NGOs, FBOs).
21
On the face of it, a substantive counter example here would be the Tanzanian government’s establishment of a national
monitoring and evaluation system for the Most Vulnerable Child (MVC) programme. This appears however to have resulted
from a United Nations agreement on the “Three Ones … one coordination mechanism, one national action plan, and one
evaluation and monitoring system” (Tanzania 2007).
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
26
2.2.2 Lessons lost due to a weak evidence base
These four reports are among the weakest across the whole meta-evaluation for evidence quality.
Excerpts are provided here to highlight lessons lost:
FINDINGS

Midterm review - Gender and Child Protection Programme (Ethiopia
2004) — Without a strategy for implementation, the policy’s value is
probably chiefly rhetorical:
o Absence of a national strategy, however, is indicated to be
lacking for them22 to strongly and exhaustively pursue their
effort along this line. The national policy (on Ethiopian
Women), thus, needs to be accompanied by strategy to
effectively address gender concerns as well as mainstream it.

Evaluation of Alternative Care Programmes (Sierra Leone 2003) —
Missing data collection methods:
o Beginning in 1999, The National Commission for Disarmament, Demobilisation, and
Reintegration (NCDDR) began a district by district systematic process of demobilizing these
children who were attached to the fighting forces. Upon demobilization they were placed
in a nation-wide Child Protection Network (CPN) Interim Care Centre (ICC) system while
their families were traced … An accurate data collection system should be developed and
updated regularly.

Final evaluation of the Empowerment of Children in Need of Special Protection Project
(Bangladesh 2005) — Suggests the concept and design were shallow:
o Finally, any future project must articulate in more detail what is meant by an enabling
environment for adolescents, and more systematically develop the local and national
interventions required to create it.

Evaluation Report: Promoting the Deinstitutionalisation Process in the Republic of Tajikistan
(2005) — Unclear where UNICEF’s efforts helped to address the State’s apparent inaction:
o The lack of a clear direction by the State has resulted in a similar lack of an integrated
response from the international donor community and frequently there are examples of
various ‘projects’ and initiatives that could be far more effectively dovetailed in order to
give them a greater potential for sustainability.
22
Four reports address
investments relating to
national plan- and policybuilding and show that these
are not strongly oriented
towards follow through.
However, poor evidence
quality constrains the
usefulness of these findings.
The author is discussing “legal provisions observed in the Constitution, National Family Law as well as the drafting and
adoption and popularisation of the regional Family Law.”
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
27
Legislation and
Enforcement
Chapter 3:
More training on rights of MVC (Most Vulnerable Children) like inheritance rights should be
organized for the caretakers, MVC, and the community. The idea of rights and legal rights is
not well understood in villages. This is an area where impact has been minimal. . . .
Impact Assessment of MVC Support Programme Implementation in Karagwe, Kisarawe, and Magu (Tanzania 2004)
Introduction and Overview
Legislation and enforcement is a second element of the Protective Environment. An adequate
legislative framework, its consistent implementation, accountability, and a lack of impunity, are
essential elements of a protective environment.
Within the universe of this meta-evaluation, the “Legislation and Enforcement” component of the
Protective Environment ranks fourth for the quality of its programming practices. Evaluation reports
were analyzed on the basis of two sub-categories:

Legal development/reform reflecting child rights principles

Strengthened enforcement
The frequency of evidence on these sub-categories is minimal, ranking second to last (just ahead of
“Monitoring and Oversight”) across all the meta-evaluation materials. Legislative investments are more
highly rated than enforcement-related investments by 15% even though enforcement-related
investments (which are primarily about advocacy for enforcement, not enforcement mechanics per se)
outnumber legislative investments by 25%.
The bulk of legislative emphasis is in juvenile justice reform and alternatives to institutionalization.
Enforcement themes range from early marriage to community based re-integration, to preventing
gender-based violence and trafficking of children.
In the aggregate, this component of the Protective Environment is

Strong on networking and advocacy

Weak on strategic planning and management
In the context of legislation and enforcement, UNICEF is most active as a resource or facilitator for
bringing child protection issues onto the table. Conversely, in the implementation or trickle down arena
—for instance, getting district and regional partners to coordinate — UNICEF is either not seeking or is
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
28
not able to procure impacts. This aligns to findings presented in the previous chapter under “What
doesn’t work,” where efforts to cultivate national political will were found to be effective on the
“mobilization” side but lacking where the apparatus of implementation pertained.
What works — supports to legislation and enforcement
 Knowledge of the big picture; community level stakeholders making use of the threat of legal
sanctions.
What is not working — constraints to legislation and enforcement
 De-emphasis on monitoring accountabilities and enforcement mechanisms at the state level; a
disposition to prioritize advocacy for enforcement instead of literal enforcement mechanisms at
the state level.
Legislation
Synopsis
Among the 12 reports addressing legislative efforts explicitly, the average evidence quality rating is
about 16% lower than the average evidence quality for all the meta-evaluation reports.23 Only four
reports had evidence quality of good or better. Findings here are thus constrained by a relatively weak
evidence base as well as by the extremely limited sample size that was available for the metaevaluation.
Two themes occur commonly in legislation-related reports:

Within the universe of this meta-evaluation, legislative effort is concentrated in juvenile justice
reform and institutionalization.

A relative de-emphasis on strategic planning and coordination weakens these efforts.
23
Average for the 12 reports is 1.21; for all reports the average quality rating is 1.44. The more poorly rated reports shared two
drawbacks: they were end-point evaluations only and had no or limited comparison groups. As discussed in the introductory
chapter and as seen in evidence quality ratings for individual reports (Appendix Five), these are the two weakest links in
evaluation practice for the meta-evaluation as a whole. To convey the magnitude of lessons lost through poor evidence quality,
here is a tally of reports containing information about legislative efforts:
Good evidence quality (four reports)
o Legal reform/juvenile justice (CEE/CIS Zambia), alternatives to institutional care (Belarus, Bosnia, and Herzegovina).
Poor evidence quality (eight reports)
o Legal reform/juvenile justice (Thailand, Yemen), legislation to support social assistance (Romania), legislation and
advocacy for a regional “family law” (Ethiopia), alternatives to institutional care (Moldova, Serbia, and Montenegro),
child protection law (Indonesia [2]).
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
29
3.1 Investment characteristics that support legal reform
3.1.1 Technical supports that grasp the big picture
The common denominator here is UNICEF’s grasp of the big picture and where the investments would
move the knowledge-practice nexus forward:

Evaluation of the Deinstitutionalisation Project (Belarus 2004) — Tangible outcomes:
o … Development of the regulatory and legal frames for implementation of the policy on
deinstitutionalisation of orphaned children was one of three “direct outcomes” of the
project.
o
… An impact of this legislative reform is “Growth of the
number of children placed into families (in 2003, in the
majority of the regions 60% to 70% of children deprived of
parental care were placed into families).”
FINDINGS
• Improved legal and
regulatory frameworks for
juvenile justice reform and
institutionalization are
validated outcomes in three
reports.
•However, in two of these,
evaluators find no monitoring
or quality assurance
mechanisms.

Evaluation of the UNICEF Impact on the Juvenile Justice System
Reform (Serbia, Romania, Tajikistan 2006) — Systematic contributions:
o On the whole, activities supported by UNICEF have been
having a satisfactory degree of influence on the overall reform
and the development of system in most of the countries.

Evaluation Report on Child Justice in Zambia (2005) — Investing in a
quality situation analysis:
o Law reform: The current legislation is antiquated24 and is increasingly an impediment to
transformation and improved service delivery. It is therefore recommended that law
reform be embarked upon as a matter of priority … special attention should be given to
the definitions of child, age of criminal capacity, custodial sentencing, remand period,
diversion, and certain offences.
3.2 Constraints to legal reform
3.2.1 Legislative efforts are not addressing accountabilities for enforcement
UNICEF is engaged in activities such as financing of reports and situation analyses, technical assistance in
reviewing existing legal frameworks and developing new ones, lobbying, and convening inter-sectoral
coordinating entities. There is no evidence of investments related to monitoring and oversight, a gap
that evaluators cite in several reports.
Three reports have good evidence quality. Excerpts from them illustrate evaluators’ concern about
coordinating matters such as national–local linkages and developing management protocols to track and
follow up on the investments:

24
Evaluation Report on Child Justice in Zambia (2005) — Observes that awareness of the issues is
greater but, practically speaking, things are staying the same, and highlights the matter of
accountability:
Juvenile Act, 1956
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
30
o
o
On a general level it can be concluded that an increased awareness and sensitivity
towards children’s rights in the criminal justice system were observed in nearly all the
sites visited during the fieldwork. This has, however, not translated into a general
improvement in service delivery (especially outside of Lusaka) and children are still
subjected to nearly all of the ills, misuses, and delays that were observed in 2000.
Firstly, good governance principles need to be complied with, with specific reference to
management skills, ability to deliver, and quality control.

Evaluation of the UNICEF Impact on the Juvenile Justice system (Serbia, Romania, Tajikistan
2006) — Working in unfamiliar areas, regular accountability protocols were lost:
o UNICEF’s culture of accountability is one of its strengths, but juvenile justice reform is a
relatively new area for the organization and limited in-house experience and expertise
have had a negative impact on strategic planning, the development of indicators, the
monitoring and evaluation of implementation. In general terms, results-based
management needs to be strengthened.

Evaluation of the Program “Improved Mechanisms of Child Protection" (Bosnia and Herzegovina
2003) — A program in place before the legal framework:
o The following factors did influence outcomes, although they could have been foreseen
at program start: Legal framework / lack thereof on entity, state but also on cantonal
level.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
31
Enforcement
Where there are social workers, you can see the difference — she’s living in the community
and they know the structures where they can report.
Child Protection Advocate, Community Based Reintegration: Programme Evaluation (Sierra Leone 2006)
Synopsis
Two themes dominate:
FINDINGS
1. UNICEF Child Protection invests more in enforcement advocacy than in Investments cannot show
results without directly
enforcement strategies and mechanisms. UNICEF invests:
addressing two enforcement
o Almost exclusively in using its credibility and convening power
challenges: creating (a) the
for advocacy and mobilization.
mechanics, and (b) the
o Substantially in building the child protection literacy of
political possibility, of
national governments and professionals.
compliance with legal
o And substantially less in promoting a culture of monitoring
mechanisms (four reports).
and systematized reporting with regard to enforcement, or in
specific enforcement mechanisms.
 There is almost no instance of UNICEF leveraging its credibility and role as an
“honest broker” to press for monitoring and oversight regarding enforcement.
2. Meanwhile, community level stakeholders use the threat of enforcement to bring about
compliance.
o Stakeholders persuade others to follow the rules by pointing to the consequences of not
following the rules.
o But only insofar as government and donors maintain good visibility and communicate
commitments.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
32
3.3 Investment characteristics that support enforcement
3.3.1 Community stakeholders build on the threat of enforcement
Although muted, a theme can be noted in several reports wherein the expectation of enforcement is
a tool to be leveraged for the sake of child protection. This idea is reported from the point of view of
community stakeholders for whom the symbolic and pragmatic possibilities of enforcement are
meaningful.
FINDINGS

Early Marriage Prevention Program, Endline Report on Kishori Abhijan:
An Intervention Research Study on Adolescent Girls’ Livelihood
(Bangladesh 2005)
Government “buy in” has to be recognized as such by members of the
community:
o The collaboration of government representatives such as the
Union Parishad Member, Chairman, in addition to the
involvement of respected community members is especially
effective … in that people should be obliged to respect
decisions given by an official they, themselves elected to
office.

Community Based Reintegration: Programme Evaluation (Sierra Leone 2006)
Government “buy-in” has to be palpable in the communities. The following are excerpts from
members of child protection networks who were interviewed by the evaluator:
o People now know that there are people in charge who will prosecute you. They will
think twice before doing anything to children.
o If they don’t see action from the Ministry, they say, why do we bother? Ministry needs
to be strong to support them. If they go back and say, the kid you identified, this is what
happened, they will be motivated to continue. Where there are social workers, you can
see the difference — she’s living in the community and they know the structures where
they can report.
The data suggest that
UNICEF de-emphasizes
investments in the
mechanisms of enforcement
while simultaneously
investing in supports to
community level Child
Protection advocates. Some
good outcomes are seen in
spite of this discontinuity.
o

Since IRC left us, we only met once … Once they left, everything died down and fell
apart. We need IRC. When we had problems we consult
"People now know that there
them. No sooner than IRC left us with the promise that the
are people in charge who will
Ministry will follow suit. As of now, they have not followed
prosecute
you. They will think
up at all.
twice before doing anything to
Lessons Learned Review on the Law Enforcement Against Sexual
children."
Exploitation and Trafficking of Children Project (Cambodia 2006)25
Well situated to press for enforcement:
Stakeholder, Sierra Leone 2006
EXCERPT
22
World Vision appears to have been the lead agency in this multi-partner effort beginning in the late 1990s. UNICEF’s
investments are unclear, however, there is a note that support is to be extended as of 2006. World Vision managed and funded
this evaluation with one objective being “to make recommendations for UNICEF’s end-of-project evaluation.” This evaluation is
poorly rated for evidence quality.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
33
o
… is ultimately a government responsibility, requiring long-term commitment to
technical support to build capacity. Therefore, establishing the project within the MoI
[Ministry of the Interior] has been invaluable. … There is excellent ownership of LEASETC
within the MoI … has enabled the project to impact directly on high level government
decisions, while working closely with the police responsible for the issue.
3.4 Constraints to enforcement
3.4.1 Enforcement is largely at the level of moral persuasion
Four reports here have good or better evidence quality ratings. These support findings that a stand out
challenge for legislative investments is creating (a) the mechanics, and (b) the political possibility, of
compliance with legal mechanisms.

Protecting Street Girls and Preventing Child Migration (Ghana 2003) — Shows limited attention
to creating a broad “enabling environment” to sustain service delivery:
o The majority of Accra-based beneficiaries were victims of gender-based violence. At the
same time, RPP [Rights Protection and Promotion Programme] had no explicit
intervention to address this issue. There were no explicit interventions to make the
legal, community, or service environment more responsive to the needs of victims. (p.
49)

Evaluation of Grassroots Peace-Building Project (Sudan 2004) — Suggests an over-reliance on
legislation at the expense of how laws get enforced:
o The project could help children and young people monitor how customary law
governing children and women’s rights in South Sudan is developed and implemented.
Understanding and influencing legal practice is usually much more important than
influencing legislation.

External Evaluation of the Child Protection Network (Cambodia 2004) — Is UNICEF investing
more in form than function when it comes to putting laws into practice?
o

The creation of networks and structures is necessary, but not sufficient to ensure
improved child protection. Systems need to be functional,
"...More emphasis should then
supported by the actors involved and oriented towards
be given to the evaluation of
achieving results. This requires, in addition to a planning
impact or outcomes, rather
framework, a commitment from government to develop and
than outputs."
apply policy and legislation.
Evaluation of the UNICEF Impact on the Juvenile Justice system
CEE/CIS 2006
(Serbia, Romania, Tajikistan 2006) — Call to strengthen
EXCERPT
accountability:
o The following are the main challenges that UNICEF should face in order to make its
current and future programmes in support of juvenile justice reform more effective and
successful:
 Reinforcement of results-based management within the UNICEF Country
Offices, with greater efforts to be deployed in order to develop: well-defined
long term strategies; more precise and concrete objectives; relevant and
quantifiable indicators; tools for data collection and monitoring systems. More
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
34
emphasis should then be given to the evaluation of impact or outcomes, rather
than outputs.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
35
Attitudes, Traditions,
Customs, Behavior, and Practices
Chapter 4:
The prevalence of FGC reported among daughters aged 0 – 10 years decreased significantly
among women directly and indirectly exposed to the program.
The TOSTAN Program, An Evaluation of a Community Based Education Program in Senegal (2004)
It is the quintessential pressure group—an intergenerational alliance to prevent early
marriage.
Bangladesh Early Childhood Marriage Prevention Program (part of the Adolescent Girls’ Livelihoods Program, 2005)
Introduction and Overview
Addressing harmful attitudes, customs, behavior, and practices is a third component of the Protective
Environment. In societies where attitudes or traditions facilitate abuse—for example, regarding sex with
minors, severe corporal punishment, the application of harmful traditional practices, or differences in
the perceived status and value of boys and girls—the environment will not be protective. In societies
where all forms of violence against children are taboo, and where the rights of children are broadly
respected by custom and tradition, children are more likely to be protected.
Three evaluation reports26 are relevant to this component of the Protective Environment:

Early Marriage Prevention Programme, Endline Report on Kishori Abhijan: An Intervention
Research Study on Adolescent Girls’ Livelihood (Bangladesh 2005),

Protection of the Girl Child and Eradication of Malpractices (Egypt 2002)

Community-Based Child Protection Programme Evaluation and Review (Somalia 2006)
Because of this paucity of data, UNICEF Child Protection obtained a Population Council evaluation report
on UNICEF investments in programmes to stop female genital mutilation:

The TOSTAN Program, An Evaluation of a Community Based Education Program in Senegal
(2004)
26
The askSam database tracks evidence on changing attitudes and practices across 35 other reports. This material falls under
other elements of the Protective Environment such as legislative reform, service delivery, and raising capacities of those closest
to the child.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
36
In the aggregate, the reports represent opposite ends of the evidence quality spectrum:

The Bangladesh report is the third most highly rated for evidence quality of the entire metaevaluation. The Senegal report is also well validated. But only two reports out of the 59 that
were rated for the meta-evaluation are rated more weakly than the
Egypt and Somalia reports.
FINDINGS

Good evaluation practice as well as good programme practice (see
below) is a common element in the Bangladesh and Senegal
investments. Both have large sample sizes, comparison groups, access
to longitudinal data.
Two reports represent the
best in evidence practice with
large sample sizes,
comparison groups, and
longitudinal data.
4.1 Investment characteristics that facilitate changed attitudes,
practices
4.1.1 Intergenerational alliances, with peer counseling and health education
Early marriage
In the Bangladesh report, children’s and young people’s participation constituted the basis of a
distinctive intergenerational alliance to prevent early marriages. Elders in the community relied on the
young people as “eyes and ears” and followed up on news of a pending marriage negotiation with
dialogue with the girls’ parents. This programme included peer counseling and health education.

Early Marriage Prevention Programme, Endline Report on Kishori Abhijan: An Intervention
Research Study on Adolescent Girls’ Livelihood (Bangladesh 2005)
o Baseline and follow up studies (2001 and 2003) involving 5,024 adolescents found:



Project members paid less in dowry than non-project members.
Greater awareness of STDs.
Half of the negotiations attempted under the Early Marriage Prevention Programme
were successful in delaying marriage.
To “avoid detection,” marriage negotiations were taking place with increasing secrecy27:

Advocating against marriage before the age of 18 years is a simple yet potentially
effective community-based intervention undertaken by the adolescent members of
CMES (Centre for Mass Education in Science). The adolescents, consisting of
unmarried girls and boys, ally themselves with Adult Support Groups composed of
respected elders in the community, both male and female. Initially a part of CMES’s
Social Action program, advocacy against early marriage through this program was
given a prime focus. … This intervention uses the presence and knowledge regarding
marriages held by peer adolescents and the members’ networks to address the
issue of early marriage while in progress.
27
In the previous chapter, this programme was presented as an example of stakeholders’ use of the threat of enforcement as a
tool of advocacy.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
37
Female genital cutting/mutilation
Like the Bangladesh Early Marriage Prevention Programme, the Senegal programme to address and
curtail female genital cutting situated girls’ involvement as peer educators within a sustained education
and advocacy framework.

The TOSTAN Program, An Evaluation of a Community Based Education
Program in Senegal
Baseline (2000) and Endline (200428?) studies involving about 800
men, women, and girls, found:
o The prevalence of FGC reported among daughters aged 0–10
years decreased significantly among women directly and
indirectly exposed to the program. Life table analysis
confirmed this change in the intervention group, but also that
the girls who were cut were being cut earlier than before.
FINDINGS
Good outcomes: an excellent
evidence base supports
findings of effective
investments that embed
youth-to-youth outreach
within community-wide
education efforts addressing
sensitive topics (two reports).
Methodologically,
o
o
The basic education program consists of four modules: hygiene, problem solving,
women’s health, and human rights. Through these four themes, emphasis was placed
on enabling the participants, who were mostly women, to analyze their own situation
more effectively and thus find the best solutions for themselves.
Since 1997, TOSTAN has organized a public declaration by a large group of villages that
have agreed to abandon FGC as a strategy to enable the people themselves to renounce
a traditional practice without fear of social stigma.
4.2 Constraints to changing attitudes, practices
4.2.1 Lessons lost due to a weak evidence base
The primary constraint with the two remaining reports is their weak evidence base. The Somalia report
says advocacy on sensitive issues like FGM had led to “some instances of changed practices around
FGM” but this is not possible to verify.29 The evaluation further says that that community members “pick
and choose” issues to engage with and there is no information on why efforts to encourage different
thinking about female genital cutting might or might not be reaching people.
The Egypt report paints a picture of an essentially intractable set of attitudes about female genital
cutting. It suggests that the project training lacked focus and needed to have been supported better by a
28
Timelines are unclear.
Child protection advocates in Somalia encountered resistance in three communities because of the lack of services being
offered by the CPCM [Child Protection and Community Mobilization] programme. Communities who rejected the programme
stated that such activities were not a priority, and that they were capable of looking after their children. Alternatively,
communities commonly stated that they had to focus on food, shelter, water, and education to assist their children, and there
was a perception that child protection programming was irrelevant given that the community was lacking basic services and
could not provide for children in their care.
29
Sixteen reports (27%) include reference to child protection needing to be embedded in baseline survival efforts such as water
and food. See Chapter 11, “Cross-Cutting Issues,” for further details.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
38
substantive media campaign. These are generally stated findings, however, and it becomes unclear
exactly where the trainings so terribly missed their targets.
30

FINDINGS
Community-based Child Protection Programme Evaluation and Review
A poor evidence base limits
(Somalia 2006)
the instructional value of
o In instances where the CPAs [Child Protection Advocates] had
findings regarding outputs,
discussed culturally sensitive information such as FGM, the
for example, of trainings on
evaluation found that many communities had generally been
FGC/M that missed their
prepared to engage. Communities commonly referred to the
targets or of village advocacy
use of relevant Koranic verses and cultural teachings which
efforts that are
had changed perceptions, and in some instances changed
acknowledged and selectively
practices, around FGM.
ignored (two reports).
o In villages where communities continued to practice FGM,
there was still a level of interest in other issues such as child labour and violence against
children. Communities seemed to pick the issues from the CPS [Child Protection Study]
that were of interest to them and would pay less attention to other issues presented. In
most instances, the evaluation found that community disagreement with certain issues
in the CPS did not discourage the community from engaging in the programme overall.

Protection of the Girl Child and Eradication of Malpractices (Egypt 2002)
o Some physicians, nurses, and health workers stated that the training duration (two
days) … was not sufficient to address all aspects related to circumcision.
o Some trainers said that some girls require circumcision.
o Although they denied practicing circumcision, nine of a total of ten daya30s said that
“circumcision is essential for some girls.”
o Two-thirds of the nurses (60 of a total of 91) said “circumcision is necessary for a small
percentage of girls.” The rest of the nurses (31) said “it was not necessary for any girl.”
(p. 21)
o The majority of physicians chose the phrase “circumcision is necessary for a small
percentage of girls” (69 of a total of 82 physicians). Only 13 physicians said “circumcision
is not necessary at all for any girl.” (p. 21)
This term is often translated as “traditional midwife”
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
39
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
40
Open Discussion and
Engagement
Chapter 5:
… level of public awareness about the policy of deinstitutionalisation is low.
Evaluation of the Deinstitutionalisation Project (Belarus 2004)
None of the programmes included a component aimed at “changing public attitudes
towards children in conflict with the law and juvenile justice system reform.”
Juvenile Justice System Reform (Serbia, Romania, Tajikistan 2006)
The activities within the service have a high level of difficulty, because the child
deinstitutionalisation culture with the purpose of returning them into the stable family
environment does not exist yet.
Evaluation of the Development of Integrated Community Based Child Protection Services (Moldova 2004)
Introduction and Overview
Open discussion and engagement with child protection issues is a fourth element of the Protective
Environment. At the most basic level, children need to be free to speak up about protection concerns
affecting them or other children. At the national level, both media attention to and civil society
engagement with child protection issues contribute to child protection. Partnerships among actors at all
levels are essential for an effective and coordinated response.
Within the universe of reports that constitute this meta-evaluation, this theme is one of the more
meagerly represented elements of the Protective Environment. Readers may refer to the Figure 9,
Appendix 6, to see the relative frequency of evidence on this theme.31 Information on what works and
what does not has been tracked on the basis of five sub-categories:





Increase in public knowledge about the issue
Open discussion on sensitive issues
Use of media for advocacy work
Formal linkages to CRC are visible, part of the programme
Children feel able to speak and/or act more freely about protection concerns
Judging from the manner in which knowledge/advocacy activities are typically situated within
programme strategies and the generally lackluster manner in which impacts are monitored and
evaluated, “open discussion” is not so much a by-product of other elements of the Protective
Environment as, it would seem, a poor relation to them. Even the high visibility investments in juvenile
31
As previously stated, “relative frequency” was determined on the basis of the number of instances that any element of the
Protective Environment Framework Classification system received a programming practices quality rating.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
41
justice reform in Central and Eastern Europe failed to include a component aimed at “changing public
attitudes.”32
In the aggregate, this component of the Protective Environment:

Needs to distinguish among knowledge, awareness, and advocacy, and map strategies and
expectations accordingly.

Would benefit from an assessment of good practices in programme communications, also
known as C4D, Communication for Development.

Might consider that evaluation practice is a barrier to the identification of actual impacts. There
is a general tendency to list activities and outputs and outcomes and impacts—suggesting
among other things that impact analysis is not a high management priority.33
Much work is being done on the matter of harmful traditional practices such as early marriage and
female genital cutting. However, for the meta-evaluation only four evaluation reports were available
specifically on those topics (see previous chapter). At the same time, many reports describe broad
“sensitization efforts” across issues like child protection; protection of orphans (orphaned because of
war, HIV/AIDs, or natural disasters); street working girls/children; disabilities. Several standout examples
of what works have been identified. That said, this is not a category where outcomes are much
available. It must be emphasized that evaluation practice is a significant factor in this poor visibility.
Increase in Public Knowledge About the Issue
Synopsis
A finding for this sub-category of “Open Discussion” is that evaluators do not delineate what they mean
by terms such as “knowledge” as distinct from “advocacy” or “awareness.” Reading either on or in
between the lines, it is probable that these distinctions were not material to the project concept/design
either. Why should this matter?

In order to advocate, the need is to bring new arguments and ideas
and create awareness.

Raising awareness is a strategy for increasing advocacy.

Knowledge and awareness come together through information.
FINDINGS
Evaluators do not delineate
what they mean by terms like
“knowledge” as distinct from
“advocacy” or “awareness.”
32
Juvenile Justice System Reform (Serbia, Romania, Tajikistan 2006). Chapter 4, “Attitudes, Traditions, Customs, Behavior, and
Practices,” was limited to customarily difficult topics such as female genital cutting and early marriage.
33
The evaluator of the Eritrea report (2004), for example, says that “on the National Child’s Day in December 2000, advocacy
and sensitization on CRC was carried out; children were made aware of their rights and adults were sensitized on the same.”
This report is highly rated for evidence quality but this is almost a comment made in passing. The issue here is not only a
confusion of outputs with outcomes and impacts, it is that “public knowledge” was positioned as more or less a by-product of
other activities.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
42
Evaluation reports with good evidence of increases in public knowledge possess at least a tacit
understanding of the knowledge/information/advocacy nexus. There are five such reports in the metaevaluation. In addition, these reports draw attention to the fact that what works:

Is not only the provision of information, but

Doing so in a context where related child protection issues are already on the table, and

Working through administrative channels that have gone some way toward maturation already.
5.1 Supports to increased public knowledge
5.1.1 Leveraging already mature networks/administrative mechanisms

Evaluation of Lihlombe Lekukhalela Child Protectors (Swaziland 2005)
FINDINGS
The community based “child protectors” programme in Swaziland
Good outcomes: three
focused on protecting orphans and vulnerable children from abuse. This
reports find a “measureable
evaluation is well rated for evidence quality. According to the evaluator:
effect" of rights promotion
efforts in communities with
o The general awareness of children’s rights and child abuse is
pre-existing child protection
generally high and this is seen as a result of the initiative.
networks.
… It is positive too that the areas where most abuse cases
are reported are LL [Lihlombe Lekukhalela Child Protectors]
areas (up to 100%). This is the one indicator that puts all but one LL area ahead of areas
where the initiative has not been implemented.
The child protector initiative was essentially a sub-strategy of a larger effort coordinated through the
Deputy Prime Minister’s office to address the needs of children in poverty and orphaned by HIV/AIDS.
Child protector networks were thus born into a framework of already
“It is positive too that the areas
regularized partnerships and administrative pathways for community
where most abuse cases are
networking in support of this demographic.
reported are LL [Lihlombe
Lekukhalela Child Protectors]
Two additional reports indicate a similar convergence among administrative
areas (up to 100%). This is the
capacities and standing networks, with the resulting potential for impacts in
one indicator that puts all but
public knowledge. The first is well rated for evidence quality:
one LL area ahead of areas
 Evaluation of Disability Project (Cambodia 2005)
where the initiative has not
This programme operates within a context where the state has already
been implemented.”
been strongly involved in disabilities issues that also have a relatively
high international visibility. According to the evaluation authors:
Swaziland 2005
o There is ample evidence that all projects have had
EXCERPT
measurable effect in … reducing discrimination and
promoting understanding of the rights of disabled people.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
43

Thao Dan Street Children Program (Vietnam 2002)
This project has been operating for ten years and is known for “sustained” educational outreach and
communications addressing health and exploitative child labor. According to the evaluation authors:
o Our survey showed … higher awareness of needs and rights of street children in
communities where Thao Dan has opened houses for street children.
5.2 Constraints to expanding public knowledge
5.2.1 Investments in awareness-raising without service or policy follow up
In three reports, evaluators cite improvements in awareness without
reciprocal changes as yet in budgets or policies. It would appear that the
administrative apparatus necessary to transform awareness into action is still
evolving. The first two reports here are well rated for evidence quality:
FINDINGS
Constraints to outcomes:
increased "awareness"
without corresponding
budget commitments or
service delivery
improvements (three
reports).

Evaluation of Street Children and Street Mother Programme (Ethiopia
2003)
Unclear whether plans are in place to address the gap between talk
and serious action here:
o Of utmost importance in terms of contribution has been the huge strides made in
consciountization [sic]of the plight of street children and mothers among the different
strata of Ethiopian society. Although it has yet to reach the critical mass whereby it
translates into the enactment of policies and the allocation of public budget, the
awareness created has been instrumental in drawing the national attention to the
phenomenon. The government has formulated a national action plan and the media has
begun publicizing the plight of street children and mothers.

Report on Child Justice in Zambia (2005)
This situation—talk without action—has been reported already with respect to Legislation and
Enforcement, and will come up again with respect to Service Delivery:
o On a general level it can be concluded that an increased awareness and sensitivity
towards children’s rights in the criminal justice system were observed in nearly all the
sites visited during the fieldwork. This has, however, not translated into a general
improvement in service delivery (especially outside of Lusaka) and children are still
subjected to nearly all of the ills, misuses, and delays that were observed in 2000.

Mid Term Review UNICEF Romania Report on Child Protection Programme (2002)
Unclear about the plan to transform the talk into action:
o Institutionalised children and families and children at risk have received much attention
and have become a high priority of the government. The hopes for system changes and
better work-methods could be regarded as a positive outcome of all the efforts of
UNICEF, official agencies, and NGOs during the last years. It is now important that these
expectations will be met and transformed into sustainable programmes of
community-based approaches to the welfare of socially vulnerable children and families
in order to avoid a back-lash that could seriously slow down the process of
improvements.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
44
Open Discussion on Sensitive Issues
Synopsis
Of immense importance to UNICEF, this is a poorly represented topic within the universe of reports that
are available for this meta-evaluation: only five reports have been identified as having content that
evaluators have flagged as sensitive. Three reports (see below) fall into the category of harmful
traditional practices and thus were presented in the previous chapter.
Fortunately, also as discussed in the previous chapter, one of these reports does describe a stand-out
methodology that Child Protection may consider promoting as a good practice.
With a call for evaluation reports addressing this topic, a fuller picture of investments could be obtained.
The topics in this category are:




Early marriage (one report)
Female genital cutting (two reports)
Incest (one report)
Institutionalisation (one report)
5.3 Supports to facilitating open discussion
5.3.1 Intergenerational alliances, with peer counseling and health education
As reported in the previous chapter (Section 4.1.1), two reports with good
evidence provide (a) replicable methodology and (b) a demonstrated good
practice in giving voice to critical stakeholders across the age spectrum of a
community (Bangladesh, Early Marriage Prevention, 2005; Senegal, TOSTAN
Community Education, FGC, 2004). The Bangladesh report is the third most
highly rated report for the meta-evaluation in terms of evidence quality.34
5.4 Constraints to facilitating open discussion
FINDINGS
The meta-analysis finds a
good outcome addressing
“open discussion on sensitive
issues”—built upon the use of
a strong inter-generational
alliance in conjunction with
community education
programmes. This is a
methodology that Child
Protection will find widely
applicable.
5.4.1 Lessons lost due to a weak evidence base
As discussed in the previous chapter (Section 4.2.1), two reports with weak evidence contained
information about investments on the topic of female genital cutting (Somalia, Community-Based Child
Protection, 2006; Eradication of Malpractices, Egypt, 2002). In these reports, findings either cannot be
verified or point to a weakly coordinated and conceived advocacy plan.
34
The first and second highly rated reports are Report on Child Justice in Zambia (2005), and Evaluation of the Street Children
and Street Mother Partnership Programme (Ethiopia 2003)
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
45
Two additional reports describe investments on sensitive topics (incest and institutionalization) but both
possess an uncertain evidence base:

Institution Building & Mainstreaming Child Protection (Indonesia 2003)
Incest is described as a “taboo” topic but details remain at the level of perception, for instance:
o The number of cases brought up in public and investigated by the police has
considerably increased. The cases in 2004 include many more issues considered taboo
(like incest) than before 2000. It is perceived by LPA [Child Protection Group] and their
partners that public opinion regarding child rights has changed considerably during the
last six years.

Evaluation of the Development of Integrated Community Based Child Protection Services Project
(Moldova 2004)
“Child institutionalization” is described as a “delicate subject” and the author’s observation
points to the need for open discussion where one apparently does not exist:
o The activities within the service have a high level of difficulty, because the child
de-institutionalization culture with the purpose of returning them into the stable family
environment does not exist yet.
Use of Media for Advocacy Work
Synopsis
This is nearly the leanest area of the whole “Open Discussion” component of the Protective
Environment. In terms of the quality of the programming practices associated with it, “use of media for
advocacy work” also ranks among the lowest in the meta-evaluation.
FINDINGS
Only six reports address media-related activities explicitly and evaluation
reporting is a definite question mark: either the investments were not a focus
so were mentioned only in passing, as outputs, or good practices for
measuring impacts were not known to the evaluators.
Media-related investments
are mostly limited to
descriptions of activities and
some outputs (six reports).
5.5 Constraints to media/advocacy
5.5.1 Lessons lost due to a weak evidence base
Four of the least well rated reports for evidence quality are also those where media-related activities are
flagged. The constraint these reports pose is that for whatever reason the evaluators focus on activities
or outputs with no concrete analysis or evidence base addressing outcomes or impacts:

Evaluation of the UNICEF-supported Chikankata CBOSP & OVC Training projects (Zambia 2002)
— Descriptive, without analysis of impacts:
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
46
o
In Lusaka, participants from Anglican Children’s Project are engaged in community
mobilization and child abuse advocacy campaigns such as presentation of drama,
theatre, and group discussions especially in markets in nine communities.

UNICEF Project on Preventing Trafficking of Women and Children, Gender-Based Violence, and
HIV/AIDS (Bosnia and Herzegovina 2003) — Descriptive, without analysis of impacts:
o Two television videos on HIV/AIDS prevention were also produced for publicity.

Evaluation of District-based Initiatives on the Prevention of CSEC and Child Labour (Indonesia
2006) — Descriptive, without analysis of impacts:
o

There is excellent evidence of advocacy and awareness raising of children’s rights
(including a child’s right to education), the Child Protection Law, and the harm caused by
child labour, through extensive media campaigns and community education initiatives …
Although the consultant touched on this topic with local villagers, there was not enough
time for the evaluation to fully assess the relevance or effectiveness of the media
campaigns. However, from conversations at the local village there was evidence that the
campaigns have had a direct impact on community, employer, and parental attitudes to
child labour, which in turn has led to a decrease in the incidence of child labour
(particularly the most hazardous forms), as well as a decrease in the number of children
dropping out of school—and this is to be commended.
Institution Building & Mainstreaming Child Protection in Indonesia UNICEF Supported Child
Protection Bodies (LPAs) (Indonesia 2003) — Unclear how conclusions were reached:
o Awareness has been raised through public campaigns, research and publications, rallies,
and radio discussions. And to a large extent the greater awareness of the press has been
achieved by including the media as stakeholders in LPA structures. Additionally, the
training of media representatives has proven to be a very effective tool.
5.5.2 Viewer/reception expectations were misguided
In two reports, the evaluators find that investments that were intended to produce greater awareness
had contradictory impacts (Guyana), or the place of the media in the whole training process had not
been fully thought through in advance (Egypt). These are two poorly rated reports for evidence quality:
 Participatory Evaluation for the “Women and Children at Risk” Project of St. Francis Xavier
(Guyana 2002) — An ineffective communications effort:
"The public was not as aware of
o The public was not as aware of the project as SFX initially
the project as ... initially
thought. Most persons seemed to be more aware of the
thought.
Most persons seemed
personalities they saw on TV than the message being sent
to be more aware of the
out.
personalities they saw on TV
 Protection of the Girl Child and Eradication of Malpractices
than the message being sent
Evaluation Report (Egypt 2002) — This investment seems to have
out."
been unable to put its finger on the roadblocks to changed attitudes
Guyana 2002
(see previous chapter) yet calls for media investments:
EXCERPT
o Recommendations … Activating the role of the media,
particularly the television because of its influence on public opinion, and encouraging
central and regional channels to raise the issue for discussion through drama and
programs. Such measures would help raise awareness and create anti-circumcision
social climate and public opinion.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
47
Formal Linkages to CRC Are Visible, Part of the Program
Synopsis
If the field of reports that has been available to the meta-evaluation is anything to go by, the
incorporation of the United Nations Convention on the Rights of the Child (CRC) is another facet of
building a protective environment that lacks a clear status.



Five reports describe impacts that can be verified, where the CRC has been an intended
mainstay of the program.
Nine reports describe the CRC as an intended program mainstay but with mixed impacts or
impacts that are difficult to verify.
Three reports describe programs where the CRC should have been a mainstay, but wasn’t.
As discussed in the introductory chapter, it is unfair to find evaluation reports wanting simply because
the meta-evaluation has defined a set of priority topics and themes in retrospect. What is necessary
here is to carry findings and observations forward into model Terms of References so that future
evaluations address matters issues that Child Protection would most like to track.
5.6 Supports to facilitating rights awareness
5.6.1 Using the CRC to a programming lingua franca
Impacts where the CRC has been a centralized component of the project are
found in five reports, of which four are well rated for evidence quality. The
findings show that the CRC helped to legitimate certain programming
emphases, that it acted as a common framework in building advocacy efforts,
and that it set the context and rationale for facilitating young people’s civic
and political participation. 35
FINDINGS
Where the CRC set the
context and rationale of CP
investments—explicitly—
outcomes were good:
activating partners' political
commitments to children's
rights and facilitating young
people’s political
participation (five reports).

Protecting Street Girls and Preventing Child Migration (Ghana 2003) —
CRC used to legitimate and organize investment emphases:
o … providing vocational training in support of Article 28 (right
to education), primary healthcare through outreach clinics in
support of Article 24 (right to health care).

Street Children Program (Peuan Mit) (Laos 2006) — CRC used to facilitate civic engagement and
education:
o Peuan Mit facilitated inputs by children and young people into the drafting of a new
Children’s Law, which is now being reviewed by the National Assembly.
35
However, it is impossible to say how much the evaluators’ own attentiveness to the CRC allows readers to see the
investments through the prism of the CRC, and whether, as a result, such realities are present but invisible in other
programmes. A Model Terms of Reference might ask evaluators to take the measure of the Convention on the Rights of the
Child in terms of its practical and rhetorical place in the programming design.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
48

Bam Emergency Programme (2003–2006) (Iran [Wiles et al] 2007) — CRC used to raise the
visibility of children’s issues:
"An extremely positive
o An extremely positive achievement for UNICEF in their Bam
achievement for UNICEF in their
earthquake response is that they succeeded in reminding
Bam earthquake response is
policy makers and government of the importance of
that they succeeded in
children’s rights in the response. Several senior civil servants
reminding policy makers and
told the evaluation team that UNICEF reminded the
government of the importance
authorities that children’s issues were important … keeping
of children’s rights in the
children and their right to assistance following the
response."
earthquake on the policy agenda was an achievement.

Iran 2007
Education as a Preventive Strategy Against Child Labour: Evaluation
of the Cornerstone Programme of UNICEF's Global Child Labour
EXCERPT
Programme (Global 2003) — Helped to frame child labor issues
around a common theme:
o In general, the Programme was considered relevant by participating countries in a
variety of ways: activating their political commitment to the CRC …

Evaluation of emergency child protection programs in Bam (Iran [Lazo and Balanon] 2006) —
Helped to introduced a rights-based approach:
o The experience promoted a paradigm shift within SWO from an orientation that
separated and orphaned children should be immediately sent to orphanages to a
rights-based approach that promoted fostering with extended family members.
5.7 Constraints to facilitating rights awareness
5.7.1 Investments are framed without reference to the CRC
In three instances (the first two with good evidence quality ratings), evaluators say explicitly that
mapping investments explicitly to the Convention on the Rights of the Child had not happened: in the
monumental CEE/CIS juvenile justice reform process, in the peace building project in Sudan, and in
Egypt’s urban child protection projects.

Report of the Organizational Development Assessment and Capacity Building of UNICEF Child
Protection Partners in Somalia/land (Somalia 2005) — No buy-in around the CRC as an advocacy
tool:
o … most (of the Child Protection Network partners) … confessed to have come together
at the behest of UNICEF and not in order to address a need they had perceived to exist
in the Children’s or Human Rights Sector and which required collective action. None of
the Networks was found to have taken part in any advocacy or openly activist activities
necessary for the challenge of impunity in the violation of children’s rights.

Evaluation of the Program “Improved Mechanisms of Child Protection” (Bosnia and Herzegovina
2003) — The civics of the CRC (participation, rights to information, non-discrimination) could
have been leveraged but weren’t:
o Gender and human rights issues should be highlighted better and incorporated
accordingly to reflect local requirements.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
49

Community-based Child Protection Programme Evaluation and Review (Somalia 2006) — An
interesting challenge here to translate universal commitments in local cultural currency:
o CPC [Child Protection Committee] and community members stated that prior to the
presentation of the CPS [Child Protection Study]36 they had not viewed issues such as
child labour and other harmful practices as a problem; rather they were seen as
common practice and a normal way of life … Whilst a handful of communities had
claimed an increased understanding of children’s rights which they credited to the CPS
presentation, the majority of communities viewed information on the Convention on
the Rights of the Child and other human rights frameworks as ‘removed’ from their lives
or as a ‘foreign’ concept.
Two other reports indicate that there was an intention to highlight or leverage the CRC to support
children orphaned by war or HIV/AIDS, but that the degree of integration of the CRC was “minimal”
(Tanzania 2004; Zambia 2002) and describes the position of the CRC in the programming as essentially
ceremonial (Eritrea 2006).
5.7.2 Lessons lost due to a weak evidence base
CRC-related ”sensitization” activities are claimed in three reports but impacts are difficult to verify due
to the weak evidence base (Bosnia and Herzegovina 2003; Indonesia 2003; Indonesia 2006).
Children Feel Able to Speak and/or Act More Freely
About Protection Concerns
The status of child participation and involvement as represented in the reports that were available for
the meta-evaluation may be found in the next chapter, “Children’s Life Skills, Knowledge, and
Participation.”
36
UNICEF 2003
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
50
Children’s Life Skills,
Knowledge, and Participation
Chapter 6:
The principle that children have the right to be heard, and to have their views taken into
account, is one of the basic principles that underlies the CRC. However, thus far children
have had little or no role in the design and implementation of the juvenile justice reform
projects.
Evaluation: Juvenile Justice System Reform (Serbia, Romania, Tajikistan 2006)
Children surveyed at the “Drop in center” rated “Life skills trainings” below “Games,”
“Dancing,” “Sport,” and “Education” in terms of services they like, but above “Clinic,”
“Counseling,” and “Haircut.”
Street Children’s Progamme Evaluation (Laos 2006)
Introduction and Overview
Children’s life skills, knowledge, and participation constitute a fifth element of the Protective
Environment. Children are less vulnerable to abuse when they are aware of their right not to be
exploited, or of services available to protect them. With the right information, children can draw upon
their knowledge, skills, and resilience to reduce their risk of exploitation. Children also need to be
provided with safe and protective channels for participation and self-expression.
Within the universe of this meta-evaluation, the “Life Skills” component of the Protective Environment
ranks fifth in size and in the quality of programming practices associated with it. Evaluation reports were
analyzed on the basis of three sub-categories:

Insights into what children/young people know/need to know.

Relevant information/services are provided (e.g., rights trainings, free counseling, condom
distribution, school curricula, child media, peer-to-peer activities).

Children/young people are facilitated to get involved (e.g., establishment of youth networks).
The latter category, participatory processes, carries about 25% less programming practice value than the
first two categories.
In the aggregate, this component of the Protective Environment:


Has room to grow where participatory design, implementation, and monitoring are concerned.
Would benefit from assessing robust good practices in life skills delivery, monitoring, and
evaluation, and from higher standards of evaluation practice.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
51


Can deliver good practices that will be valuable across a range of sectors.
Should probably be reviewed for its relative degree of priority in Child Protection investments.
Insights into What Children and Young People Know
and Need to Know
Synopsis
This category was set up to track information that would shed light on how program designers figured
out what young people know and need to know. The results are extremely sparse:

Only three evaluation reports imply or are explicit in describing life skills investments as guided
by direct consultations with children and young people.

Another three reports indicate that a proper needs assessment or outreach mechanism was
missing.
This is an alarming finding. Two possible interpretations are:

Consultative processes with younger stakeholders are relatively de-emphasized at the design
stage in building protective environment investments in reports that are available for the metaevaluation, or

Evaluators and evaluation managers are not prioritizing analysis on this topic.
The remainder of this section and the rest of the chapter suggest that both interpretations are probably
correct.
6.1 Finding out what young people know and need to know
Findings can be grouped into two themes:

Physical proximity to and regular contact between service providers
and stakeholders sharpen programme relevance.

Nevertheless, top-down planning led Child Protection to miss the
mark in several instances. This is the mirror opposite of the bullet
point above, suggesting that in all probability a standard of
programming practice has not been set or monitored effectively.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
52
FINDINGS
Regular contact between the
service providers and the
young people’s networks
serves as a rolling needs
assessment, but examples
are few and no set standard
of practice seems to be in
play (six reports).
6.1.1 Regular contact acts as a rolling needs assessment
In the three reports excerpted here, regular contact between the service providers and the young
people’s networks acts like a sort of rolling needs assessment. The localized geographic settings
(especially in the investments addressing street working children) may be significant in making this
feedback loop work. The relevance of geographic proximity between young people and programme
design and budget focal points for a program’s relevance and insightfulness would be a fruitful topic for
further research.
Of the following three reports, only the first report has a good evidence
quality ranking:

Review of Street Children’s Project (Laos 2006) — Strong outreach
efforts in a mostly localized area:
o The team maintains an intense schedule of work, regularly
visiting 50 specific locations around the city. The outreach
team was found to have good knowledge about individual
street children, knew many by name and had developed
trusting relationships.
"The [outreach] team
maintains an intense schedule
of work, regularly visiting 50
specific locations around the
city."
Laos 2006
EXCERPT

Thao Dan Street Children’s Program (Vietnam 2002) — Strong outreach efforts in a mostly
localized area:
o … To do this work, Thao Dan volunteers and staff rely on their network of street children
in various locations to assist them in locating new and vulnerable children on the street
so they can explore the possibility of reuniting the child with his/her parents before the
street life truly takes hold.

Report on the Evaluation of the UNICEF-Supported Chikankata CBOSP & OVC Training Projects
(Zambia 2002) — Suggests good networking with young people but without describing the terms
of reference between programme designers and young people:
o Through drama and discussions, 19 cases of various types of child abuse have either
been identified or reported to the outreach workers. Of the 19 cases, 10 are sexual
abuse cases involving all female children ranging from 6 months to 18 years of age.
6.2 Constraints to finding out what young people know and need to know
6.2.1 Top-down programming missed vital details
The excerpts here mostly speak for themselves: missing needs assessments
limited service providers’ ability to be proactive in designing relevant supports.
The first two of the following three reports have good evidence quality ratings:

FINDINGS
Poor outcomes resulted from
top-down planning, bypassed
needs assessments, and
missed opportunities (three
reports).
External Evaluation of the Child Protection Network (Cambodia 2004)
— Programme designers were operating more on instinct and less on
fact:
o There is some concern that the CBCP [Community Based Child Protection Network] was
established without undertaking a substantial assessment of child needs and protection
issues in the target areas. As a result there is a lack of baseline data and information on
children’s issues, which would ensure that interventions are focused and relevant.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
53


Impact Assessment of MVC Support Programme Implementation in Karagwe, Kisarawe, and
Magu (Tanzania 2004) — Suggests a lack of contact between service
"…the community based child
providers and stakeholders who were supposed to benefit from the
protection network was
services:
established without
o It seems that MVC are still not aware of their legal rights.
undertaking
a substantial
The idea of legal rights seems to be new and unknown for
assessment of child needs and
most caretakers.
protection issues in the target
Evaluation of Psychosocial Rehabilitation of Children (Thai-Burma
areas. As a result there is a lack
border 2006) — In the haste to provide services, a broad one-sizeof baseline data and
fits-all mentality meant that programme designers never had access
information on children’s
to essential facts:
issues…"
Cambodia 2004
o Psychosocial programming … was made even before the
Baseline Needs Assessment was completed. There is no
EXCERPT
indication that adjustments in the programming were made
later. Some data gathered, particularly of coping techniques
of children in the context of their culture … were not incorporated in the programming.
For example, children were found to prefer to remain silent, to forget and refrain from
expressing emotions related to their experiences as refugee children.
6.2.2 Lessons lost due to a weak evidence base
Several additional reports describe investments in outreach workers or mobile teams who act as “eyes
and ears” and deliver services. Unfortunately, the extent and timing of young people’s inputs was
missing or impossible to infer. Because all of the reports for the meta-evaluation have been entered into
a keyword-searchable database, “askSam” (see “Tools and Methods, Introductory Chapter) readers may
use mobile* to tap into this state of play in the askSam database.
Relevant Information and Services Are Provided
Synopsis
"Life skills"… refers to a large group of psycho-social and interpersonal skills which can help
people make informed decisions, communicate effectively, and develop coping and selfmanagement skills that may help them lead a healthy and productive life. Life skills may be
directed toward personal actions and actions toward others, as well as actions to change the
surrounding environment to make it conducive to health. 37
The meta-evaluation finds one report addressing “relevant information and services” where the
evidence quality is reasonable. 38 The overall picture, however, is perplexing:
37
http://www.unicef.org/lifeskills/index_7308.html
38
A boolean search in the askSam database facilitates access life skills-related information.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
54

Whereas 17 evaluation reports contain information about life skills-related investments,

16 of these cannot deliver information on impacts because of poor evidence quality (seven
reports) or for a variety of other reasons (nine reports).
It is impossible to know whether this situation stems from a relative deemphasis within Child Protection on the impact analysis of life skills
investments (and if so, why that would be the case)—the Terms of Reference
were not available in too many instances to make this determination. In the
opinion of this evaluator, life skills investments were generally treated in a
taken-for-granted way in the evaluation reports: not exactly decorative, but
not strategically worth much of a second glance.39
FINDINGS
Whereas 17 evaluation
reports contain information
about delivery of life skillsrelated outputs, only one has
a clear outcome with a clear
evidence base to support it.
6.3 Supports to delivering relevant life skills
Findings can be grouped into two themes:
 Relevance is facilitated by the ongoing contact between service providers and young people’s
networks.
 Life skills investments are unmonitored, missing, or un-ambitious.
6.3.1 Regular contact acts as a rolling needs assessment

Early Marriage Prevention Program, Girls Livelihoods Program (Bangladesh 2005) — The
practical applicability of the life skills investments were important
“… the programs … hoped to
here (see also 4.1.1):
instill ‘life skills’ in the form of
o Members were significantly more likely to report awareness
improved negotiating ability,
concerning STDs and pregnancy.
increased information
o … the programs comprising Kishori Abhijan hoped to instill
regarding reproductive health,
‘life skills’ in the form of improved negotiating ability,
etc. that can potentially
increased information regarding reproductive health, etc.
improve the situation of new
that can potentially improve the situation of new brides in
brides in their marital homes.”
their marital homes.
Early Marriage Prevention Program,
o Some of the qualitative data suggest that members were
Bangladesh 2005
able to positively utilize their experience in programs to
cultivate good relations in their marital home.
EXCERPT
6.4 Constraints to delivering relevant life skills
In the following nine reports, the common elements are that life skills trainings were missing (two
reports), were not monitored for impact (two reports), were constrained by service providers’
39
In a model Terms of Reference, Child Protection may consider the level of priority to be placed on impact analysis of life skills
investments, with supporting information on good practices in monitoring and evaluating them.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
55
unfamiliarity with the materials (two reports), could have been taken to scale but weren’t (one report),
or described in terms that are too generic to be informative (two reports).
FINDINGS
In reality, the elements described just now run through all the reports to some
extent. The shared element in common is that, very clearly, they were not the
evaluator’s main area of focus. Except for the last report in this section, all the
reports excerpted here possess a good evidence quality rating.
6.4.1 Life skills component is missing
Across nine reports (nearly
10% of the meta-evaluation
content), life skills trainings
were needed but missing
(two), were not monitored
for impact (two), outcomes
were constrained by service
providers’ unfamiliarity with
the materials (two), limited in
scale (one), or described too
generically (two).

Evaluation Report on Child Justice in Zambia (2005) — Are life skills
trainings “elective”?
o A planned training programme for 2003 that would have
focused on the life skills programme did unfortunately not
materialise due to funding constraints.

Report of projects aimed at the prevention of child abandonment in maternities (Romania 2003)
— Trainings would have been vital:
o According to her own experiences, the social worker noticed that the family planning
staff did not have sufficient patience to explain to their customers the mode of
employment of the various family planning methods … One of the mothers, who chose
contraceptive pills, took all of them at once, because she didn’t know what was the
correct way of taking these pills. Also, many mentally retarded or mentally ill women
were given contraceptive pills.
6.4.2 Training deficiencies constrain delivery

Evaluation: Protecting Street Girls and Preventing Child Migration (Ghana 2003) — Is there a de
facto assumption that anyone can teach life skills?
o “… life skills is a new area … many staff … do not have the knowledge or skills to teach
life skills or health education …”
o Life skills has been identified as a weakness in the programmes. While RPP (Rights and
Protection Programme) is making progress toward the objective with regard to basic
services, it is not making headway in enhancing life skills.

Community Based Reintegration programme (Sierra Leone 2006) — Trainings were insufficient
and narrowly conceived:
o Unfortunately, by and large, the trainers had little pedagogical know-how and little
experiences in teaching.
o … while there are some important life skills emerging from the *children’s+clubs, such as
leadership and public speaking, the main activities are recreational.
6.4.3 No impact monitoring
 Review of the Street Children Project (Laos 2006) — No impact monitoring:
o A “’mobile school” includes trainings on life skills which have not been monitored for
impact.
o A “Drop in” center offers life skills trainings but “sessions were not taking place at the
time of this review therefore no assessment of quality is possible.”
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
56
o

Children surveyed at the “Drop in center” rated “Life skills trainings” below “Games,”
“Dancing,” “Sport,” and “Education” in terms of services they like, but above “Clinic,”
“Counseling,” and “Haircut.”
Evaluation of the UNICEF Impact on the Juvenile Justice System Reform (Serbia, Romania,
Tajikistan 2006) — No monitoring of life skills:
o No attempt has been made to measure any impact of the activities, projects, and
strategies used by UNICEF on the empowerment of children and families, in any of the
countries covered by this evaluation.” Examples of program models include:
o A non-custodial, community based programme (district level, Tajikistan) for the
rehabilitation of juvenile offenders was established in 2004 … The services offered
consist of a package that includes remedial education, vocational training, life skills
training, family counseling, and free medical care … official statistics indicate that
offending by juveniles has decreased … since this programme began.
o Community based alternatives (Serbia): there is ample anecdotal evidence that the life
skills of some and perhaps many of the children in this facility have improved.
6.4.4 Not taken to scale

BAM Emergency Programme 2003–2006 (Iran [Wiles et al]2007)
o It seemed that expansion of life skills training to all children, which UNICEF started in 16
junior schools and high schools in 2006, would have been very relevant.
6.4.5 Lessons lost due to a weak evidence base

Education as a Preventive Strategy Against Child Labour (Global 2003) — The description is too
generic:
o … it appears that several participating countries are incorporating life skills trainings
(e.g., Cambodia, Panama, West Bank and Gaza), however:
o … critical dimensions linked to child labour, such as girls’ education, life skills education,
HIV/AIDS, ECCD, health, nutrition, and water, environment, and sanitation (WES), did
not receive adequate attention in the timeframe of this programme with the exception
of a few countries.

Evaluation of the project on reintegration and rehabilitation of children associated with militia
(Somalia 2006) — A generic and uninformative description:
o … non-formal training, combined with the acquisition of generic skills (applied
non-formal education and life skills) and vocational skills is most appropriate and should
be continued.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
57
Children and Young People Get Involved
Synopsis
Of 59 reports that constitute the universe of this meta-evaluation:

Seven reports contain evidence about the efficacy of children’s and young people’s
participation; the evidence quality is good in just four of these reports.

Six reports take note of participation without analysis of impacts.

Eight program evaluations found that child participation had not been incorporated but should
have been.
FINDINGS
Given the corporate emphasis on children’s participation, these statistics are
alarming. Still, from the point of view of the meta-evaluation, is this glass half
empty or half full?
21 reports mention child
participation but only 4
contain validated evidence
and of these, most are
output, not outcome,
analyses.

Half full: at least 21 reports—one-quarter of all reports in the metaevaluation—carry some mention of child participation.

Half empty: in just seven reports, child protection stands to obtain
information on what works and what doesn’t regarding child participation in the service of
building a protective environment. Only four of these reports have good or better evidence
quality ratings.
6.5 Investment characteristics that support child participation
Findings can be grouped into three themes:



In three instances, the proximity of service providers and stakeholders facilitated participatory
processes.
Nevertheless, in four instances children and young people were excluded at the level of
programme design.
And in eight reports, children’s and young people’s participation is said to be missing altogether.
6.5.1 Close proximity of service provides and young people
Three evaluation reports contain good practices for child participation, the first two with evidence
quality ratings. It should be emphasized, however, that only in the first case (the intergenerational
alliance to contest inappropriate early marriages [Bangladesh 2005]) were evaluators specifically looking
at participatory methodologies.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
58

Bangladesh Early Childhood Marriage Prevention Program (part of the Adolescent Girls’
Livelihoods Program, 2005) — The service providers are in regular touch with and close
geographic proximity to the young people’s networks, with a very specific mandate:
o Girls and boys as “Peer Leaders” participate in an inter-generational alliance with
respected elders and parents to advocate against early marriage.
o “It is the quintessential pressure group”

Street Children Program (Peuan Mit) (Laos 2006) — Regularized opportunities for feedback, with
close geographic proximity between service providers and young people:
o
o

Peuan Mit facilitated inputs by children and young
people into the drafting of a new Children’s Law,
which is now being reviewed by the National
Assembly.
At the Drop-in Centre, weekly meetings are
organized between children and project staff to
discuss ideas for improving services; children have
elected a youth representative to act as a focal point
for staff-child dialogue and to follow up on key
issues.
Thao Dan Street Children’s Program (Vietnam 2002) — This
program is well known for its outreach efforts:
o Outreach at Thao Dan is a huge effort led by ten
student-age volunteers and directed by the
“volunteer coordinator.”
"At the Drop in Centre, weekly
meetings are organized
between children and project
staff to discuss ideas for
improving services; children
have elected a youth
representative to act as a focal
point for staff child dialogue
and to follow up on key issues."
Laos 2006
EXCERPT
6.6 Constraints to child participation
6.6.1 Exclusions at the program design and planning stages
Four reports cite participatory processes that did not go far enough. The first two reports are well rated
for evidence quality:

Sierra Leone Community Based Reintegration Program (2006) — Weak buy-in to leadership
training and gender exclusivity are not explained:
o … while there are some important life skills emerging from the *children’s+ clubs, such as
leadership and public speaking, the main activities are recreational and “Girls are grossly
underrepresented.”
"A weakness in almost all of the
programmes is that children do
not participate in the planning,
or running of the programme."
Ghana 2003
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
59

Protecting Street Girls and Preventing Child Migration (Ghana 2003)
EXCERPT
— A typical model of top-down planning that is participatory only
during the implementation phase:
o Child migrants returning from Accra have also been mobilized as “Anti Child Migration
Peer Educators” in order to raise awareness on the risks and harmful effects of
rural-urban migration.
o A weakness in almost all of the programmes is that children do not participate in the
planning, or running of the programme.
 Indonesia Psychosocial Interventions (2003) — Another design phase exclusion:
o The strengths of this design [Psychosocial Training Project] in West Timor were that it
was comprehensive and multi-levelled—parents, teachers and youth volunteers trained
by the project provided social support to beneficiaries, while counsellors and social
workers also provided counselling for those needing more specialised assistance.
o Future projects should involve children in their design and implementation.
 Indonesia Evaluation of district-based initiatives on the prevention of CSEC and child labour
(2006) — Marginalized stakeholders are twice marginalized since participatory opportunities
are not reaching them:
o In Indramayu children are being trained as facilitators and are training their peers on
Children’s Rights, and protection from abuse of exploitation. However, it is observed
that the network of this forum needs to be expanded to include children who are no
longer in school and who are involved in, or at risk of being
involved in, prostitution.
FINDINGS
Eight reports describe
6.6.2 Investments are not framed with respect to participation
investments where child
Eight reports describe investments where child participation should have been
participation should have
been in-built. These include
built in. These include three community-based child protection programs and
three community-based child
the very sizeable investments in juvenile justice reform in Central and Eastern
protection programs and the
Europe:
very sizeable investments in
 Community based child protection programs (Somalia, Cambodia,
CEE/CIS juvenile justice
Egypt)
reform.
 Iran earthquake emergency programming

Juvenile justice efforts in the CEE/CIS and Thailand

Peace building in the Sudan
6.6.3 Lessons lost due to a weak evidence base

At least 21 reports—one-quarter of all reports in the meta-evaluation—carry some mention of
child participation. In just seven reports, child protection stands to obtain information on what
works and what doesn’t regarding child participation in the service of building a protective
environment. Only four of these reports have good or better evidence quality ratings.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
60
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
61
Building Capacities of Those
Closest to the Child
Chapter 7:
Community frustration and desperation to access basic resources proved to be the biggest
barrier.
Evaluation: Community Based Child Protection (Somalia 2006)
An important outcome of all activities has been the establishment of a professional
community of specialists who have special knowledge and skills to use modern technologies
of work … Today in Belarus, the existence of the substitution families does not depend on the
UNICEF funding.
Deinstitutionalisation and Alternative Placement of Children (Belarus 2004)
Introduction and Overview
Building the capacities of those closest to the child to recognize and respond to child protection issues
is a sixth element of the Protective Environment. Parents, health workers, teachers, police, social
workers, and many others who care for and live, deal, and work with children need to be equipped with
the skills, knowledge, authority, and motivation to identify and respond to child protection problems.
Within the universe of this meta-evaluation, the “Capacities of Those Closest to the Child” component of
the Protective Environment is the third most highly rated for the quality of its programming practices. It
is also an area of dense aggregate activity evidence (see Fig 9, Appendix 6). Evaluation reports were
analyzed on the basis of these three sub-categories:
a) Major actors are facilitated in recognizing child protection needs.
o Family
o Community/Civil Society Organizations
o Professionals
b) Major actors are facilitated in responding to child protection needs.
o Family—parenting, etc.
o Community/CSOs
o Professionals
c) Capacity development is becoming systematized and integrated, not ad hoc.
Among these sub-categories:
 The meta-evaluation tracks 31% more activity in the area of responding to child protection
issues than in recognizing these issues.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
62



Across both “response” and “recognition” categories, professionals’ capacity building saw the
highest programme practices quality, with 19% more overall quality than investments in
communities’ capacity building.
Capacity building of families was not an area of much emphasis in the reports that were
available for the meta-evaluation, carrying about half the evidentiary information as the other
two categories.
As a whole, efforts to strengthen recognition and response capacities of families, communities,
and professionals have a higher programming practices quality by 11% than efforts to
systematize capacity development.
Reminder: 6a, 6b, 6c40 refer respectively to enhancing stakeholders’ capacities
 To recognize child protection issues (6a)
 To respond to child protection issues (6b)
 To have steady (as opposed to ad hoc) capacity building opportunities on both recognizing and
responding to child protection issues
In the aggregate, this component of the Protective Environment:
FINDINGS

Points to the importance of training programme content and delivery,
such that length and/or quality of training and support make the
difference, not just a difference. Combining results for families,
communities, and professionals,
o Training is touted as reason for success in (a) 4 of 17; (b) 11 of
18; and (c) 4 of 14 reports.
o Inadequacies in training are said to impede success in (a) 5 of
16; (b) 13 of 24; and (c) 10 of 20 reports.

Is probably subject to some degree of selection bias. Certain data
findings—the relative difference, for instance, in the number of
reports (35 of 58) addressing subcategory 6(a); and 6(c), more
information on what doesn’t work than on what does work—may be
the result of what’s available for the meta-evaluation as opposed to
the actual state of play across the whole of the CP investment effort in
building capacities.

Ownership issues are most frequently reported as affecting project
sustainability, and also impinge on recognition of and response to CP
needs (6a and b), and occur at the level of the community and professionals as well as
government institutions.
40
• The meta-evaluation tracks
31% more activity in the
“response” category than in
the “recognition” category.
• Across both “response” and
“recognition” categories,
professionals’ capacity
building saw the highest
programme practices quality,
with 19% more overall
quality than programming
design and execution relating
to communities’ capacity
building.
• Capacity building of
families was not an area of
much emphasis in the
reports.
The numerics here refer to the codes that were assigned to this area of the Protective Environment
Classification System (see Appendix Two).
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
63
Major Actors Are Facilitated in Recognizing Child
Protection Needs
Programmes like the Child Protection Teams are well-designed community based
programmes using various community members to educate on the risks of child migration as
well as other aspects of child rights.
Protecting Street Girls and Preventing Child Migration (Ghana 2003)
Synopsis
Data addressing this theme, “Major actors are facilitated in recognizing child protection needs,” suggest
that raising awareness about rights and needs of vulnerable children is often an afterthought, secondary
to actionable objectives of strengthening the capacity to respond to needs (see 6(b), next section).
A total of 35 reports, 18 of which are rated credible in quality of evidence, address recognition of CP
needs among major actors. The majority of the 18 projects target recognition on the part of
communities and CSOs; smaller numbers target professionals and, fewer still, families.
One might question, however, whether evaluators actively sought data in this category; is evidence of
“recognition” less easily observed and measured than “capacity to respond” (investigated in 6b)?
All 6a reports (total 35) are separated into two groups by quality of evidence (18 are good to excellent;
17 are poor to unacceptable).
FINDINGS
7.1 Investment characteristics that facilitate recognition of child
protection needs
7.1.1 Reaching families via adolescent livelihoods interventions
Promising approaches to adolescent livelihoods interventions are
seen in two studies:

Adolescent Girls’ Livelihood (Bangladesh 2005), Early Marriage
Prevention Program

Protecting Street Girls and Preventing Child Migration (Ghana 2003)
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
64
Six reports show good
outcomes signaling the
importance to awarenessraising of community
involvement in programme
planning and
implementation. Four reports
show good outcomes linking
rights awareness and
community participation.
7.1.2 Reaching families/communities via involvement in programme planning

The importance of community involvement in programme planning and implementation is
shown in six studies: the Community-Based MVP (Most Vulnerable Children) programmes in
Tanzania (2004, 2004 and 2007); Reintegration and Rehabilitation of Children Associated with
Militia (Somalia 2006); Protecting Street Girls and Preventing Child Migration (Ghana 2003);
Adolescent Girls’ Livelihoods/Early Marriage Prevention Program (Bangladesh 2005); and
Psychosocial Rehabilitation of Children Traumatized by Terrorist Violence (Algeria 2003).

Algeria 2003: the difficulty of earning the trust of the community is acknowledged together with
the importance of doing so.

Ghana 2003: “Programmes like the Child Protection Teams are well-designed community based
programmes using various community members to educate on the risks of child migration as
well as other aspects of child rights.”
7.1.3 Reaching communities via capacity building programming
A positive feedback loop whereby needs recognition and capacity-building
programmes reinforce each other is seen in:

Psychosocial Rehabilitation of Children Traumatized by Terrorist
Violence (Algeria 2003): “Developing and supporting spaces for
expression and rehabilitation for children and women has invited
better community participation and involvement and more learning
about community and culture.”

The Improved Mechanisms of Child Protection programmes (Bosnia
and Herzegovina 2003): “… promoting wider civic involvement;
including the involvement of vulnerable population as implementer
rather than the recipient of these projects.”

“Programmes like the Child
Protection Teams are welldesigned community based
programmes using various
community members to
educate on the risks of child
migration as well as other
aspects of child rights.”
Ghana 2003
EXCERPT
Education as a Preventive Strategy against Child Labour (UNICEF/Global 2003): “Some countries
were successful in gaining community support on eliminating child labour and child trafficking
through members’ participation in village committees (Benin), forums (Indonesia), and
community funds (the Philippines).”
7.1.4 Reaching professionals via training and agent-partner linkages

Training programs are cited as improving needs recognition in several studies: Improved
Mechanisms of Child Protection (Bosnia and Herzegovina 2003); Social Work Training
Programme (Cambodia 2005); and Mahzel Project for Social Reintegration and Protection of
Disadvantaged Children as Future Development (Eritrea 2004).

Linkages among agents and participating partners facilitated Lihlombe Lekukhalela (LL) Child
Protectors (Swaziland 2005): “Where strong linkages have been achieved with the police, Rural
Health Motivators, clinic staff, and local schools, there is a high level of reporting and referral of
cases to the relevant partners.”
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
65
7.1.5 Reaching all stakeholders via rights awareness training

Four studies, all grounded in community participation, address the importance of rights
awareness among all stakeholders: Street Children/Street Mother Program (Ethiopia 2003);
Impact Assessment of MVC Support Programme Implementation (Tanzania 2004); Project for
Social Reintegration and Protection of Disadvantaged Children (Eritrea 2004); and LL Child
Protectors (Swaziland 2005).

Ethiopia 2003: “The stakeholders have become more conversant with the nature, extent, and
ways of addressing the phenomenon of street life in the Ethiopian context. The project has been
instrumental in creating awareness among teachers, students, and administrators about the
plight of street children and mothers.”
7.2 Constraints to greater recognition of child protection needs
7.2.1 Abusive relatives not being engaged by community-based project

Abusive relatives not being engaged by community-based project, LL Child Protectors program
(Swaziland 2005): “*The+ pattern is that abuse mainly takes place at home in the form of physical
abuse most often at the hands of brothers, fathers, and stepmothers. To this end there appears
to be a need to engage with these groups (fathers, brothers, stepmothers, and absentee
parents) in particular.”
FINDINGS
7.2.2 Communities/CSOs need more intense training

The need to improve or intensify training was cited in two communitybased studies: MVC Support Programme Implementation (Tanzania
2004) and LL Child Protectors program (Swaziland 2005).

LL Child Protectors program (Swaziland 2005): “Training should be preplanned as a series of interventions in each community rather than
one or two contacts only. More sessions of shorter periods are
required.”
Constraints to awareness
raising include over-reliance
on volunteers, "one-off"
trainings, inaccurate
situation analyses, and
failure to engage the actual
offenders.
7.2.3 Over-reliance on volunteers

Over-reliance on volunteers was cited in three studies: volunteer workers - Protecting Street
Girls and Preventing Child Migration (Ghana 2003); and/or trainers - LL Child Protectors
(Swaziland 2005); or services - Child Protection Network (Cambodia 2004).
"Experience in peer education
programmes, which are largely
voluntary, suggests that
volunteerism may be more
costly than anticipated."
Ghana 2003
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
66
7.2.4 Weak front-end needs assessments

EXCERPT
Inadequate assessments interfere with needs recognition in two studies:
o Child Protection Network (Cambodia 2004): Failure to carry out assessment of child
needs and protection issues: “…there is a lack of information on children’s issues which
would ensure that interventions are focused and relevant.”
o MVC Programme and Potential for Scaling Up (Tanzania 2007): Failure to follow up
initial assessments.
7.2.5 Lack of community involvement in programming

Lack of community involvement in programming impedes CP awareness in three studies:
Improved Mechanisms of Child Protection (Bosnia and Herzegovina 2003); Protecting Street
Girls and Preventing Child Migration (Ghana 2003); and Community Based Reintegration (Sierra
Leone 2006).

Community Based Reintegration (Sierra Leone 2006): “There was little or no consultation with
the communities in terms of program design.”
7.2.6 Professionals’ training is inadequate

Inadequate training programs:
o
Social Work Training Programme (Cambodia 2005): “A significant percentage of trainees
are coming out of the course still believing that domestic
“Training should be previolence is the fault of the woman. More training materials
planned as a series of
need to be developed.”
interventions in each
o Prevention of Child Abandonment in Maternities (Romania
community rather than one or
2003): “Prior to the official launch of the project, the
two contacts only. More
maternity staff should be trained on issues concerning child
sessions of shorter periods are
development, the rights and needs of the child, in order to
required.”
better understand the sense and importance of the child
Swaziland 2005
abandonment prevention activity.”
7.2.7 Poor coordination among institutions and agencies
EXCERPT
o Thematic Evaluation of UNICEF’s Contribution to Juvenile
Justice System Reform (CEE/CIS 2006): “The good cooperation established with the
relevant university departments should be further enhanced.”
o Street Children and Street Mothers (Ethiopia 2003): “Advocacy and social mobilization
activities were not sufficient and sustained due to the pulling out of many of the
implementing partners; inadequate coordination on advocacy and social mobilization
among partners…”
7.2.8 Lessons lost due to a weak evidence base

All “recognition capacities” reports (total 35) are separated into two groups by quality of
evidence (18 are good to excellent; 17 are poor to unacceptable).
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
67
Strengthened Capacities for Responding to Child
Protection Needs
Village committees, who provided bottom-line support for the most vulnerable children …
“reported cases which they could not manage; this arrangement proved to be cost effective
and also empowered communities in addressing the issue of orphans at a grass root level.”
Project for Social Reintegration and Protection of Disadvantaged Children (Eritrea 2004)
Synopsis
Data addressing “Strengthened capacities for responding to child protection needs” suggest that “topdown” (vs. “bottom-up”) strategies predominate in capacity-building CP programmes. Among reports
showing instances of facilitation of capacities to respond (“What works,” shown below), 4
programmes—all dealing with street children—target families. In comparison, 12 programmes target
capacity of communities and 13 target professionals. A similar pattern in “What doesn’t work” (families
are targets of 1 programme, communities of 17, and professionals of 21) indicates that families are
rarely targeted, despite the fact that those closest to the child are generally family members.
Community involvement in various phases of program planning, design and
implementation appears to be a facilitating factor for recognition of and
response to child protection needs by families and communities. Yet six
studies suggest that over-reliance on volunteer workers, trainers, and/or
services impedes CP recognition and response (6(a) and (b) combined). How
can project planning and funding balance the efficacious effects of community
involvement with volunteer fatigue?
All 6b reports (total 52) are separated into two groups by quality of evidence
(25 good to excellent; 27 poor to unacceptable).
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
68
FINDINGS
Data addressing
“Strengthened capacities for
responding to child
protection needs” suggest
that “top-down” (vs.
“bottom-up”) strategies
predominate in capacitybuilding CP programmes.
Families are rarely targeted;
most investments target
professionals.
7.3 Investment characteristics that facilitate response capacities
7.3.1 Providing relevant supports (e.g., crèches) to families
FINDINGS

Building a Sustainable Street Children Project (Laos 2006): “Substantial
support has been provided to the families of street children in an
attempt to address the factors that led children onto the streets.”

Protecting Street Girls and Preventing Child Migration (Ghana 2003):
“…one of the most innovative forms of programming in the RPP
portfolio are the crèches. These programmes are not high cost, but
have the potential for ongoing contact with a large number of women
and children.”

Bam Emergency Programme (Iran [Wiles et al] 2007): “UNICEF supported a family reunification
programme with the Social Welfare Office of the government which was relevant, appropriate,
and effective and resulted in the majority of children without primary care givers living in
extended family-based care… This was probably one of UNICEF’s greatest achievements in its
work in Bam.”
What works for CP response
capacities: relevant supports
(e.g., creches, basic needs),
village-level CP committees,
cross-departmental/sectoral
trainings, hands-on social
work experience (16 reports).
7.3.2 Rooting child protection efforts in the community

Evidence of success in four programmes where child protection is community-based:
o Most Vulnerable Children (MVC) Support Programme Implementation (Tanzania 2004):
“MVC committees operate in villages which have organizational structures in the form
of village governments. Generally the village government works well with the MVC
committee in the village… Many objectives of the programme have been met because of
efforts done by these committees.”
o Potential for Scaling Up MVC Programmes (Tanzania 2007): “Response systems
successfully established at all levels, and particularly at community level, to facilitate
provision of services to most vulnerable children.”
o Project for Social Reintegration and Protection of Disadvantaged Children (Eritrea 2004):
Village committees, who provided bottom-line support for MVCs “…reported cases
which they could not manage; this arrangement proved to be cost effective and also
empowered communities in addressing the issue of orphans at a grass root level.”
o Community Based Reintegration (Sierra Leone 2006): Child Welfare Committees
perform “…house to house sensitization of child rights violations in the community,
registering vulnerable children and identifying a few who were able to get assistance to
go to school.”
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
69
7.3.3 Community outreach makes a difference

“Irrespective of the approach
Success related to community outreach is demonstrated in three
adopted, the key [to the
programmes:
programme's success] is that
o Social Work Training Programme (Cambodia 2005): Four
the work is clearly grounded at
projects engaging disabled people in developing greater selfcommunity level and involves
efficacy. “Irrespective of the approach adopted, the key is
substantial capacity building of
that the work is clearly grounded at community level and
local actors.”
involves substantial capacity building of local actors.”
Cambodia [Social Work Training
o Child Soldier Rehabilitation and Reintegration (Somalia
Programme] 2005
2006): Reaching out to local community through outreach
and a focus on training of trainers (TOT) and outreach
EXCERPT
workers.
o Prevention of Child Abandonment in Maternities (Romania 2003): Mobilizing
community resources for the prevention of child abandonment and for keeping the child
together with his parents.
7.3.4 “Indigenized” models are effective

Two studies suggest “indigenized” models are more effective in local communities:
o Child Justice (Zambia 2005): “It is not sufficient to expose the functionaries in the
system to information and examples. It is required that the model be ‘indigenised’ with
its own set of local performance procedures and indicators.”
o LL Child Protectors (Swaziland 2005): “…communities request help in forming networks
that can help to protect children more effectively using existing structures such as the
traditional leadership.”
7.3.5 Facilitating agency-community cooperation

Benefits of cooperation between agencies and community workers: Prevention of Child
Abandonment (Romania 2003): “The cooperation with the CDPRC (County Department for the
Protection of the Rights of the Child) and the community healthcare worker supports the
sustainability of the interventions even after the departure of the mothers…”
7.3.6 Training is a vital investment for building professionals’ response capacities

Positive effects of UNICEF-funded training are cited in numerous studies:
o Child Justice (Zambia 2005): “A body of knowledge and common understanding
between magistrate and social welfare officers developed over time regarding
appropriate sanctions. This was particularly the case where the role players concerned
participated in the same training.”
o Psychosocial Rehabilitation of Children Traumatized by Terrorist Violence (Algeria 2003):
“A large number of psychologists, psychiatrists, teachers, school counselors, and general
practitioners have been reached by the training component. The outcome is the
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
70
o
o
o
o
emergence of an informal network of professionals active in the area of psychosocial
care for children in general and with those affected by violence in particular.”
Improved Mechanisms of Child Protection (Bosnia and Herzegovina 2003): “The MMBs
[Municipality Management Boards] had by the time of the Key sources interviews in
autumn 2004 been established in all five core municipalities, which is a significant
achievement… This work was facilitated by the management training provided … and is
in itself a novelty for the municipal planning systems.”
Thematic Evaluation of UNICEF’s Contribution to Juvenile Justice System Reform
(UNICEF 2006): “UNICEF has persuasively demonstrated its
“The MMBs *Municipality
comparative advantage in providing valuable capacity
Management Boards] had …
building of the actors involved in juvenile justice reform.
been
established in all five core
Most capacity building measures have consisted either of
municipalities, which is a
training, or the constant exchange of information and ideas
significant achievement… This
that can be seen as a sort of informal mentoring.”
work was facilitated by the
Building a Sustainable Street Children Project (Laos 2006):
management training provided
“Through a combination of training, mentoring and on-the… and is in itself a novelty for
job experience… the level of capacity achieved within Peuan
the municipal planning
Mit is particularly impressive given the lack of professional
systems.”
social expertise in the country.”
Bosnia and Herzegovina 2003
Psychosocial Interventions in Gujarat (India 2002): There
were myriad inconsistencies, but “Most teachers who were
EXCERPT
interviewed remembered the training process as unique in
its delivery and content. The key session or process that was
remembered was the “anxiety sharing” session wherein they were able to express what
they had faced in the wake of the earthquake.”
7.3.7 Specialist capacity building has clear value

Social work experience gained in several programmes contributed to capacity-building:
o Deinstitutionalisation and Alternative Placement of Children (Belarus 2004):
Establishment of a professional community of specialists and a network of institutions
for prevention of social orphanage.
o Child Protection Network (Cambodia 2004): Specialist capacity building in sexual abuse
cases.
o Improved Mechanisms of Child Protection Programmes (Bosnia and Herzogovina 2003):
“As a result of the MMBs *Municipal Management Boards+ and other outputs, relevant
stakeholders know one each other better and are more ready to work together.”
7.3.8 Advocacy and mobilization supports professionals’ response capacities

Thematic Evaluation of UNICEF’s Contribution to Juvenile Justice System Reform (UNICEF 2006):
“…growing support for and participation in juvenile justice reform by national and local NGOs is
due in very large part to UNICEF’s advocacy and mobilization efforts.”
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
71
7.3.9 Giving the same trainings to all stakeholders:

Bam Emergency Programme (Iran *Wiles et al+ 2007): “The effectiveness of the project is largely
due to the basic techniques that were taught to people in the counseling sessions. Teachers
interviewed said that they had also used the techniques for relaxation and coping with
difficulties themselves. Many parents believed that the briefing sessions helped them better
understand the behaviour of their children.”
7.4 Constraints to strengthening response capacities
7.4.1 For young people, unsupportive parents are a constraint

Protecting Street Girls and Preventing Child Migration Programmes (Ghana 2003): “Almost all of
the programmes mentioned lack of parental commitment or unsupportive parents, as the main
barriers in implementing programmes.”
FINDINGS
7.4.2 Families’ economic needs are unmet

Reintegration Programme for Children Associated with Fighting Forces
(Liberia 2007): “Based upon the lessons learned 10 years ago … where
many children left their parents again after placement due to
economic constraints, it was advised to provide income generating
support to the parents of the younger children. This recommendation
was not taken. Funding could not enable this to happen.”
Constraints to CP response
capacities: disabling
economic environment,
inability to meet basic needs
(e.g., water); over-reliance on
unpaid volunteers and on
non-professionals.
7.4.3 For communities/CSOs, an over-reliance on unpaid volunteers

Three studies show over-reliance on unpaid volunteers:
o Child Protection Networks (Cambodia 2004): One district
"Members of the committees
working group said that in Cambodia “a volunteer is a
reported problems of the size of
person that volunteers to do something, but still expects
their villages and the workload
some incentives for time spent on the tasks he or she is
—work for which they were
undertaking.” As a consequence the assessment found
considered volunteers and
unpaid work (interviewing of children) to be of low quality.
therefore without
o Potential for Scaling Up of MVC Programme (Tanzania 2007):
compensation."
“Members of the committees reported problems of the size
Tanzania 2007
of their villages and the workload—work for which they
were considered volunteers and therefore without
EXCERPT
compensation.”
o Social Work Training Programme (Cambodia 2005): “Ability to sustain motivation and
commitment among volunteers, as well as maintain and strengthen their technical skills
and capacity, poses an on-going risk to projects relying on volunteer input. In this
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
72
context, provision of allowances and training need to be considered essential core
project costs.”
7.4.4 For communities/CSOs, an over-reliance on non-professionals

Three studies sound cautionary notes about relying on non-professionals:
o Child Protection Networks (Cambodia 2004): “The sensitive nature of sexual abuse
requires special support and this expertise is not widely available. The evaluation team
discovered that the majority of the girls are visited by men for follow up.”
o LL Child Protectors (Swaziland 2005): “In some areas the LL members do not know the
limits of their powers nor the extent to which they are protected in intervene in case of
abuse.”41
o Assessment of Potential for Scaling Up the MVC Programme (Tanzania 2007): “This form
of programme delivery puts much of the onus on local committee structures which
themselves frequently need strong oversight.”
7.4.5 For communities/CSOs, material resource constraints

Material resource constraints are reported by two studies:
o Community Based Reintegration Programme (Sierra Leone 2006): “All CWCs [Child
Welfare Committees] visited complained that the absence of mobility, funding, and
supplies significantly hampered their ability to perform their duties.”
o Assessment of Potential for Scaling Up the MVC Programme (Tanzania 2007).
7.4.6 For communities/CSOs, inadequate training

Inadequate training (three reports):
o Assessment of Potential for Scaling up the MVC Programme (Tanzania 2007):
“…difficulty maintaining trained facilitators.”
o Building a Sustainable Street Children Project (Laos 2006): “Less training has been
conducted with local government officials and capacity for understanding and
responding to street children issues was found to be correspondingly lower.”
o Community Based Reintegration Program (Sierra Leone 2006).
7.4.7 For communities/CSOs, vertical communication breakdowns

41
Vertical communication and/or coordination breakdown between administrators or
professionals and local agents (three reports):
o MVC Programme Assessment (Tanzania 2004): “The MVC coordination/facilitation team
has not been receiving or provided with support from the district leadership due to
communication breakdown. Communication breakdown resulted from among others,
improper handing over of the MVC program from the previous District Development
Officers (PDOs) to the incumbent.”
o Assessment of Potential for Scaling Up the MVC Programme (Tanzania 2007): Lack of
coordination.
The writer probably meant to say “…extent to which they are expected to intervene in case of abuse” (underlining added).
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
73
o
Community Based Reintegration Program (Sierra Leone 2006): Coordination between
two implementing partners was weak “…when it came to distributing items and
managing caseload… Children were confused as to which agency was carrying out what
component and often came to the wrong one...” Also, community groups who received
less attention from a social worker tended to lose interest and motivation.
7.4.8 For communities/CSOs, failure to build public awareness

Failure to build public awareness of services (two reports):
o Community Based Reintegration Program (Sierra Leone 2006): “Interviews with children
demonstrated a lack of involvement of CWCs [Child Welfare Committees] in the day-today lives of children. Only 38% of children surveyed (both beneficiaries and controls)
reported knowing of the existence of the CWC. It is clear that there is a disconnect
between children’s awareness of the CWC and its intended functions.”
o Prevention of Child Abandonment Programme (Romania 2003): Evaluator suggests
marking “…the launch / beginning of the project with a ceremony in order to increase
awareness of the intervention model used, the values it is built upon, and the content of
the services provided.”
7.4.9 Other constraints for communities/CSOs: unmet basic needs, ownership confusion

Other serious constraints reported by single evaluations:
o Communities express the need for basics (water, food) preliminary to embracing child
protection issues: Impact Assessment of MVC Support Programme Implementation
(Tanzania 2004): “The welfare of the MVC has worsened in the past two years because
even the neighbors who were giving them food cannot do so now because they have
food shortage themselves.” The same assessment also indicated that communities were
impeded from responding to MVC needs because of the HIV/AIDS epidemic.
o CSOs exist only to collect UNICEF money: Child Protection, Capacity Building, Monitoring
and Advocacy (Somalia 2005): “Though they were reported to have transacted some
activities, most of them [CSOs] confessed to have come together at the behest of
UNICEF and not in order to address a need they had perceived to exist in the Children’s
or Human Rights Sector and which required collective action. None of the Networks was
found to have taken part in any advocacy or openly activist activities necessary for the
challenge of impunity in the violation of children’s rights.”
o Ownership issues: Assessment of Potential for Scaling Up the MVC Programme
(Tanzania 2007): “…in many areas, the programme continues to be viewed as an
externally driven programme with consequent lack of local commitment among
stakeholders and facilitators at all levels and mobilisation of community participation is
limited.”
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
74
7.4.10 For professionals, myriad training inadequacies

15 studies show myriad training inadequacies:
o
o
o
o
o
o
FINDINGS
One-off training inadequate: Psychosocial Rehabilitation of
Training-related problems
Children Traumatized by Terrorist Violence (Algeria 2003);
are cited as constraints to
Mahzel Project for Social Reintegration and Protection of
outcomes in at least 15 wellDisadvantaged Children (Eritrea 2004); Social Work Training
validated reports.
Programme (Cambodia 2005).
Need opportunities to apply theory acquired during training: Social Work Training
Programme (Cambodia 2005); Mahzel Project for Social Reintegration (Eritrea 2004).
Specific training needs: Building a Sustainable Street Children Project (Laos 2006)—need
for more general training and specific request to address drug use; Improved
Mechanisms of Child Protection programmes (Bosnia and Herzegovina 2003)—
Municipality Management Board members wanted “…training specially designed for
them, more focused on management and development issues.”
Psychosocial Interventions in Gujarat (India 2002)—“the training process for teachers
was riddled with difficulties.”
Lack of relevancy of HIV/AIDS education: Protecting Street Girls and Preventing Child
Migration (Ghana 2003)—“Simply describing the modes of transmission does not take
into account the reasons why young women are having sex. A large proportion of girls
interviewed had experienced forced sex, an experience that is likely under-reported.”
Training of trainers inadequate: Deinstitutionalisation and Alternative Placement of
Children (Belarus 2004); Social Work Training Programme (Cambodia 2005);
Reintegration and Rehabilitation of Children Associated with Militia (Somalia 2006).
o
Language barriers in training: Psychosocial Rehabilitation of Children Traumatized by
Terrorist Violence (Algeria 2003)—“…young psychologists
“Less training has been
complained that they could not understand some of the
conducted with local
training material and lectures given in French. They were
government
officials and
embarrassed however to show that to the trainer, since
capacity for understanding and
language is a marker of social status.”
responding to street children
issues was found to be
7.4.11 For professionals, too much staff turnover
correspondingly lower.”
 Staff turnover impedes capacity to respond to CP needs (three
(Laos 2006
reports):
o Child Justice Reform (Zambia 2005): “Staff constraints and
EXCERPT
turnover continue to limit the ability and consistency of a
child-friendly court.”
o Street Children/Street Mother Program (Ethiopia 2003): High level of turnover affecting
advocacy efforts, also insufficient monitoring and evaluation of performance and
impact.
o Child Protection, Capacity Building, Monitoring, and Advocacy Project (Somalia 2005):
“… it is expected that all the relevant cadres of Child Protection Officers specifically and
UNICEF generally would be familiar with all the elements and stages of the project. This
however was not always the case on the ground.”
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
75
7.4.12 For professionals, lack of human resources and expertise

Lack of human resources and expertise (four reports):
o UNICEF’s Support to Mine Action (Global 2005): Noted need for specific expertise in
Victim Assistance.
o Deinstitutionalisation and Alternative Placement of Children (Belarus 2004): “The
number of employees is determined by local authorities; therefore, the workload on
specialists is very high and varies by the districts.”
o Prevention of Child Abandonment (Romania 2003): “The social workers from the child
abandonment prevention cabinet in the maternity units do not have the competencies
needed in order to intervene in so-called ‘decided child abandonment’ cases.”
o Education as a Preventive Strategy against Child Labour (UNICEF/Global 2003): “The
proposed internal capacity-building plan, which appeared to be practical and necessary,
never got off the ground, as it required human and financial resources and
commitment.”
7.4.13 For professionals, lack of material resources/transport

Lack of material resources, transport (three reports:
o Deinstitutionalisation and Alternative Placement of Children (Belarus 2004): Financial
resources cannot provide technical support to the centres or ensure “mobility of
specialists to travel to remote rural areas.”
o Community Based Reintegration (Sierra Leone 2006): “All CWCs *Child Welfare
Committees] visited complained that the absence of mobility, funding, and supplies
significantly hampered their ability to perform their duties.”
o Prevention of Child Abandonment (Romania 2003): Social workers have no financial
means to check on the situation of children after birth: “It is not known whether the
child is well taken care of, or whether he or she is neglected. The social worker does not
have the financial means to check on the situation of the child, even by telephone,
through the clinic or the city hall.”
7.4.14 For professionals, failure to integrate approaches

Failure to integrate approaches to the implementation of goals and tasks:
o Deinstitutionalisation and Alternative Placement of Children (Belarus 2004): “There was
no uniform understanding of the principles of selection of children for social shelters,
nor there was any integrated approach to the implementation of goals and tasks set
before such institutions.”
o Education as a Preventive Strategy Against Child Labour (UNICEF/Global 2003): “No
systematic capacity-building component was included in the strategic guidance to
countries…”
7.4. 15 Lessons lost due to a weak evidence base

All “Response capacities reports” (total 52) are separated into two groups by quality of evidence
(25 good to excellent; 27 poor to unacceptable).
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
76
Capacity Development Is Becoming Systematized and
Integrated, not Ad Hoc
A body of knowledge and common understanding between magistrate and social welfare
officers developed over time regarding appropriate sanctions. This was particularly the case
where the role players concerned participated in the same training.
Child Justice Reform (Zambia 2005)
… the problems plaguing the implementation of the pilot programme persist: in many areas,
the programme continues to be viewed as an externally driven programme with consequent
lack of local commitment among stakholders and facilitators at all levels and mobilisation of
community participation is limited.
MVC Impact/Scaling Up Assessment (Tanzania 2007)
Synopsis
Out of 59 reports, 41 addressed systematization and/or integration of the programme under evaluation.
Note therefore, that nearly 20% of the evaluations failed to even address the sustainability of capacity
development. And, of the 41, only 24 are rated with credible quality of evidence.
Twenty of the credible reports described some instance of failure to systematize (or some impediment
to sustainability). Fourteen reports suggested some positive instance of systematization and/or
integration leading to sustainability in capacity development. 42 These figures seem to indicate that
significantly more of the projects evaluated are failing to become systematized than those that are. Is
this truly the case, or are the figures artifacts of an evaluation methodology or culture that emphasizes
shortcomings rather than factors leading to success?
Not surprisingly, evaluations suggesting sustainability of capacity development
frequently attribute the facilitation of recognition of CP needs among
stakeholders (factors in 6a) and the strengthening of capacity of major actors
to respond (6b, usually professionals). Other factors said to be contributory
include clear project aims and strategies and effective communication and
linkages between participants and partners.
All 6c Reports (total 41) are separated into two groups by quality of evidence
(24 good to excellent; 17 poor to unacceptable).
42
FINDINGS
The data suggest that
significantly more
investments remain ad hoc
than those that are
systematic. Is this truly the
case, or is it an artifact of
negative expectations as a
culture of evaluation
reporting?
Some of the evaluations report both a positive (“Works”) and negative (“Doesn’t Work”) instance. The same is true for 6a
and 6b.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
77
In summary:
What works—positive impacts on sustainability
 Effective CP awareness raising, establishing a cohort of trained Child Protection professionals,
and local ownership of programs (found in 14 reports, credible evidence ratings).43
What is not working—negative impacts on sustainability
 Inadequacies of training programs (9 projects), ownership issues (4), lack of commitment to
funding, poor coordination, unclear goals and strategies (in total, 20 reports with credible
evidence).44
7.5 Investment characteristics that facilitate systematic capacity development
7.5.1 Integrating capacities development with awareness raising
Successful building of awareness and recognition of CP needs is linked to
systematization and/or integration of capacity development (three reports):
FINDINGS
What works—building blocks
of consistency: effective CP
awareness raising,
establishing a cohort of
trained Child Protection
professionals,local ownership
of programs (14 reports).

Rehabilitation/Re-integration of Mine Victims and People with
Disabilities (Cambodia 2005): Concerted efforts to ensure the
sustainability of project outcomes have involved “…raising awareness
of disability and the rights of disabled people, building self-efficacy
among primary care givers and groups of disabled people, mobilizing
local authorities and commune councilors in support of responses, and
strengthening referral linkages with specialized services.”

Evaluation of Street Children and Street Mother Program (Ethiopia 2003): “Huge strides made in
consciountization *sic+… the awareness created has been instrumental in drawing the national
attention to the phenomenon.”

Thematic Evaluation of UNICEF’s Contribution to Juvenile Justice System Reform (UNICEF 2006):
“Within the relevant Ministries and other sectors, there are often departments or groups of
professionals that do believe that juvenile justice needs to be recognized as a priority. Working
with them can lead higher authorities to recognize this and make a commitment to juvenile
justice reform.”
7.5.2. Professionals’ capacity building integrated into long term CP programming
Strengthening of capacities of professionals is linked to systematization and/or integration of CP
programming:
43
Another 12 reports proffer evidence of positive systematic capacities building efforts, however, the evidence base in these
was insufficient. Investments addressing the needs of street working girls, life skills, and harmful traditional practices, were
among these 12 reports.
44
Another 15 reports proffer evidence of negative impacts on sustainability but these have an insufficient evidence base.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
78

Long-term results of capacity-building of professionals in the juvenile justice reform projects
reviewed in Thematic Evaluation of UNICEF’s Contribution to Juvenile Justice System Reform
(UNICEF 2006):
o “The ultimate test of any programme on the rights of the child is how it impacts the lives
of children and there is some evidence that interventions that have taken place in
Romania, Serbia, and Tajikistan have had a positive impact on the lives of hundreds of
children.”
o “Even in Montenegro, where very few changes in law and policy have reached the point
where they are being implemented, there is evidence that the involvement of juvenile
justice and law enforcement professionals in the process of reform has had a positive
effect on the way they deal with children.”
o “UNICEF has persuasively demonstrated its comparative advantage in providing valuable
capacity building of the actors involved in juvenile justice reform. Most capacity building
measures have consisted either of training, or the constant exchange of information and
ideas that can be seen as a sort of informal mentoring.”
7.5.3 Establishing a cohort of trained child protection professionals
Importance of establishing a cohort of trained CP professionals:
o Deinstitutionalisation and Alternative Placement of Children (Belarus 2004): “An
important outcome of all activities has been the establishment of a professional
community of specialists who have special knowledge and skills to use modern
technologies of work… Today in Belarus, the existence of the substitution families does
not depend on the UNICEF funding.”
o Psychosocial Rehabilitation of Children Traumatized by Terrorist Violence (Algeria 2003):
“What is emerging as a result of interventions in repeated crises in the Algerian context
is an informal network of professional mental health workers.”
o Thematic Evaluation of UNICEF’s Contribution to Juvenile Justice System Reform
(UNICEF 2006): “Training on child rights has been institutionalized for judges,
prosecutors, social workers who work in juvenile justice, and police officers.”
7.5.4 Developing local ownership
Local sense of ownership is linked to integration of CP capacity development
(two reports):

Protecting Street Girls and Preventing Child Migration (Ghana 2003):
“This (success) is perhaps because the community itself has played
an integral role in the development of the programme and sees it as
consistent with community goals and values. Similarly, the Child
Protection Teams are a community-based initiative, largely designed
by the communities themselves…”

Rehabilitation/Re-integration of Mine Victims and People with
Disabilities (Cambodia 2005): “Strategies for ensuring sustainability
… are contingent of underlying assumptions about community or
government buy-in… sustainability will be determined by how
“This (success) is perhaps
because the community itself
has played an integral role in
the development of the
programme and sees it as
consistent with community
goals and values. Similarly, the
Child Protection Teams are a
community-based initiative,
largely designed by the
communities themselves…”
Ghana 2003
EXCERPT
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
79
successful the partners are in guiding and supporting the transition of project activities to either
community or government actors.”
7.5.5 Effective cooperation and division of labor between institutions
Effective cooperation and division of labor between institutions (two reports):

Deinstitutionalisation and Alternative Placement of Children (Belarus 2004): “Each Oblast of
Belarus has developed its own mechanism of interdepartmental cooperation for prevention of
family problems and social orphanage… Socially important outcomes: Sustainable development
of the foster families; and Cooperation between concerned departments, institutions and
organisations at the municipal and regional levels in the field of the organisation of prevention
of social orphanage.”

Mahzel Project for Social Reintegration and Protection of Disadvantaged Children (Eritrea 2004):
“The clear understanding of the roles and responsibilities of each ministry and the smooth
“transfer” of activities … has facilitated the implementation of Mahzel activities without
disrupting the existing infrastructure.”
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
80
7.6 Constraints to systematizing capacities development
7.6.1 Training is limited to a one-time event (“one off”)
FINDINGS
A recurrent theme is that training programs need to be extended, with followup to ensure sustainable success (six reports):
Constraints to sustainable
capacity development:
o Reintegration Programme for Children Associated with
trainings without follow up,
Fighting Forces (Liberia 2007): “There is an urgent need to
ownership confusion,
build a sustainable system that will ensure that business
dependency syndrome,
trainers remain in their contracts to provide follow-up training
unpredictability of material
and mentoring…”
supports, poor programme
o Psychosocial Rehabilitation of Children Traumatized by
clarity, instability of conflictTerrorist Violence (Algeria 2003): “A one-week training course
affected areas.
on psychotrauma is hardly sufficient to answer all needs of
mental health workers and provide them with necessary tools
to intervene efficiently with traumatized children.”
o Grassroots Peace-Building Project (Sudan 2004): “Workshops are not usually enough to
give people and organisations the skills, connections, and resources to do new things…
The one-off training on mainstreaming is a hostage to staff turn-over: none of the
people trained in peace-building in the Upper Nile MOE [Ministry of Education] are still
there.”
o Training inadequacy is also reported in: Community Based Reintegration (Sierra Leone
2006); Bam Emergency Programme (Iran 2007); Social Work Training Programme
(Cambodia 2005).
7.6.2 No quality control on trainings
Lack of quality control:
o Reintegration and Rehabilitation of Children Associated with Militia (Somalia 2006):
“Training manuals were designed by each trainer and no system could be identified how
these manuals were prepared, counter-checked, and how they were ultimately
approved… “
FINDINGS
Nine reports find training7.6.3 Trainees have no practical work experience
related constraints, e.g., “The
New trainees require work experience in order to connect theory with practice one-off training on
(two reports):
mainstreaming is a hostage
o Social Work Training Programme (Cambodia 2005): “Longerto staff turn-over."
term impact on competence depends on the Social Workers
regularly practising their skills (and having the logistical and material support to be able
to do so)… Professionally, there is need for closer, long-term technical supervision and
monitoring of Social Workers…”
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
81
o
Community-Based Child Protection Programme (Somalia 2006): “The current approach
of mentoring and ‘on the job training’ lacked the structure and resources to develop a
satisfactory foundation in child protection to be able to carry out their tasks.”
7.6.4 Unmet needs and ownership confusion are not addressed45
Ownership issues impact negatively on sustainability (four reports):

Evaluations of the community-based Most Vulnerable Children Programmes in Tanzania (2004,
2004, 2007) note the persistence of perceptions that MVC Support is
externally-driven:
FINDINGS
Six reports find ownership
o MVC Support Programme Implementation (Tanzania 2004):
confusion and dependency
“…when it comes to ownership there is mixed feeling and
syndrome as constraints to
many people believe that the MVC programme is a “UNICEF
good outcomes, e.g., "in
project”… Maximum participation of all community members
many areas, the programme
should address the problem of sense of ownership and
continues to be viewed as an
increase impact of the process.”
externally driven…"
o MVC Impact/Scaling Up Assessment (Tanzania 2007):” …the
problems plaguing the implementation of the pilot programme persist: in many areas,
the programme continues to be viewed as an externally driven programme with
consequent lack of local commitment among stakeholders and facilitators at all levels
and mobilisation of community participation is limited.”
o Community-Based Child Protection Network (Cambodia 2004): “All participants in the
CPN receive per diems for each activity conducted. While this is common in Cambodia, it
raises concerns regarding the sustainability of the approach as well as the ownership of
the programme.”
o
Bam Emergency Programme (Iran *Wiles et al+ 2007): “It is believed that the recent
changes in the management and staffing of MoE [Ministry of
Education] in different levels can affect the “ownership” of
"... a regular request was made
the education programme by the government negatively
to
provide incentives to the CSH
since the new people need time and motivation to feel
[Community Social Helper] in
attached to the achievements of the project and follow up
the form of rice or cash. One
actions to fill in the gaps.”
district working group said that
in Cambodia “a volunteer is a
7.6.5 Dependency syndrome
person that volunteers to do
Dependency and lack of commitment to funding post-UNICEF (two reports):
something, but still expects
 Social Work Training Programme (Cambodia 2005): “While MoSVY
some incentives for time spent
[Ministry of Social Affairs, Veterans and Youth Rehabilitation]
on the tasks he or she is
commitment to the Social Work Training Programme has been
undertaking.”
encouraging, e.g., by providing human resources and the offer of a
Cambodia 2004
site for a Training Centre, there has been little financial commitment
and the programme remains heavily dependent on UNICEF.”
EXCERPT

45
Improved Mechanisms of Child Protection Programmes (Bosnia and
Herzegovina 2003): “Ownership of the method/model must be passed to the local stakeholders
but with it must come the responsibility of financing it, at least partially.”
This topic is addressed in depth in Chapter 11, Section 4, “Cross-cutting Issues/Intersectorality.”
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
82
7.6.6 Confusion among partners
Division of labor and coordination problems between departments and
institutions, as well as failure to involve critical partners, may decrease
likelihood of sustainable success (four reports):
“All participants in the *Child
Protection Network] receive per
diems for each activity
conducted. While this is
common in Cambodia, it raises
concerns regarding the
sustainability of the approach
as well as the ownership of the
programme.”

Lihlombe Lekukhalela (LL) Child Protectors (Swaziland 2005):
o “In some areas lack of coordination between various child
protection interventions may cause confusion as people may
duplicate activities or leave gaps in the belief that others are
responsible…”
o “There is some criticism from national level stakeholders
(Cambodia 2004
that their field officers have not been properly integrated
into the initiative so they can work hand in hand with the LL.
EXCERPT
This will inevitably lead to a duplication of resources or
competition for resources.”
o “The partnership with the police has to be strengthened… also partnerships with clinic
and health center staff.”

Deinstitutionalisation and Alternative Placement of Children (Belarus 2004): “…further attention
should be paid to differentiation of powers between the services and specialists, clear
identification of the organisation which orders work with every specific family; development of
criteria for the completion of the specific work by various specialists.”

Psychosocial Interventions in Gujarat (India 2002): “The focus is on capacity building without
significant involvement of the Department of Primary Education for implementation.”

Thematic Evaluation of UNICEF’s Contribution to Juvenile Justice System Reform (UNICEF 2006):
“In none of these countries has a concerted effort to involve the legal profession (i.e., lawyers)
been made. The only exception can be found in Tajikistan, where UNICEF supported a little
project providing legal services to juvenile offenders through a national NGO.”
7.6.7 Inadequate material support
Inadequate material support (two reports):

Improved Mechanisms of Child Protection programmes (Bosnia and
Herzegovina 2003): Most of the CSW directors “…reported in interview
that the equipment received did not really meet their requirements
and therefore can not lead to enhanced capacities. In some cases
request for particular materials/supplies, which would have had a
better chance of enhancing capacity, were not granted because they
were not in line with UNICEF procurement/administration rules.”

Child Protection Networks (Cambodia 2004): “Material assistance …
was provided on a one-off basis only. There is a lack of
institutionalising support for children that are in need of long term
support and assistance. The impact of follow up services provided are
not well documented, but there are indications that the assistance
provided through these visits is not effective.”
7.6.8 Unclear project aims and strategies
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
83
FINDINGS
Outcomes are constrained by
unclear project aims and
strategies, e.g., “The
evaluation recommends the
Programme adopt a series of
guiding principles and
indicative strategies that will
not only help better define its
strategic framework, but
provide clearer guidance to
potential partners as to what
is expected in terms of
project design (Cambodia
[Disabilities] 2005).
Lack of clarity of project aims and strategies (four reports):

UNICEF 's Support to Mine Action (UNICEF/Global 2003): “UNICEF’s mine action support is not
fully tailored to match UNICEF’s priorities, strategy, or programmes.”

Rehabilitation/Re-integration of Mine Victims and People with Disabilities (Cambodia 2005):
“The evaluation recommends the Programme adopt a series of guiding principles and indicative
strategies that will not only help better define its strategic framework, but provide clearer
guidance to potential partners as to what is expected in terms of project design. Equally,
resolving issues with how country level programming is translated into complementary and
coordinated responses on the ground will significantly enhance program impact.”

Improved Mechanisms of Child Protection Programmes (Bosnia and Herzegovina 2003): “The
best test to the effectiveness and efficiency of these boards will be the revision of the 2yr plans.
…can *they+ revise the development plans so they really reflect, not only the local needs but also
the changes in the political and legislative environment of social protection sector. This element
of the program, given stronger guidance and improved linkage with policy bodies (cantonal,
entity, and state) could ensure sustainability of the mixed welfare methodology.”

Child Justice Projects (Zambia 2005): “…it is therefore recommended that in order to kick-start a
reform process that is explicitly aimed at improved service delivery, that an action plan with not
more than ten key deliverables is widely consulted upon and that this be communicated
thoroughly to all levels of government and civil society.”
7.6.9 Need to grapple with conditions in conflict-affected areas
Two reports comment on difficulties of sustainability in conflict-affected areas:

Child Protection Networks (Somalia 2005): “…unless urgent steps are taken to mitigate their
current weaknesses, the CPN’s *Child Protection Networks+ do not represent the best avenue for
the protection of Children’s rights in Somalia. This is especially so since many of the violations
are perpetrated in the name of religion or traditions which are powerful social forces. This
reality is complicated further by the prevailing state of violence, which puts at risk any persons,
or institutions that dare raise their voices against the violations.”

Reintegration and Rehabilitation of Children Associated with Militia (Somalia 2006): “Partners in
conflict-affected countries, as the Somali example shows, often lack adequate organisational
capacity since most of the capable people have left the country. Capacity enhancement of
partner organisations and their staff should therefore from the outset be a key program
component.”
7.6.10 Lessons lost due to a weak evidence base

Although 12 reports proffer evidence of positive systematic capacities building efforts, the
evidence base in these was insufficient. Investments addressing the needs of street working
girls, life skills, and harmful traditional practices were among these 12 reports.

Another 15 reports proffer evidence of negative impacts on sustainability but these have an
insufficient evidence base.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
84
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
85
Chapter 8:
Services Delivery
The skills training programme has managed to engage large numbers of children in learning
vocational and business skills, but it has not always increased their employability. The
apprenticeship programme however, seems to book good results.
Reintegration Programme Evaluation (Liberia 2007)
Capacity for program innovation limited by activity-based funding; little scope to respond to
emerging needs and deviate from planned activities … Activity-based funding and reporting
tends to fragment global strategies.
Cambodia Disabilities Project (2005)
The situation of girls confined in facilities for adult women has received scant attention in
these projects … On the whole, the evaluation team believes that the attention paid by
UNICEF to gender issues and gender sensitivity has been limited both during planning and
implementation. (emphasis in original)
Evaluation: Juvenile Justice System Reform (Serbia, Romania, Tajikistan 2006)
Introduction and Overview
The delivery of necessary and child friendly services is a seventh element of the Protective
Environment. Child victims of any form of neglect, exploitation, or abuse are entitled to care and nondiscriminatory access to basic social services. These services must be provided in an environment that
fosters the health, self-respect, and dignity of the child. The services ultimately must become integrated
into the overall social welfare system.
Within the universe of this meta-evaluation, the “access to services” component of the Protective
Environment is the second most highly rated for the quality of its programming practices. Evaluation
reports were analyzed on the basis of three sub-categories:

Provision of necessary and child-friendly services

Child victims of violence, abuse, and exploitation facilitated to access these services

Strengthened social welfare systems
Across the whole of the meta-evaluation, no element of the Protective Environment saw more activity
than “Access to necessary and child friendly services.” The quality of its programming practices was 26%
greater than “Strengthened social welfare systems,” where the evidence across the meta-evaluation
was about 20% less. Yet, while not well represented in the meta-evaluation, contributions to stronger
social safety nets are substantial.
A striking feature of the “Access to necessary and child friendly services” materials is that in spite of the
relative density of evidence addressing this topic, evidence quality is a significant constraint: of 27
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
86
reports possessing information on service delivery efforts, only 15 rated credibly in terms of their
evidence base.
The category “Child victims of violence, abuse, and exploitation, facilitated to access these services” was
de-limited originally to evaluations of programs addressing the needs of orphans and unaccompanied
children. Roughly 10 evaluation reports fit this category, clustering in these countries: Eritrea, Iran,
Ghana, Somalia, Swaziland, Tanzania, and Zambia. In three instances (Eritrea, Iran, and Tanzania)
investments are found to be contributing to strengthened social safety nets and/or substantive policy
shifts. Because of the many overlaps with the “Access to necessary and child friendly services” category,
these two categories are treated as one in this chapter.
In the aggregate, this component of the Protective Environment:

Demonstrates better impacts when the program focus is narrow (e.g., thematically discrete).

Can be improved by more sophisticated needs assessments, for instance, mapping vocational
training to actual labor market requirements, and more focus on quality assurance monitoring.

May need to consider whether services are specified equally for the needs of girls as well as
boys: a constant theme across the evaluation reports is that programs are paying insufficient
attention to gender.
Lastly, evaluation reporting for this component of the protective environment can be a constraint
because what actually got provided and to whom is often the least of an evaluator’s concern:
disentangling who was supposed to do what, while meeting the many requirements of current global
evaluation reporting protocols, means that reports can be unfocused.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
87
Provision of Necessary and Child-Friendly Services
The decision to work on pilot projects while simultaneously working on law reform was a
good one, because the success of pilot projects contributed to support for the draft law.
Juvenile Justice System Reform Impact Evaluation (CEE/CIS 2006)
The appropriateness of the training package was not based on training needs assessment
and mapping of employment opportunities of the local market to absorb would-be
graduates.
Evaluation: Reintegration and Rehabilitation of Children Associated with Militia Program (Somalia 2006)
Synopsis
Data addressing “Provision of necessary and child-friendly services” is not exactly an afterthought in the
evaluation reports that were available for this meta-evaluation. However, as suggested above,
evaluation reporting around this topic can be intensely cluttered and the bottom line is not always
accessible. Delivery of services, even if the programme intended to produce them, is sometimes so
constrained that the constraints become the evaluator’s focus rather than who accessed what services
to what impact. The askSam database offers an excellent window into this state of play.
As a result, in the universe of materials that were available for this metaevaluation there is substantially more information on what doesn’t work—up
and down the lines of coordination especially—than on what does work.
Taking the foregoing into account, then, the meta-evaluation findings for this
element of the Protective Environment are necessarily circumscribed.
What works—supports to service provision
 Specific themes in combination with targeted demographics (six
reports with credible evidence; themes noted were street children,
orphans and vulnerable children, and juvenile justice). 46
FINDINGS
Evaluation reporting on this
topic can be intensely
cluttered … constraints to
service delivery can become
so distracting that who
accessed what services to
what outcome can be hard to
trace. The askSam database
offers an excellent window
into this state of play.
What is not working—constraints to service provision
 Narrowly framed needs assessments with limited quality assurance mechanisms, overestimating
volunteer capacities, and underestimating professionalization requirements (nine reports with
credible evidence).47
1.
46
Effective service delivery investments are also seen in the context of natural disasters, a topic that is covered in
Chapter 10, “Child Protection, Emergencies.”
47
Poor or unacceptable evidence of investments that did not meet their intended targets was found in 12 reports.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
88
8.1 Investment characteristics that facilitate service delivery
The reports in this section share two feature in common: (a) the services to be delivered have either a
single discrete demographic in a relatively localized geographic setting (e.g., urban street working
children, orphans in a village community, youth offenders in targeted communities) or the thematic
emphasis of the services is narrow (e.g., child abuse, housing, education, victim-offender mediation),
and (b) either of these is combined with community outreach and awareness-raising.
8.1.1 Specific services for specific demographic, with community outreach

FINDINGS
Street Children/Street Mother Program Evaluation (Ethiopia 2003)
What works in service
This program had five primary components: education, health, microdelivery: an on-the-ground
credit, housing, and awareness raising. Education supports were
presence, a topic with wide
implemented in two specified localities:
inter/national visibility,
o Number of street children and youth enrolling in formal
steady/sturdy community
education with the support of the project has reached 2,282,
outreach, and partners with
which is almost double of the originally envisaged plan.
a good track record (six
o The program’s “more profound achievement” has been in
reports).
raising awareness about street children … The introduction of
the idea of partnership into the project has paved the way for opportunities to come
together and co-create visions, strategies, and programs for joint action to create
synergistic impacts [for street children and mothers].

Street Children, Program Evaluation (Laos 2006)
Project targets children who are “at risk” for entering street life:
o Community outreach activities … reaching approximately 450 children and young people
each week at the time of this review.
o …most “at risk” children supported by [the program] to return to school did so after
completing remedial classes with the mobile school.
"One of the main findings was
that improvement in housing
has begun … this is a great
innovation. Few programmes
(in the history of the country)
have ever been able to bring
changes in housing. The success
in building and repairing some
MVC’s houses is a clear
indication that it is possible to
address the problem of housing
for MVC at village level.”
Sheeran Consulting, Inc.
Final Report, 15 May 2008
Tanzania [Magu] 2004
2008 UNICEF Child Protection Meta-Evaluation
89

Impact Assessment of MVC Support Programme Implementation in
EXCERPT
Karagwe, Kisarawe, and Magu (Tanzania 2004)
Housing improvements are a singular success but the evaluator does not say why:
o One of the main findings was that improvement in housing has begun … this is a great
innovation. Few programmes (in the history of the country) have ever been able to bring
changes in housing. The success in building and repairing some MVC’s houses is a clear
indication that it is possible to address the problem of housing for MVC at village level.”

MVC Impact/Scaling Up Assessment (Tanzania 2007)
Although service delivery here faced some constraints (see next section), UNICEF worked
through community networks with external funding for education, clothing, food, medical
supplies, and soap:
o The UNICEF/Most Vulnerable Child (MVC) programme provided support to an increasing
percentage of children over time. By 2005 and 2006, almost 30 per cent of surveyed
children reported support from the programme.

Lihlombe Lekukhalela (LL) Child Protectors (Swaziland 2005)
Greater gains where community members are involved in efforts to reduce child abuse:
o Generally LL [Child Protector] areas fair [sic] better in terms of reporting of abuse.

Juvenile Justice system Reform Impact Evaluation (CEE/CIS 2005)
The evaluator finds that “there is some evidence that interventions that have taken place in
Romania, Serbia, and Tajikistan have had a positive impact on the lives of hundreds of children,
or perhaps thousands.” Pilot projects offering specific types of services in localized settings
helped to “win acceptance” for new, community-based alternatives to incarceration. Taking
Serbia as an example:
o The decision to work on pilot projects while simultaneously working on law reform was
a good one, because the success of pilot projects contributed to support for the draft
law. The part of the project focused on community based alternatives provided
considerable support to two quite different approaches: the mobile child rights teams
designed to support preventive and rehabilitative services that focus on the adolescent
and his/her family, and victim-offender mediation (VOM). VOM was a new concept in
Serbia and is, by nature, an approach that is difficult to sell to the public and authorities,
who often see it as being soft on offenders and failing to understand the importance of
mediation to the victim and the utility of VOM in preventing recidivism. The decision to
support these two complimentary approaches, and to make a long-term investment in
winning acceptance of VOM in a carefully chosen pilot community, was a wise one.
8.2 Constraints to service delivery
8.2.1 No substantive accountabilities

Zambia Child Justice Program Evaluation (2005)
FINDINGS
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
90
More awareness without tangible changes:48
Constraints to service
delivery: weak outcome
o On a general level it can be concluded that an increased
accountabilities (eight
awareness and sensitivity towards children’s rights in the
reports).
criminal justice system were observed in nearly all the sites
visited during the fieldwork. This has, however, not translated
into a general improvement in service delivery (especially outside of Lusaka) and
children are still subjected to nearly all of the ills, misuses, and delays that were
observed in 2000.

Street Children and Street Mother (Ethiopia 2003)
Partners mobilized without administrative or institutional accountabilities:
o Had it not been for the numerous bottlenecks entangling the implementation process,
the performance of the program could have ridden on a higher plane. The most
prominent bottleneck with a pervasive effect on the implementation process had been
the form of the partnership, which was characterized powerless [sic]entity and weak
internal process. The non-existence of rules and regulations has preempted the practice
of transparency and accountability casting very long shadows on the emergence of good
governance that could have otherwise prevailed. As procedural and operational
guidelines were not prepared, the performance of duties became hostage to the rule of
thumbs; or at best, personal caliber and goodwill. To sum up, it is above all the absence
of the major institutional elements in the partnership forged that underlie the
curtailment of performance.

Cambodia Disabilities Project (2005)
Piecemeal programme planning:
o There seems to be a need across UNICEF to examine
inter-program and sub-program coordination. In this
respect, the Child Protection Programme needs to take a
closer look at how its supported activities dovetail with
broader UNICEF programming, particularly with its
education and social development sections.
48

UNICEF’s Support to Mine Action (Global 2005)
Limited accountability for outcomes:
o The effect that UNICEF support to mine action has had on
the lives of mine/UXO affected populations remains
insufficiently verified and UNICEF can do much more to
develop adequate performance and quality management
systems. … Quality control mechanisms are used more than
quality assurance.

Evaluation: Reintegration and Rehabilitation of Children Associated
with Militia Program (Somalia 2006)
Training is not aligned to market realities:
o The appropriateness of the training package was not based
on training needs assessment and mapping of employment
Also reported in Chapter Three, “Legislation and Enforcement.”
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
91
“On a general level it can be
concluded that an increased
awareness and sensitivity
towards children’s rights in the
criminal justice system were
observed in nearly all the sites
visited during the fieldwork.
This has, however, not
translated into a general
improvement in service delivery
(especially outside of Lusaka)
and children are still subjected
to nearly all of the ills, misuses,
and delays that were observed
in 2000.”
Zambia 2005
EXCERPT
o

opportunities of the local market to absorb would-be graduates.
The fact that no critical examination of the project proposal was made before approval
raises questions as to how the project rational [sic] was identified, whether the various
stakeholders have actually been involved in the identification and definition of the
core-problem of children associated with militia. This question is directly linked to the
issue of ownership.
Psychosocial Care for Children Traumatized by Terrorist Violence (Algeria 2003)
Limited outcomes accountabilities:
o Although most critical, the question about the outcome of activities and their impact in
improving the wellbeing of children traumatized by terrorist violence remains to be
answered.
8.2.2 Unassertive management

Reintegration Programme Evaluation (Liberia 2007)
Evaluator does not say whether management had an exit strategy or
whether management was aware of this concern:
o The majority of schoolteachers have received psychological
training that they found to be useful in their work with
ex-CAFF and other children; they are concerned about the
continuity of educational and psychosocial care and support
activities after the project end.
FINDINGS
More constraints to service
delivery: weak response to
emerging evidence of design
flaws including an overreliance on volunteers and an
assumption that girls’ needs/
opportunities are the same
as boys’ (eight reports).

Protecting Street Girls and Preventing Child Migration (Ghana 2003)
Giving information without the means of acting upon it:
o The HIV/AIDs/STI information given in the context of the RPP programme was
inadequate … the failure in the programmes was that promotion of condoms was weak
or non-existent.
o …vocational programs reach a very small # of people … need for management skills to
run a business

Child Protection Networks Program Evaluation (Cambodia 2004)
No explanation given for the neglect of a needs assessment:
o …There is some concern that the CBCP *Community Based Child Protection Programme]
was established and expanded without undertaking a substantial assessment of child
needs and protection issues in the target areas. As a result there is a lack of information
on children’s issues, which would ensure that interventions are focused and relevant.

Cambodia Disabilities Project (2005)
Fiscal rigidity weakened project’s responsiveness to new opportunities:
o Capacity for program innovation limited by activity-based funding; little scope to
respond to emerging needs and deviate from planned activities … Activity-based funding
and reporting tends to fragment global strategies.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
92

Community Based Reintegration (Sierra Leone 2006)49
Singular, or “one-off “ investments with no long term plan:
o Many children stopped attending once CEIP [Community Education Investment
Programme] had ended. The only reason that children cited for not attending was that
they were unable to afford school fees.
8.2.3 Misunderstanding programme contexts

Reintegration Program Evaluation (Liberia 2007)
While every service discontinuity is unintended, giving selective assistance for whatever reason
caused wide confusion:
o There are also emerging tensions around the issue of
"There are also emerging
fairness of the programme. The provision of Transitional
tensions around the issue of
Safety Allowances, first to the children and later in the
fairness of the programme. The
programme to the parent, gave the wrong signal that child
provision of Transitional Safety
soldiering pays, hence fostering feelings of unfairness among
Allowances, first to the children
other war affected children and their parents.
and later in the programme to
o In addition, the targeting of the skills training programme on
the parent, gave the wrong
ex-CAFF (with a maximum of 10% of non-CAFF) has
signal that child soldiering
exacerbated these feelings of unfairness, especially because
pays, hence fostering feelings
there are no such programmes available for other children
of unfairness among other war
and youth … Since there are hardly any other vocational
affected children and their
training programmes ongoing in the country it can be
parents."
concluded that the DDR programme has created an unfair
Liberia 2007
balance of trained ex-CAFF in a situation where alternative
training is not available to other children.50
EXCERPT

Child Justice Program Evaluation (Zambia 2005)
Service delivery that inadvertently levels the playing field between victim and aggressor:
o Victims are also referred to the diversion programme by the magistrate. Two cases were
reported (one gang rape and one defilement) where the victim and offenders were all
sent to the life skills programme.

MVC Impact/Scaling Up Assessment (Tanzania 2007)
Investments do not free the community from dependency on UNICEF:
o An additional concern arises from the fact that the programme was seen to be a
“UNICEF” programme, in spite of efforts to have support for most vulnerable children to
be community-based. With many demands on communities for contributions for
development programmes, especially those related to education and water, those
programmes with the strongest political and governmental pressures take priority, and
those with obvious external support may then be marginalised in local resourcing.
49
An earlier evaluation report addressing the needs of war-affected children in Sierra Leone, Evaluation of Alternative Care
programmes (2003), has a poor evidentiary rating with mostly piecemeal impact analyses.
50
These findings should be considered alongside findings addressing another aspect of the program: “The apprenticeship
programme however, seems to book good results. A relatively high percentages [sic] of apprentices remain with their masters,
to do some self-employed work within their workshops, combined with school. This is a new approach that must be supported
and potentially become one of UNICEF’s approaches for the reintegration of youth.”
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
93
8.2.4 Inattention to girls’ needs, opportunities

Protecting Street Girls and Preventing Child Migration (Ghana 2003)
Unclear why the programme had no plan to address an obvious concern for this target
demographic:
o No explicit attention to gender-based violence … The
“…Program is very inattentive
majority of Accra-based beneficiaries were victims of
to “serious protection concerns
gender-based violence. At the some time, RPP [Rights
(that) face girls.”
Promotion and Protection Programme] had no explicit
intervention to address this issue. There were no
Sierra Leone 2006
explicit interventions to make the legal, community, or
service environment more responsive to the needs of
EXCERPT
victims. In addition, partners of RPP were rather weak
with regard to their response to and support of victims.

Community Based Reintegration (Sierra Leone 2006)
The de facto subjects of this investment would seem to be school-going boys:
o … While Children’s Clubs are functioning well, they are not inclusive of girls and children
that do not attend school. Girls face a number of serious protection concerns that need
immediate attention and action. …Program is very inattentive to “serious protection
concerns (that) face girls.”

Juvenile Justice system Reform Impact Evaluation (CEE/CIS 2005)
Another instance of a de facto gender bias:
o The situation of girls confined in facilities for adult women has received scant attention
in these projects.
o On the whole, the evaluation team believes that the attention paid by UNICEF to gender
issues and gender sensitivity has been limited both during planning and
implementation.
8.2.5 Overestimating volunteer capacities

MVC Impact/Scaling Up Assessment (Tanzania 2007)
There is no information about project managers’ reactions to this situation, but the project
budget was reported to be a serious constraint:
o This form of programme delivery puts much of the onus on local committee structures
which themselves frequently need strong oversight.
o Members of the committees reported problems of the size of their villages and the
workload—work for which they were considered volunteers and therefore without
compensation.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
94

Evaluation of Lihlombe Lekukhalela Child Protectors (Swaziland 2005)
There was no information on project managers’ responses to the volunteer fatigue or ownership
problem:
o A problem identified by partners in the initiative is the lack of ownership in some areas
where members are still reliant of the donor for further assistance.
o Volunteer fatigue happens for a number of reasons. …the moment an employment
opportunity occurs they must take it. …they have to supplement to food or clothing of
neglected children at the expense of their families … in one area the LL states that the
bulk of them are only available during certain seasons.

Child Protection Networks External Evaluation (Cambodia 2004)
Again, no information here on project managers’ responses or remedies:
o Based on the assessment, services have been provided to some children. However, the
support was limited and mostly on a one-off basis.
o It is also problematic to base community based services to a large extent on volunteer
contributions, in the form of CSH as well as the village social funds.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
95
Strengthened Social Welfare Systems
The impact of the project activities extend beyond the initial objectives and has contributed
to developments in mental health policies and services in general.
Psychosocial Care for Children Traumatized by Terrorist Violence (Algeria 2003)
Synopsis
Signs of strengthened social safety nets were verified in five reports of investments supporting orphans,
children in residential care, psychosocial, and juvenile justice reform. The main
FINDINGS
indicators were:
— Shift from institutional to
family-based care.
o Shift from institutional to family-based care for children without
— Shift in mental health
primary caregivers (Iran).
o Shift in mental health policies and services (Algeria).
policies and services.
o Program lessons being used to scale up protection/support for orphan
— Pilot lessons used to scale
and vulnerable children country-wide (Tanzania).
up protection for vulnerable
children country-wide.
o Reunification of orphan children and improving economic status of host
— Reunification of children
families (Eritrea).
and improving economic
o Community based alternatives to detention (CEE/CIS).
status of host families.
— Community based
Another four reports listed signs of strengthened social safety nets (three
alternatives to detention.
addressing juvenile justice investments, one for gender) but without a
supporting evidence base.
What were the primary constraints to building stronger social safety nets? Of 15 reports where
information was found to be relevant (that is, one-quarter of the entire meta-evaluation corpus of
materials),51 10 possessed a credible evidence base. Three constraints were identified:
o
o
o
Poor strategic information led to inaccuracies in program concept/design.
Implementation was pushed ahead of institutional capacity building.
Short-term outlooks compromised sustainability.
51
Readers will recall, further, that the meta-evaluation addresses a fraction (40%) of the total Child Protection evaluation
reports that were produced between 2002 and 2007. For the sake of argument, if only 15 reports out of 147 comment on
systemic change, the lynchpin of the Protective Environment Framework, this would suggest that only 10% of Child Protection
evaluation reporting is turning up information on this foundational matter.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
96
8.3 Investment characteristics that contribute to stronger social safety nets
Five reports possess credible evidence of shifts in the direction of strengthened social safety nets. These
shifts show UNICEF investment in a highly contextually relevant manner and they are expressive, in the
opinion of the evaluator, of a patient long-term outlook:
FINDINGS
8.3.1 Context-driven, with long-term outlooks
What works to facilitate
stronger social safety nets:
creating an enabling
economic environment,
forging a common body of
knowledge and practice (five
reports).

Mahzel Project for Social Reintegration and Protection of
Disadvantaged Children as Future Development (Eritrea 2004)
Creating an enabling economic environment:
o … 3703 families (103% of the set target) and 8589 orphans
(107% of the set target) have been assisted to date.
o Up to 70% of the assisted families increased the capital value
of their assistance.

BAM Emergency Programme Evaluation 2003–2006 (Iran [Wiles et al] 2007)
Contributing a substantive alternative during a time of crisis:
"Up to 70% of the assisted
families increased the capital
o The impact of the joint SWO (State Welfare Organization)
value of their assistance."
and UNICEF programme has been significant. The majority
of children without primary care givers are living with
Eritrea 2004
extended family — “family based care” … This is a major
departure from the prevailing practice, if not policy, in
EXCERPT
Iran.
o The experience has also influenced the way that the State Welfare Organisation views
childcare in the aftermath of disasters.

Psychosocial Care for Children Traumatized by Terrorist Violence (Algeria 2003)
Helping to forge a common body of knowledge and practice:
o The impact of the project activities extend beyond the initial objectives and has
contributed to developments in mental health policies and services in general.

MVC Impact/Scaling Up Assessment (Tanzania 2007)
Effective stewardship of a pilot programme:
o Many of the outputs from the MVC programme are currently used in the process of
scaling-up protection and support of the MVC in the country. These include the various
guidelines and training manuals.

Juvenile Justice system Reform Impact Evaluation (CEE/CIS 2005)
Taking the long view with regard to a highly technical area of Child Protection:
o Despite the complexity and long term nature of the process precludes [sic] a simplistic
comparison of the investments made with the results obtained, especially while the
process is still ongoing, it can be affirmed that, on balance, the results are positive and
there is reason to believe that these processes will continue moving forward.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
97
8.4 Constraints to strengthening social safety nets
Ten reports had credible evidence to back up findings that certain practices were interfering with
progress towards strengthening social safety nets. Three constraints were
FINDINGS
identified:
Constraints to developing
stronger social safety nets:
 Skills training in a vacuum
one-off investments, missing
 Implementation pushed ahead of capacity building
exit strategy, investments
 Short term outlooks
ahead of infrastructure or
capacities (10 reports).
8.4.1 Skills training in a vacuum
The Liberia evaluation (2007) uses the phrase “enabling environment” to
describe the context that, properly accounted for, might have increased the long-term value of
vocational skills training investments:

Reintegration Program Evaluation (Liberia 2007)
o The skills training programme has managed to engage large numbers of children in
learning vocational and business skills, but it has not always increased their
employability (underline added).
o No efforts have been made to create an enabling environment for economic
reintegration and the linkages required with other actors have been too limited to make
a significant difference. … Follow-up of new businesses is weak and therefore there are
failures that are already being seen.
o … the main areas of training are ‘traditional’ choices of vocation such as carpentry,
cosmetology, and other typical skills that are already on the market and in high
competition.

Protecting Street Girls and Preventing Child Migration (Ghana 2003)
The evaluator calls for a greater emphasis on making the “programmatic environments” safe
and supportive for the target demographic:
o Many programmes focus on providing services and skills and neglect creating a safe and
supportive environment.
8.4.2 Implementation pushed ahead of institutional capacity building
In five reports the evaluators found problems relating to sustainability: oneoff investments, missing exit strategy, investments where the community
infrastructure was not yet in place:

Child Protection Networks Program Evaluation (Cambodia 2004)
“One-off” spurts of assistance:
o Material assistance to [program] was provided on a one-off
basis only.
o There is a lack of institutionalising support for children that
are in need of long term support and assistance.

Cambodia Disabilities Project (2005)
Concern that management commitments are not in place:
o Sustainability will be determined by how successful the
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
98
“Capacity for program
innovation limited by activitybased funding; little scope to
respond to emerging needs and
deviate from planned activities
… Activity-based funding and
reporting tends to fragment
global strategies.”
Cambodia Disabilities Project,
Evaluation 2005
EXCERPT
partners are in guiding and supporting the transition of project activities to either
community or government actors. In this respect, while funding security is an important
element, high level strategic advocacy and sound technical advice on planning and
managing effective exits is probably more relevant and of greater value.

MVC Impact/Scaling Up Assessment (Tanzania 2007)
Community capacities and buy-in, and program resources, are unresolved:
o In programmes for most vulnerable children in Tanzania to date, support has been
provided though structures of local committees and generally in-kind—clothing, school
books, for example. The overhead costs of maintaining support and supervision for this
type of delivery system are high.
o Weak commitment and motivation among the stakeholders and facilitators at all levels,
especially at the lower levels (ward and community level), are major challenges in
efficient realization of the programme’s outputs.

MVC Community Based Care, Support, and Protection in Musoma (Tanzania 2004)
Constraints in program financing and management capacities:
o The program implementation process at all levels is constrained by poor coordination;
lack of commitment by the government leaders; lack of integration of the program to
other development efforts; under-resourcing the programme; poor MVC data
management and lack of follow-up; and lastly, lack of integration of the non-state actors
in the implementation of the program (i.e., NGOs, FBOs)
Evaluation of the Program “Improved Mechanisms of Child Protection” (Bosnia and Herzegovina
2003)
Delivery of services seems to be an end in itself, the broader enabling environment (e.g., policy
development) was ignored:
o … the program … is implemented in such way that its relevance is greater in the area of
service delivery than in improved systems for developing mechanisms of child
protection.
o The aspect that is almost absent from this program, yet would be of great importance to
its success, is a link with the policy-making institutions …
o Of course, it is questionable whether systems as a whole are improving given that in
some of the municipalities the system itself is scarcely operational. It might take several
years until a clear improvement could be seen.

8.4.3 Small scale investments and limited time horizons
Constraints include a small target population, an over-reliance on non-core funding, a misunderstanding
of activity-outcome-impact relationships and timelines, and limiting activities to the development of
symbolic measures such as national plans with no effective implementation strategy:

Protecting Street Girls and Preventing Child Migration (Ghana 2003)
The evaluator comments on the limited reach of this basket of investments:52
52
Another report addresses investments that were too limited in scope to generate systemic changes: Street Children, Program
Evaluation (Laos 2006). The intended target population for this project is 500 street children.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
99
o

Most of the programmes in the RPP [Rights Promotion and Programming] portfolio
reached a relatively small number of children.
“Reintegration takes much
Reintegration Program Evaluation (Liberia 2007)
longer than the remaining 7
The results chain is poorly understood here:
months of the current
o …there is a need to come up with [a] Bridging Programme
programme with currently
that can build upon the positive dynamics initiated through
approximately 70% of the
the DDR programme for children. If this cannot be assured,
target group (9,100) working,
there are serious risks that the results of the reintegration
in school or in training. The
programme will not be sustainable.
community structures that have
o Reintegration takes much longer than the remaining 7
been set up are good but need
months of the current programme with currently
more support to become
approximately 70% of the target group (9,100) working, in
sustainable. The capacities of
school or in training. The community structures that have
national actors to ensure that
the positive results can be
been set up are good but need more support to become
sustainable. The capacities of national actors to ensure that
maintained are not fully in
the positive results can be maintained are not fully in place.
place.”

Liberia 2007
Evaluation: UNICEF Support to Mine Action (Global 2005)
Ad hoc financial plan:
EXCERPT
o Mine action support is dependent on non-core funding and
is struggling to prove its outputs and outcomes in an environment where competing
demands, both internally and externally with donors, jeopardise sustainability of its
approach.

Education as a Preventive Strategy Against Child Labour (Global 2003)
Ad hoc activities:
o Programme activities … remain relatively discrete and isolated from each other … it
remained to be seen whether it (the Programme) could get beyond the formulation of a
national plan/programme to support the execution of the plan/programme.

Juvenile Justice System Reform Impact Evaluation (CEE/CIS 2005)
Need for a plan to move beyond pilot investments:
o Sustainability concerns arise above all with regard to the pilot diversion/alternative
sentence pilot projects. There are hopeful signs that the government will rise to the
challenge of funding similar projects throughout the country.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
100
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
101
Chapter 9:
Monitoring and Oversight
After the mother leaves the maternity together with the child, no one monitors what
happens next.
Evaluation of Projects Aimed at the Prevention of Child Abandonment in Maternities (Romania 2003)
With functioning databases the program/municipalities could have had a better base for
evidence based policy making, which in turn would better inform the needs of each
municipality.
Evaluation of the Program “Improved Mechanisms of Child Protection" (Bosnia and Herzegovina 2003)
Introduction and Overview
Monitoring and oversight is an eighth element of the Protective Environment. A protective environment
for children requires an effective monitoring system that records the incidence and nature of child
protection abuses and allows for informed and strategic responses. Such systems can be more effective
where they are participatory and locally based. It is a responsibility of government to make sure that
every country knows the situation of its children with regard to violence, abuse, and exploitation.
Within the universe of this meta-evaluation, the “Monitoring and Oversight” component of the
Protective Environment is the lowest rated for the quality of its programming practices and ranks as the
most notoriously deficient and frequently lamented investment area across the entire Protective
Environment investment portfolio. Statistically, the Protective Environment investments that align to
centralized political power—as expressed, for instance, in the areas of enforcement and building
national partners’ capacities—are among the least likely to be monitored for quality assurance.53
Evaluation report contents were analyzed on the basis of five “monitoring and oversight” subcategories:

Monitoring systems including data collection on CP issues

Establishment of participatory and locally-based mechanisms

Establishment of reliable reporting systems

Establishment of community surveillance systems

Information on sensitive subjects, e.g., closed institutions
53
This finding is congruent with the current global analysis of constraints to aid effectiveness. As discussed in the Conclusions
chapter, however, it is striking that evaluation report authors put their very dramatic findings regarding monitoring into no
particular context (other than resource constraints).
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
102
In the reports that were available for the meta-evaluation, there was no instance of an investment
relating to “Information on sensitive subjects.” The one instance of a participatory process is—at the
other extreme—a replicable methodology and a good practice (see below). Additionally:

In 16 reports—almost one-quarter of the meta-evaluation content—monitoring-related
information had a good evidence support base. Of these:
o
o

Good investments are seen in seven instances.
Constrained investments are seen in nine instances.
Monitoring-related investments were described as missing altogether in half of the evaluation
reports (29).
Essentially, then, through the lens of the reports that were available for the meta-evaluation,
monitoring and oversight are working properly in just 7 of 59 of the meta-evaluation programmes.
Indeed, across the whole of the meta-evaluation no element of the Protective Environment saw less
activity and less overall programming quality than “Monitoring and Oversight.” To see just how poorly
this component of the Protective Environment Framework is faring relative to other components in the
context of the meta-evaluation, the quality of its programming practices has a value of:


26% less than the programming practices for “Open Discussion,” which is the next most poorly
rated for programming practice quality.
41% less than the element with the highest programme practices ratings overall, “Strengthening
Government Commitments.”
In the aggregate, the “Monitoring and Oversight” component of the Protective Environment:




Significantly obscures the child protection investments that are working and worthwhile and
makes them impossible to distinguish from investments that are not working and are not
worthwhile.
Shows limited managerial and donor commitment without specifying either’s operating
environment.
Will not see improvements without de-mystifying measurement strategies, concepts, and
rationales and making these more accessible and mandatory.
Would benefit from a good practice review of participatory approaches.
This chapter itemizes what is known about what works and what is not working for Tier One findings—
that is, findings that are supported by good to excellent quality evidence.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
103
Monitoring Systems, CP Data Collection
The President of the Republic of Belarus instructed the Ministry of Labour and Social
Protection to set databanks on graduates of children’s homes and state schools for children
and to organise their follow-up after they graduate from such institutions through the
regional social protection centres.
Deinstitutionalisation Project Evaluation (Belarus 2004)
The handing over of the project responsibility from one staff to the other within UNICEF
Somalia was apparently not well done, which has resulted to a meager trail of documents.
This combined with a limited ability for project monitoring meant that a professional and
broad mentoring and monitoring of the project did not take place as would have been
expected.
Evaluation of Reintegration and Rehabilitation of Children Associated with Militia Program (Somalia 2006)
Synopsis
This area of the meta-evaluation is almost barren, which is hard to understand given the emphasis on
results based tools and methods across UNICEF as a whole. The assumption is that the poor showing on
monitoring systems and Child Protection data collection is an artifact either of evaluation reporting
priorities or the sampling of evaluation reports that were available for the meta-evaluation. For
instance:
FINDINGS :


Only three reports in 59—five percent of the whole meta-evaluation
content—contained well-supported evidence of workable investments
in data collection and monitoring. The distinctive element in these
instances is the commitment of managers and/or government
partners. 54
Over one-third (23) of the reports reported that monitoring and data
collection systems were missing altogether.
Moreover,

Child Protection can find lessons learned about constrained or failed
monitoring and data collection investments in only seven reports. The
main lessons are:
54
Only three reports in 59—five
percent of the whole metaevaluation content—
contained well-supported
evidence of workable
investments in data
collection and monitoring.
The distinctive element in
these instances is the
commitment of managers
and/or government partners.
Four additional evaluation reports described workable investments but the evidence quality limits their usefulness for the
meta-evaluation.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
104
o
o
o
An “off the shelf” data collection system is far preferable to “build as you go”
approaches.
Management commitments –not management capacities—need strengthening.
Across the board, the “why” of monitoring, not just the “how” of monitoring, needs
reinforcing and demystifying.55
9.1 Investment characteristics that support systematic monitoring, data collection
9.1.1 Management or (high level) government commitment
In the three reports where the evidence base is good, the driving factor behind the presence and
functionality of monitoring and data collection systems is the commitment of the management or
government partner. The epigram set at the beginning of this chapter
(showing that monitoring commitments flowed directly from the President
“During a 12 month period…
of Belarus) illustrates this point.
the reintegration team made a
This commitment, illustrated further in the following short excerpts, sets
these programs apart from any others across the entire meta-evaluation:



total of 1,796 follow‑up visits—
an average of 150 each month
—to children in 28 different
schools and as many different
communities."
Street Children, Program Evaluation (Laos 2006) — Commitment,
ambition, and vision:
Laos 2006
o Peuan Mit staff have tracked the progress of all children
EXCERPT
supported to return to family care and to school through
regular follow-up visits to meet children, their teachers, and
parents. … During a 12 month period … the reintegration team made a total of 1,796
follow-up visits—an average of 150 each month—to children in 28 different schools and
as many different communities.
Deinstitutionalisation Project Evaluation (Belarus 2004)— Commitment at the highest level
possible:
o The President of the Republic of Belarus instructed the Ministry of Labour and Social
Protection to set databanks on graduates of children’s homes and state schools for
children and to organise their follow-up after they graduate from such institutions
through the regional social protection centres. (Attribution: UNICEF budgeted
213222.76 USD between 1999–2003 inclusive of support for the development of
regulatory and legal frameworks)
MVC Impact/Scaling Up Assessment (Tanzania 2007) — Collective commitment:
o … the national guidelines and monitoring and evaluation system which resulted from
the pilot phase of the programme are now used by collaborating partners who support
programmes for most vulnerable children with resources from the Global Fund and
from PEPFAR [President's Emergency Plan for AIDS Relief].
55
Another four reports also contain descriptions of constrained or failed investments, but the evidence quality is too uncertain
for these to be utilized for the purpose of lesson learned.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
105
9.2 Constraints to systematic monitoring, data collection
9.2.1 Call for an “off the shelf” system
Two reports that have a reliable evidence base suggest that UNICEF Child
Protection might invest in an “off the shelf” data tracking system where issues
such as search functionality, user-friendliness, and data-entry parameters can
essentially be largely pre-set.
FINDINGS
Constraints to CP data
collection:
— An “off the shelf” data
collection system is far
preferable to “build as you
go” approaches.
— Management
commitments—not
management capacities—
need strengthening.
—The “why” of monitoring,
not just the “how” of
monitoring, needs reinforcing
and demystifying.

Mahzel Social Reintegration Project Evaluation (Eritrea 2004)
o The central office of the MLHW (Ministry of Labour and
Human Welfare) has developed a simple database from which
information can be retrieved; however this data base is totally
dependent on reports sent from the region, which in turn, is
dependent of information forwarded by the sub zobas.
o The ET (Evaluation Team) also notes that the Children Data
Bank (Child Info data base) is a “complex” software and has
not been applicable either at the central or regional level.
Efforts were made by partners including UNICEF in setting up a
child info data base by procuring supplies and equipment in
the first year of project; staffs from the central and regional
level were trained both locally and at UNICEF Regional Office Nairobi. However, several
factors hindered the adoption of the database, one of them being the lack of retention
of trained staff by the MLHW.

Bam Emergency Programme Evaluation 2003–2006 (Iran [Wiles et al] 2007)
o
o
An area in which UNICEF has not been as effective as it might have been is in the
development of a database to support the SWO (State Welfare Organization)
programme. The database is currently being revised, as the first version did not really
allow users to search for data. The main reason for this is
" IT projects developed quickly
that data entry was done on the basis of the case-note
under duress rarely work—
forms that contained as much qualitative data as
a
key
learning point for UNICEF
quantitative.
from this experience will be
IT projects developed quickly under duress rarely work—
to devise an ‘off the shelf’
a key learning point for UNICEF from this experience will
system…"
be to devise an ‘off the shelf’ system similar to the one
Iran 2007
operated by the ICRC.
EXCERPT
9.2.2 Management commitments (not capacities) are not evident
Due either to evaluation reporting gaps or to gaps that are intrinsic to the programmes being evaluated,
a strong managerial hand on the matter of monitoring and data collection is hard to detect. Further, the
multiple ways that managers’ hands can be tied is not an explicit point for discussion in the evaluation
reports.
Three of seven reports with a good evidence base reflect on the matter of management commitments:
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
106



Evaluation: Reintegration and Rehabilitation of Children Associated with Militia Program
(Somalia 2006)
“The handing over of the
o The handing over of the project responsibility from one staff
project responsibility from one
to the other within UNICEF Somalia was apparently not well
staff to the other within UNICEF
done, which has resulted to a meager trail of documents.
Somalia was apparently not
This combined with a limited ability for project monitoring
well done, which has resulted
meant that a professional and broad mentoring and
to a meager trail of
monitoring of the project did not take place as would have
documents.”
been expected.
Somalia 2006
Child Protection Networks Program Evaluation (Cambodia 2004)
EXCERPT
o The national level of MOSALVY was insufficiently involved in
the development and implementation of the CBCP [Community Based Child Protection
Networks] … Monitoring and evaluation conducted by the national level are superficial
and lack analysis.
MVC Community Based Care, Support, and Protection in Musoma (Tanzania 2004)
o …break-up of program implementation facilitation and monitoring. District facilitators
are not following-up or monitoring the implementation of the program. For instance in
Isaba, village community dialogue and MVC identification was done in the year 2002,
and the MVC committee is still waiting for instructions from the district facilitators on
how to proceed from that point…
9.2.3 The “why” of monitoring—not just the “how”—needs reinforcing
Of the seven reports where the evidence quality allows for a reasonable degree of certainty, three
describe difficulties that stem from a fundamental lack of understanding of
“The indicators and means of
the rationale of monitoring and data collection. As the following excerpts
verification are almost never
suggest, the justification for watchfulness and follow through is not
expressed in a quantifiable
instinctive. In place of a coherent framework where expected outcomes and
way: for example “Number of
impacts are attached to inputs, investments seem inconsistent, free-floating,
participants included in the
and unscientific:
network” does not actually
 Evaluation of the Program “Improved Mechanisms of Child
express a minimum acceptable
Protection” (Bosnia and Herzegovina 2003)
number, which makes it
A need for technical training in results based tools and methods is
impossible in principle to
evident here, but an appreciation of its rationale must also have
actually use the logframe to
been lacking:
measure program success.”
o As the outcomes and outputs required by the UNICEF
Bosnia and Herzegovina 2003
evaluation framework are not separately identified … their
indicators are necessarily mixed in with the indicators for
EXCERPT
the outcomes.
o The indicators are not well distinguished from the means of verification. Sometimes
suggested means of verification are actually indicators not covered elsewhere…
o The indicators and means of verification are almost never expressed in a quantifiable
way: for example “Number of participants included in the network” does not actually
express a minimum acceptable number, which makes it impossible in principle to
actually use the logframe to measure program success.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
107

Evaluation of Projects Aimed at the Prevention of Child Abandonment in Maternities (Romania
2003)
The rationale for follow up was missing:
o After the mother leaves the maternity together with the child, no one monitors what
happens next. It is not known whether the child is well taken care of, or whether he or
she is neglected. The social workers think that the Bucharest district Departments for
the Protection of the Rights of the Child are not sufficiently involved and do not have a
coherent strategy for the field monitoring of mothers at social and child abandonment
risk.56
o In order to better evaluate the activity of the “counseling cabinets,” we need to define /
delimit and monitor certain result indicators, and also to conduct a long term study
evaluation of the results and of the impact of counseling on the child’s welfare.

Appreciative Assessment of Psychosocial Interventions in Gujarat (India 2002)
An “off the shelf” system would be useful when the need to respond is sudden, such as the
aftermath of a natural disaster:
o Most importantly the complete process requires predefined impact assessment
systems. The monitoring using the existing educational infrastructure would be helpful
but some effort is needed to monitor indicators like child labor, trafficking, and
second-generation commercial work.
9.2.4 Lessons lost due to a weak evidence base

Only three reports in 59—five percent of the whole meta-evaluation content—contained well
supported evidence of workable investments in data collection and monitoring.

Four evaluation reports described workable investments but the evidence quality limits their
usefulness for the meta-evaluation.

Over one-third (23) of the reports reported that monitoring and data collection systems were
missing altogether.
56
Although this is another instance of evaluation findings resting on what people say, the author communicates the contexts
usefully.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
108
Participatory and Locally Based Mechanisms
Synopsis
This aspect of monitoring and oversight is not well represented in the reports that were available for the
meta-evaluation. In fact:


Only one report in 59 shows a good practice in combination with a good evidence base.
There were no reports of investments in participatory mechanisms that went awry; instead, five
reports in 59 cite the absence of participatory mechanisms as a missed opportunity.
9.3 Investment characteristics of functional participatory, locally based mechanisms
9.3.1 Strong intergenerational alliance at the community level
The single report of a viable investment in participatory and locally based mechanisms has been
presented in previous chapters: the Early Marriage Intervention Program, a sub-programme of the Girls’
Livelihoods Programme (Bangladesh 2005). The intervention encouraged adolescent girls to work with
community leaders to advocate on behalf of a bride-to-be to convince her guardians not to commit her
to a marriage that was inappropriate. Young people act as a community’s “eyes and ears” and elders
with some degree of influence do the negotiating with other decision makers in the community.
9.4. Constraints to participatory, locally based mechanisms
9.4.1 Failing to recognize and make good use of obvious opportunities
Five reports indicate that a failure to utilize participatory and locally based mechanisms resulted in
missed opportunities. These pertain to investments in the following areas:

Street Children (Laos)

Juvenile Justice Reform (CEE/CIS)

Psycho-social/Emergencies (Iran, Indonesia)

Institutionalization
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
109
Reliable Reporting Systems
Synopsis
This is not a well represented area in the reports that were available for the meta-evaluation. In fact,


Child Protection stands to gain lessons learned about reliable reporting from just three out of 59
program investments.
In two of these instances, the investments were not successful largely because programs were
established before reporting capacities/protocols were put into place.
9.5 Investment characteristics of reliable reporting systems
9.5.1 High level government involvement
In the case of supports to most vulnerable children, primarily children orphaned by HIV and AIDS, the
Tanzania evaluation implies that a reliable reporting system was in place, and that it had become a
resource in broadening the scale and definition of the programme:
FINDINGS
 MVC Impact/Scaling Up Assessment (Tanzania 2007)
Three reports in 59 contain
o Many of the outputs from the MVC programme are currently
validated findings about
used in the process of scaling-up protection and support of the
reporting systems. Of these,
MVC in the country. These include the various guidelines and
two were not successful
training manuals…
because programs were
However, this same evaluation also reports near epidemic-levels of
established before reporting
coordination problems and a lack of managerial follow through.
capacities/protocols were put
into place.
9.6 Constraints to reliable reporting systems
9.6.1 Programmes implemented without reporting plans
Two reports find that programs had been established before reporting capacities/protocols were put
into place:
 Evaluation of the Program “Improved Mechanisms of Child Protection” (Bosnia and Herzegovina
2003)
o With functioning databases the program/municipalities could have had a better base for
evidence based policy making, which in turn would better inform the needs of each
municipality.
 Juvenile Justice System Reform Impact Evaluation (CEE/CIS 2005)
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
110
o
o
UNICEF’s culture of accountability is one of its strengths, but juvenile justice reform is a
relatively new area for the organization and limited in-house experience and expertise
have had a negative impact on strategic planning, the development of indicators, the
monitoring and evaluation of implementation.
In general terms, results-based management needs to be strengthened … the objectives
of juvenile justice reform are often defined in rather vague terms, while indicators,
when identified, are mainly non-quantifiable and of limited use in measuring progress
towards the pertinent objective; data collection, monitoring and supervision have not
been given the necessary attention…
9.6.2 Lessons lost due to a weak evidence base

Five additional reports cite investments in reporting systems (two successful, three
unsuccessful). However, in these cases the evidence quality is poor to unacceptable.
Community Surveillance Systems
Synopsis
Two evaluation reports carry reliably supported evidence of community surveillance systems. One
additional report, also with a credible evidence base, described community surveillance to have been a
missing link.
The two reports of effective investments are:

Adolescent Girls’ Livelihood Program Evaluation, Early Marriage Prevention Sub-Project
(Bangladesh 2005)

Evaluation of Lihlombe Lekukhalela Child Protectors (Swaziland 2005)
9.7 Investment characteristics of working community surveillance systems
9.7.1 Strong intergenerational alliance at the community level
According to the Bangladesh report authors, it is the “presence and knowledge” of peers about pending
early marriage arrangements that allows the community to mobilize and advocate for a different
outcome. The “eyes and ears” of adolescents in combination with the solidarity and involvement of
respected community leaders gives this surveillance method a practical orientation toward action and
activism as the following excerpts illustrate:

Adolescent Girls’ Livelihood Program Evaluation (Bangladesh 2005)
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
111
o
o
Marriages are arranged within a matter of days of the proposal. The interlude is
characterized by a brief period of information gathering by the groom and the bride’s
family regarding each other and marriage negotiations. The sudden nature of the
marriage event along with careful management of information and the overall will of
the community for early marriage make a solution to the problem of early marriage
elusive.
This intervention uses the presence and knowledge regarding marriages held by peer
adolescents and the members’ networks to address the issue of early marriage while in
progress. It uses dialogues and discussions, without external intrusion, to de-motivate
marriages at least until the legal age of 18 is reached.
In the case of the “Child Protectors” program in Swaziland, the evaluation implies that an effective level
of surveillance had taken place on the difficult topic of sexual abuse, and that the “Child Protectors”
program could carry on the effort:

Evaluation of Lihlombe Lekukhalela Child Protectors (Swaziland 2005)
o The DPM’s office coordinated the survey involving community youth assessments.
These assessments highlighted the urgent need to protect children from all forms of
abuse, to help communities understand that sexual abuse of children is a major
contributing factor to the continuing HIV and AIDS epidemic…
o Qualitative results show that the area where all abuse cases are believed to have been
reported is one in which the LL was introduced by Umphaksatsi57 and welcomed by the
community. This group sees their role as children’s welfare in general not exclusively
child abuse issues. Many of their activities have been in conjunction with the local
schools as well as conducting home visits. They work closely with the Rural Health
Motivators in their area.
9.8 Constraints to mobilizing community surveillance systems
9.8.1 Lessons lost due to a weak evidence base

57
Two additional reports carry information on purportedly successful investments in this area, but
their evidence quality is poorly rated.
The evaluation report does not explain what entity this term refers to.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
112
Child Protection in
Emergencies
Chapter 10:
The dividends of good relations were illustrated at Bam. Pre-disaster partnerships between
UNICEF and MoH [Ministry of Health] helped in facilitating program take off.
Evaluation of Emergency Child Protection Programs in Bam (Iran [Lazo and Balanon] 2006)
Gender considerations should move beyond programmes for women and girl children and
analyse how programmes will differentially impact according to gender . . .
BAM Emergency Programme Evaluation 2003–2006 (Iran [Wiles et al] 2007)
Introduction and Overview
Child Protection investments in the context of emergencies is represented in the meta-evaluation in
eight reports. 58 Thematically these cover natural disasters (four) and war-affected children (four).59
These reports are unusually—in the context of this meta-evaluation—systematic in analyzing impacts
against international and UNICEF corporate standards.
Emergencies-centered evaluation reports were analyzed on the basis of five sub-categories:

Ability to execute rapid assessments that yield necessary information in a timely manner.

Overall management to deliver a holistic response on the Child Protection-related Core
Commitments for Children in emergencies.

Protection and care of separated/unaccompanied children and children associated with armed
groups.

Psychosocial support interventions are targeted and quickly established.

Capacities for prevention and response to gender-based violence are quickly in place.
In the aggregate, these investments:

Require more diligence in the area of gender analysis starting with needs assessments and
running through program concept, design, and implementation.
58
Three recent evaluation reports addressing UNICEF’s tsunami responses in Sri Lanka, Maldives, and Indonesia contain
information on Child Protection in emergencies. Since they are multi-sectoral reports (Child Protection is one of four areas of
analysis) they were not included in the meta-evaluation proper. However, where feasible given their very late acquisition in the
meta-evaluation process, findings and lessons learned can be commented upon. Titles and dates for these reports may be
found in Appendix Five.
59
An additional report (Evaluation of Alternative Care Programs, Sierra Leone 2006) addressed investments that had been
designed during an earlier emergency demobilization phase. The evidence base is too unclear for conclusions to be drawn,
although one of the objectives was to assess whether children are “still stuck” in interim care settings years after the
emergency phase had ended.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
113




Are much more systematic compared to the meta-evaluation’s non-emergency investments,
and are more likely to follow UNICEF and international expected standards of practice.
Highlight the significance of good pre-existing relationships with governments in shaping an
effective response to crises.
Face difficulties in forging sectoral cross-cooperation.
Highlight the importance of preparedness measures (e.g., trainings).
Ability to Execute Rapid Assessments
… among the cultural issues that came to light in the course of the Bam emergency program
were gender biases that excluded children who lost their mothers in the category of
“orphans”...
Evaluation of Emergency Child Protection Programs in Bam (Iran [Lazo and Balanon] 2006)
Synopsis

Mostly positively rated in five instances (Algeria, India, Indonesia, Iran, Thailand)60

Rated as negligent (Thai-Burma) and unproductive (Angola)

Ambivalently rated (Thailand post-tsunami)
FINDINGS
UNICEF can execute rapid
assessments but may not
always be asking the right
questions (four reports).
The investments that were most positively rated for programming quality in
this category suggest that UNICEF can execute rapid assessments but that
UNICEF may not always be asking the right questions. This has fostered some
unintended consequences including: neglecting the needs of disabled children, making erroneous
assumptions about cultural notions of gender and orphanhood, and making erroneous assumptions
about what types of services people would find to be most relevant.61
10.1 Investment characteristics of good rapid assessment capacities
10.1.1 Situation analysis has to ask the right questions
Four reports describe the difficulties summarized above and point again to the value of a good situation
analysis:
60
There were two reports on the Bam emergency, Iran.
The Sri Lanka Tsunami evaluation reports a relative de-emphasis in community participation in defining services and the
absence of a uniform assessment instrument to track the needs of internally displaced people (p.51). In the Maldives context it
was found that no rapid assessments of the needs of internally displaced peoples had been undertaken, but there was no
explanation as to why that was so (p.61).
61
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
114

BAM Emergency Programme Evaluation 2003–2006 (Iran [Wiles et al] 2007) — A comprehensive
needs assessment was missing:
o However, the evaluation team did not find evidence of a written vulnerability analysis,
economic, social or otherwise. This may explain why, for example, the needs of disabled
children appear to have been neglected.

Psychosocial Care for Children Traumatized by Terrorist Violence (Algeria 2003) — Failure to
harmonize approaches and operating assumptions:
o … while psychologists have created a functional network
“However, the evaluation team
around focal points in the several Wilayas, representatives
did not find evidence of a
of the public sector services were calling far [sic] ‘installation
written vulnerability analysis,
of a network through creating a committee of coordination.’
economic, social, or otherwise.
o From the side of UNICEF, decentralization is understood in
This may explain why, for
network terms, through supporting initiatives and activities
example, the needs of disabled
in the different Wilayas, and supporting the function of focal
children appear to have been
points, while the central management of the project
neglected.”
perceives decentralization in structural, policy and financial
terms.
Iran [Wiles et al] 2007

Evaluation of Emergency Child Protection Programs in Bam (Iran
EXCERPT
[Lazo and Balanon] 2006) — Investment lacked fundamental cultural
insights:
o … among the cultural issues that came to light in the course of the Bam emergency
program were gender biases that excluded children who lost their mothers in the
category of “orphans”… children who lost mothers were not registered and counted as
orphaned children. Discussions and training courses clarified the definition of orphans
and separated children and thus improved the registration.

Psychosocial Interventions Evaluation of UNICEF Supported Projects (Indonesia 2003) — A lack
of relevance followed an apparently missing or deficient needs assessment:
o The Eye Movement Desensitization and Reprocessing (EMDR)62 project was designed to
focus exclusively on problems resulting from exposure to trauma, but many
beneficiaries stated that psychosocial problems resulting from other issues—such as
dislocation, family problems, or financial problems—were a higher priority for them.
10.2 Constraints to rapid assessment capacities
10.2.1 Situation analysis ignored or missing
In two instances (both poorly rated for evidence quality), operational information that was on hand
seems to have been overlooked, but the evaluators do not provide details about the management
explanation for this:

62
Evaluation of Data Collection Project on the Impact of Armed Conflict on Children (SSRC/UNICEF,
Angola 2005)
“a therapy protocol to treat the effects of trauma”
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
115
o

A global inventory of existing information on children and armed conflict was conducted
in order to indicate knowledge gaps … However, since it was only delivered after the
end of the data collection phase, the insights from this work could not have been
applied at the project design stage. The evaluators have thus been unable to locate a
“survey of existing knowledge” that would have served to guide the creation of research
questions.
Evaluation: Psychosocial Rehabilitation Project (Thai-Burma 2006)
o Psychosocial programming, including the Training Manual for (Program) Social Workers,
was made even before the Baseline Needs Assessment was completed. There is no
indication that adjustments in the programming were made later.
In one instance the evaluator gives a conflicting picture of the rapid assessment capabilities:

Evaluation of UNICEF Thailand’s Post-Tsunami Psychosocial Initiatives Report (Thailand 2006)
o One commendable feature of the tsunami response was the prompt rapid appraisal of
the situation on the ground. Notably, local experts (one from a private hospital and a
team from the academe) were tapped and mobilized to provide basic information and
to assess the needs for assistance.
o In brief, the assessment during the emergency was not mindful of appraising the social
ecology and the cultural context for psychosocial responses. Efforts to check possible
involvement of peer groups, temple and church groups, cooperatives and community
associations as psychosocial counselors and helping agents were nil.
Overall Management to Deliver a Holistic Response
In terms of planning, there was a lack of a single overarching document or plan of action
that described the rationale for UNICEF’s programming decisions.
BAM Emergency Programme Evaluation 2003–2006 (Iran [Wiles et al], 2007)
Synopsis
The meta-analysis shows that inasmuch as a holistic response starts with starting the conversation with
government and civil society partners, UNICEF is well placed to deliver. At the same time, there is a
relative de-emphasis on quality assurance monitoring and follow through.

The four natural-disaster-related reports (earthquakes, Iran and India, tsunami, Thailand)
describe how good pre-existing relationships with government partners paved the way for an
effective immediate response.63
63
Evaluators of UNICEF’s Sri Lanka tsunami response found that UNICEF had not been proactive enough in addressing known
gaps in the government’s emergency child protection preparedness. Evaluation of UNICEF’s Response (Emergency and Initial
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
116


Two conflict-related reports (Algeria, Indonesia) describe building
highly sustainable ships while sailing them through rough local
capacities-building waters—efforts that Child Protection will want to
study in greater detail.
Of the seven emergencies-related investments, two (Angola, ThaiBurma) could be described as having been weakened through
management negligence,64 although improvements in results based
mechanisms are called for across the board.
FINDINGS
UNICEF's on-the-ground
networks were mobilized to
leverage government “buyin” in the early stages of the
crises (four reports, natural
disasters).
10.3 Investment characteristics of well managed, holistic responses
10.3.1 Essential: partnership-building in non-emergency times

BAM Emergency Programme Evaluation 2003–2006 (Iran [Wiles et al], 2007)
The authors emphasize the absolutely essential value of building good working relations with
the right partners in non-emergency times, a finding that is repeated by the evaluator of the
India Gujarat earthquake report (see below):
o UNICEF’s pre-existing relationships with government partners in its country programme
provided a crucially important asset in carrying out the Bam programme work.

Evaluation of Emergency Child protection Programs in Bam (Iran [Lazo and Balanon] 2006)
Pre-existing good relationships with the Ministry of Health and UNICEF paid “dividends”:
o The dividends of good relations were illustrated at Bam. Pre-disaster partnerships
between UNICEF and MoH [Ministry of Health] helped in facilitating program take off.

Appreciative Assessment of Psychosocial Interventions in Gujarat (India 2003)
UNICEF’s on-the-ground networks were mobilized to leverage government “buy-in” in the early
stages of the crisis:
o At the initial stage a strong ‘buy in’ from the Government of Gujarat was created. Their
initial skepticism about the psychological impact of the earthquake and its prime
importance was worked upon by providing information and consultations with all
sectors and civil society. The concept was discussed with officials from the education
department and modalities worked out. Suggestions given by the NGOs through
discussions were also incorporated into the program.

Evaluation of UNICEF Thailand’s Post-Tsunami Psychosocial Initiatives Report (Thailand, 2006)
Strong pre-existing relationships facilitated a prompt management response:
o UNICEF promptly mobilized its partners and resources during the emergency. UNICEF
and the Royal Thai government have been well credited for their prompt and timely
responses to the tsunami survivors such as the provision of drinking water. The RTG was
Recovery Phase), Sri Lanka (UNICEF Evaluation Office, May 2006). Evaluators found weak monitoring and follow up in Maldives,
Sri Lanka, and Indonesia.
64
The Thai-Burma evaluation is based on a Project of the Catholic Office for Emergency Relief and Refugees (COERR).
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
117
seen by the world as having taken a “take charge” attitude relative to the tsunami.
(emphasis in original)

Evaluation: Psychosocial Care for Children Traumatized by Terrorist Violence (Algeria 2003)
On-the-ground networking pays long-term dividends:
o The project has achieved an important objective of developing the capacity of human
resources. A large number of psychologists, psychiatrists, teachers, school counselors,
and general practitioners have been reached by the training component. The outcome is
the emergence of an informal network of professionals active in the area of
psychosocial care for children in general and with those affected by violence in
particular.
10.4 Constraints to management response capacities
10.4.1 Limited quality assurance mechanisms

BAM Emergency Programme Evaluation 2003–2006 (Iran [Wiles et al], 2007)
The Bam evaluators saw inadequacies on quality assurance monitoring and overall coherence:
o UNICEF Iran had an Emergency Preparedness and Response plan (EPRP) in place … The
EPRP was not used during the Bam response … it apparently lacked usefulness as a
practical document and was not used as a live document in the heat of the moment.
o In terms of planning, there was a lack of a single overarching document or plan of action
that described the rationale for UNICEF’s programming decisions, analysis, the
assessment of need, etc. … Monitoring activity tended to focus on outputs and on the
liquidation of cash assistance to government, rather than looking at issues of
programme quality.

Evaluation of Emergency Child Protection Programs in Bam (Iran [Lazo and Balanon] 2006)
Authors raise the matter of coherence, noting that results-based approaches had not been fully
consolidated:
o … use of the Results based Management (RBM) approach to programming was uneven.
10.4.2 Unresolved administrative, fiscal, coordination Issues
FINDINGS

Evaluation: Psychosocial Care for Children Traumatized by Terrorist
Violence (Algeria 2003)
Management did not get ahead of inter-sectoral coordination
difficulties:
o Communication and coordination of activities among the
different sectors and partners has been a major problem since
the start of the project.

Psychosocial Interventions Evaluation of UNICEF Supported Projects (Indonesia 2003)
UNICEF had no proactive management response plan in place, and fiscal planning was weak:
o The major constraints in implementation were linked to limited local capacities and lack
of experience with psychosocial programmes.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
118
For the West Timor
emergency work, programme
analysis and fiscal planning
were disjointed, causing the
cessation of a well designed
and run program.
o

Apart from some follow-up workshops, the Psychosocial Training Project was
discontinued due to a lack of funds after the introduction of the EMDR project. In
retrospect, this was regrettable, since the Psychosocial Training Project was an
appropriately orientated, cost-effective, and practical project with the potential to be
continued and expanded.
Evaluation of Data Collection Project on the Impact of Armed Conflict on Children (SSRC/UNICEF,
Angola 2005)
The administrative and managerial magnitude of the project seems to have been misperceived:
o Given the size and importance of the project, its pilot nature, the expectations
generated by it, the centrality of UNICEF’s role and the institutional and other difficulties
encountered along the way, the lack of a fully dedicated focal point in the early stages
was a serious managerial oversight. … A separate yet related problem has to do with the
genesis of the project at the international level and the related low level of buy-in
among UNICEF country offices and local stakeholders such as NGOs, governments, and
other potential user groups.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
119
Protection and Care of Separated/Unaccompanied
Children and Children Associated with Armed Groups
Synopsis
Reports available for the meta-evaluation show investments addressing protection and care of
separated/unaccompanied children and children associated with armed groups in ten instances
(five/armed conflict, five/institutionalization and reform). However, these investments were for postemergency, recovery, work.
With respect to emergencies, the Bam earthquake emergency response addresses protections to
separated/unaccompanied children.
10.5 Supports to separated/unaccompanied children
10.5.1 Pre-planning for family based care
As reported elsewhere in the meta-evaluation:

BAM Emergency Programme Evaluation 2003–2006 (Iran [Wiles et al], 2007)
o The impact of the joint SWO (State Welfare Organization) and UNICEF programme has
been significant. The majority of children without primary care givers are living with
extended family — “family based care.” …This is a major departure from the prevailing
practice, if not policy, in Iran.
o The experience has also influenced the way that the State Welfare Organisation views
childcare in the aftermath of disasters…
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
120
Psychosocial Support Interventions Are Targeted and
Quickly Established
What is emerging as a result of interventions in repeated crises in the Algerian context is an
informal network of professional mental health workers
Psychosocial Care for Children Traumatized by Terrorist Violence (Algeria 2003)
The psychosocial program was prompt and wide in scope … due to the
disaster-preparedness efforts of MoH [Ministry of Health] and UNICEF.
Evaluation of Emergency Child Protection Programs in Bam (Iran [Lazo and Balanon] 2006)
Synopsis
Seven of the eight emergencies-related reports revolve directly around psychosocial support
interventions.65
FINDINGS

A recurrent theme is that children and young people have not been
involved in design, implementation, or evaluation.

There seems to be some knowledge management-related confusion
regarding UNICEF’s investments in psychosocial supports to
adolescents.

Government and civil society partners’ uptake is substantial.

Teachers are over-burdened; out-of-school children do not get the
support that they need due to a first emphasis on school-going
children.
Government and civil society
partners show substantial
uptake of psychosocial
supports, but teachers are
over-burdened and out-ofschool children are
inadvertently excluded due to
a first emphasis on schoolgoing children (seven
reports).
10.6 Investment characteristics of targeted psychosocial interventions
10.6.1 Ability to mobilize quickly

Psychosocial Care for Children Traumatized by Terrorist Violence (Algeria 2003)
65
For the three additional reports on the tsunami response, the psychosocial support response was “strongly coordinated
among international and national agencies” in the Maldives (p. ii), “effective” in Sri Lanka (p.5). In Indonesia UNICEF “fulfilled its
coordination role well in psychosocial assistance” (p. iii). Evaluators found there, however, that a greater attention to gender
was needed in the context of psychosocial supports (p.iv).
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
121
The mobilization took place quickly, with sustainable partnerships emerging gradually:
o The project has developed through several stages from an initial response to the crises
to a partnership in development with public sectors and non-governmental organization
[sic]… What is emerging as a result of interventions in repeated crises in the Algerian
context is an informal network of professional mental health workers.

Evaluation of emergency child protection programs in Bam (Iran [Lazo and Balanon] 2006)
Uneven application of the psychosocial framework; generic use of the UN CRC:
o The psychosocial program was prompt and wide in scope … due to the
disaster-preparedness efforts of MoH [Ministry of Health] and UNICEF.
10.6.2 Providing a safe haven

BAM Emergency Programme Evaluation 2003-2006 (Iran [Wiles et al], 2007)
o The establishment of community counselling centres/drop-in centres became a
significant component of UNICEF’s psychosocial work in Bam.
10.6.3 Training develops intergenerational cadre

Psychosocial Interventions Evaluation of UNICEF Supported Projects (Indonesia 2003)
Good, efficient design, but the model apparently was not disseminated as a good practice:
o This project effectively implemented the main UNICEF principles on psychosocial
programming in emergencies.
o The strengths of this design in West Timor were that it was comprehensive and
multi-levelled—parents, teachers, and youth volunteers trained by the project provided
social support to beneficiaries, while counsellors and social workers also provided
counselling for those needing more specialised assistance.
o The Psychosocial Training Project is an easily implementable, efficient, and practical
project with proven psychosocial benefits for children and their communities. It is an
approach that should be continued and strengthened.
10.7 Constraints to psychosocial interventions
10.7.1 Inadvertently excluding out-of-school children

Appreciative Assessment of Psychosocial Interventions in Gujarat (India 2003)
Inadvertently exclusionary investments:
“The training process could
o The training process could address only the child who was
address
only the child who was
able to come to the school … The child who had
able
to
come to the school…”
experienced personal trauma and loss got overlooked in
the training process as the process focused more on a
generic response to trauma. The child who … dropped out
due to migration, trauma, disability, or adversity could
not be addressed by these skills. The most vulnerable
child then may have been missed out of the net and most
impacted.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
122
India 2003
EXCERPT
 Evaluation of UNICEF Thailand’s Post-Tsunami Psychosocial Initiatives Report (Thailand, 2006)
Inadvertently exclusionary investments:
o The psychosocial first aid activities effectiveness also suffered because there were no
follow-up activities. The activities also focused only on the orphans and some of the
severely affected. The needs of the other children who were also affected but did not
lose any family member were not addressed. Because of the focus on the schools, the
responses fell short of providing support to those who are not in school, to the
indigenous groups, and the migrant children.
10.7.2 Haphazard and non-integrative service delivery

Appreciative Assessment of Psychosocial Interventions in Gujarat (India 2003)
Haphazard and inadvertently exclusionary investments:
o …the effort to integrate the learnings from other organizations’ work to arrive at a more
complete psychosocial process, did not happen. There was no continuity of care. Most
people were met for an assessment interview and there were no therapeutic processes
in place. …the greatest constraint of the process was that for psychological intervention
to be therapeutic a degree of continuity and consistency was required in follow up.
10.7.3 Confusion on service provision to adolescents

BAM Emergency Programme Evaluation 2003-2006 (Iran [Wiles et al],
2007)
Authors say that UNICEF needs to build the case for service provision
to adolescents from the ground up, but the Psychosocial Training
Project in West Timor (2003, see next) seems already to have laid that
groundwork:
o The provision of services to adolescents is important but
challenging, especially where there is no obvious partner.
UNICEF needs to develop guidance on best practice in this
area.
FINDINGS
The Bam report (Iran 2007)
says that UNICEF needs to
build the case for service
provision to adolescents from
the ground up, but the
Psychosocial Training Project
in West Timor (2003) seems
already to have laid that
groundwork.
10.7.4 Inconsistent application of the psychosocial framework

Evaluation of emergency child protection programs in Bam (Iran [Lazo and Balanon] 2006)
Uneven application of the psychosocial framework; generic use of the UN CRC:
o The psychosocial program was prompt and wide in scope … due to the
disaster-preparedness efforts of MoH [Ministry of Health] and UNICEF.
o UNICEF should uphold the psychosocial framework as the primary framework in
emergencies per the Emergency Field Manual. This has not been consistently applied in
the programs and projects of UNICEF Bam. Further, in promoting the UN CRC in the
programs and projects, there is a need to go beyond the usual concepts of child abuse
and prevention, towards applying the UN CRC in case work, counseling, group therapy,
and even every day life and rights-based child rearing.
10.7.5 Missing needs assessment
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
123

Evaluation: Psychosocial Rehabilitation Project (Thai-Burma, 2006)
A missing needs assessment:
o Significant sources of psychosocial support to the children were absent. These include
awareness raising and psycho-education for the community, such as information on
stress and trauma and methods of coping or discussions related to issues of violence
and conflict, training parents to help children manage stress or advising persons with
particular symptoms to seek out local and traditional service providers…
10.7.6 Over-reliance on teachers

Evaluation of UNICEF Thailand’s Post-Tsunami Psychosocial Initiatives Report (Thailand, 2006)
An over-reliance on teachers:
o The psychosocial first aid activities were effective in a sense that they did maximize the
resources of the schools and the teachers were equipped with the needed knowledge
and skills in assisting in the provision of psychosocial support.
o The activities fell short of working with the significant adults in the children’s lives—
their parents, families, and the communities. Some parents were not even aware that
there were psychosocial activities conducted in the schools.
o The psychosocial first aid activities effectiveness also suffered because there were no
follow-up activities. The activities also focused only on the orphans and some of the
severely affected. The needs of the other children who were also affected but did not
lose any family member were not addressed. Because of the focus on the schools, the
responses fell short of providing support to those who are not in school, to the
indigenous groups, and the migrant children.
Prevention/Response Capacities for Gender-Based
Violence Are Quickly in Place
Synopsis
No meta-evaluation reports address the topic of gender-based violence in emergencies.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
124
Gender-Differentiated Needs Are Addressed in
Emergencies
Synopsis
Reports addressing emergency responses to natural disasters—earthquakes in Iran and India—have
cited gender analysis as insufficient.66 This is in line with the general meta-evaluation finding of a de
facto gender bias in programming, which may be reprised here:

Although 42 of 59 reports mention gender, the majority describe analyses that are superficial or
missing.

This neglect extends into the evaluation reports themselves, where it is not uncommon to see a
gender component listed as a programme or project objective with no follow up in the course of
the evaluation.
10.8 Investment characteristics of poorly gender-differentiated needs in emergencies
10.8.1 Ignorance of basic socio-cultural definitions

Evaluation of emergency child protection programs in Bam (Iran
[Lazo and Balanon] 2006)
Describes how the category “orphan” excludes children who lost
their mother in the Iran cultural context:
o As it came to light in the course of the FR [Family
Reunification] program, SWO [State Welfare Organization]
had to revisit and update its definition to become more
inclusive.
10.8.2 Programmatic gender apartheid, no impact analyses

FINDINGS
The Iran and India reports
cite gender analysis as
insufficient. This is in line
with the general metaevaluation finding that Child
Protection fails to
acknowledge significant
differences in the needs and
opportunities and outlooks of
girls and boys.
BAM Emergency Programme Evaluation 2003–2006 (Iran [Wiles et
al], 2007)
o …gender analysis was largely absent from programme documentation and UNICEF’s
response to the disaster lacks a well defined gender policy approach.
o Gender considerations should move beyond programmes for women and girl children
and analyse how programmes will differentially impact according to gender and be
designed accordingly, consideration being given to the needs of boys as well as girls.
66
This finding is consonant with findings in the three evaluations of UNICEF’s tsunami emergency and initial recovery phase
response.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
125
10.8.3 Failure to advocate for girls’ dignity and equity

Appreciative Assessment of Psychosocial Interventions in Gujarat (India 2003)
Inexplicably, the program managers allowed the popularization of
a song that clearly diminished women and anyone with
"One particular song that was
disabilities
very popular amongst
participants
was about a queen
o One particular song that was very popular amongst
who needed some things from
participants was about a queen who needed some things
the king. The refrain was to
from the king. The refrain was to make light of her needs,
make
light of her needs. …This
as these were so “contradictory” to her attributes. For
song was repeated over and
example, one stanza of the lyrics talked of a lame queen
over again through the various
asking the king for a cycle. The king asks her what she
training programs."
would do with a cycle, as she was lame. This song was
repeated over and over again through the various training
India 2003
programs and even performed in front of the assessors as
a snap shot from the training.
EXCERPT
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
126
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
127
Cross-Cutting Issues
Chapter 11:
…the managers do not have any financial incentives to implement a social policy aimed at
the development of foster families.
Deinstitutionalisation Project Evaluation (Belarus 2004)
Too many limitations in the way budget is allocated rather than the cost itself might affect
the sustainability of the programme. Targeted allocation of the funds is crucial and it should
more often reflect specificity of the local needs rather than specific programme budget lines.
Evaluation of the Program “Improved Mechanisms of Child Protection" (Bosnia and Herzegovina 2003)
Introduction and Overview
The meta-evaluation Terms of Reference specified a review and analysis of five themes that cut across
all areas of the Protective Environment Framework. According to the authors of the terms of reference,
“These themes are not direct elements of the PEF but are still critical for successful Child Protection
strategies.” The five themes are:

Costs

Partnership arrangements (who, what, where, when)

CP themes and sectoral good practices

Scale (micro, mezzo, macro)

Prevention or response
Information on these themes is sparse. For ease of reference, the state of play is tabularized here.
Details may be found in the respective sub-sections of this chapter.
Cross-cutting theme
Costs
Partnerships
CP themes and sectoral good
practices
Scale
Prevention or response
Sheeran Consulting, Inc.
Final Report, 15 May 2008
Findings/comments
Consonant with findings reported for monitoring and oversight,
cost analyses are almost non-existent.
Evaluators do not reflect on the seniority level of UNICEF’s key
partners
Juvenile justice and family based care have good correlations,
but nearly one-quarter of the evaluations report on a different
type of sectorality: CP investments need companion investments
in areas such as water and sanitation, and poverty alleviation
More than scale, the axis of programme success is management
accountability combined with government buy in.
The majority (40 reports) were response-oriented.
2008 UNICEF Child Protection Meta-Evaluation
128
Costs
Synopsis
As discussed already with respect to monitoring and oversight, management accountabilities for
investments to strengthen the Protective Environment are largely absent from the reports that are
available for the meta-evaluation. Cost oversight is also missing from most of the reports.” Cost-related
data was tracked in three categories:
FINDINGS

Ability to calculate any cost-benefit or cost-effectiveness ratios.

Ability to assess CP program cost “drivers” (e.g., what propels or
restrains costs).

How important is the known level of resources invested to program
success?
The findings are marginal in all three categories.
Two reports have
comprehensive cost-benefit
findings. Two reports carry
explicit findings on issues
that propel or restrain costs.
A common resource-related
constraint is "do-more-forless-itus."
11.1 Effective cost analyses
11.1.1 Cost-benefit analysis is extremely limited
Only eight reports provided cost-benefit information. The discussion is comprehensive in two instances.


Deinstitutionalisation Project Evaluation (Belarus 2004)
o . . . the managers do not have any financial incentives to implement a social policy
aimed at the development of foster families.
Juvenile Justice System Reform Impact Evaluation (CEE/CIS 2005)
o . . . no attempts to calculate or estimate the cost to society of laws and practices that
violate the rights of children in contact with juvenile justice have been done in any of
the four countries visited, and consequently it would not be possible to weigh the
investments made against the benefits realized as a result of them. Indeed, whether it is
appropriate to use such arguments in an area such as this is open to question.
11.1.2 Two evaluation discuss connections between costs and sustainability
Only two reports reflected explicitly on issues that propel or restrain costs (Laos Street Children, Bosnia
and Herzegovina Improved Child Protection Mechanisms), one positive and one negative.

Review of the Street Children Project (Laos, 2006)
o The project has achieved results with a relatively modest budget. Cost-effectiveness has
been aided by significant government contributions: the Ministry of Labour and Social
Welfare has provided the premises for the Drop-in Centre free of charge and assigned
two government staff to
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
129

Evaluation of the Program “Improved Mechanisms of Child Protection" (Bosnia and Herzegovina
2003)
o Too many limitations in the way budget is allocated rather than the cost itself might
affect the sustainability of the programme. Targeted allocation of the funds is crucial
and it should more often reflect specificity of the local needs rather than specific
programme budget lines.
11.2 Constraints to costs analyses
11.2.1 Rigid budget requirements constrain the program
o Bosnia and Herzegovina (see above, 11.1.2)—finds that “rigidity of the financing might
prove to be fatal” and that program needs are constrained
by rigid adherence to preset budget lines
"Organizations are working
under considerable resource
11.2.2 “Do-more-for-less-itus”
constraints, and yet they are
expected to perform a
o The resource/program success relationship was found to be
multiplicity of functions."
sufficient, explicitly or implicitly, in three reports and
constrained in 14 others. Typical constraints: “do-more-forGhana 2003
less-itus,” volunteer fatigue, expectations of community inkind supports are too high.
EXCERPT
11.2.3 Lessons lost due to a weak evidence base

Only four in 59 reports carry a serious costs analysis.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
130
Partnership Arrangements
Synopsis
The meta-evaluation Terms of Reference included tracking partnership arrangements in four respects:

Who (family, community, youth, international, ministry, professionals)

What (facilitate, implement, enforce, advocate)

Where (decision-making seniority, influence)

When (at what point in the project are partners coming on)
FINDINGS
Partnership profiles are unor under-reported. Average
seniority level is low to
middle.
Data addressing these themes was spare, imprecise, or scattered. The askSam
database will confirm, for instance, that only a handful of reports were found
to address specifics of What partners were doing (e.g. “facilitate,” “implement,” “enforce,” and
“advocate,” and none at all for the temporal issues (“When”).
This finding accords with findings from a recent review of UNICEF evaluation report quality, where
stakeholder analysis is one of six areas of lagging evaluation practice:
Areas remaining for improvement are the description of stakeholder participation and analysis
of relative contributions of stakeholders to the results…67
Indeed, it can be observed with respect to the meta-evaluation reports that this area of evaluation
practice, while not critically sub-standard, requires more than just a description of the information that
is required. It would appear that the rationale is lacking for evaluation teams and evaluation managers
to drive the case for strong analyses of partnerships, the relative areas of contributions, and the level of
seniority of the government partner. This must be so, otherwise the information on this topic would not
be so low a priority in the reports.
As a result of the issues flagged just now, the level of specificity that was hoped for on the matter of
partnerships cannot be delivered. However, the general picture is not entirely elusive:
67
UNICEF Evaluation Report Quality Review 2006
http://www.unicef.org/evaldatabase/files/UNICEF_Evaluation_Report_Quality_Review_2006.pdf The other areas noted for
improvement are “attention to eligible persons not reached by the project/programme, cost analysis, and a consideration of
HRBAP, gender analysis, and RBM.”
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
131
Partnerships
Comments
Who
-
Chapter 6, “Children’s Life Skills, Knowledge, and
Participation,” describes three good practice
investments in children’s and young people’s
participation. These are notably valuable
partnerships.
-
The Sudan Peace building evaluation describes
unusual partnerships with tribes.
-
Partnerships profiles are un- or under-reported;
evaluations are typically quite self-referencing,
particularly with respect to international peer
organizations.
-
Enforcement is a significant investment at the level
of training (e.g., judiciary, police), but not at the
level of data collection or monitoring.
-
Seniority is higher with regard to juvenile justice
investments, but evaluators or evaluation managers
did not apparently ask the question—consistently—
about the level of seniority that UNICEF is working
with .
Majority:
- Civil society organizations,
government ministries
Minority:
- Children and young
people
- International agencies
What
Majority:
- Implement, advocate
Where
-
When
n/d
Low to middle68
11.3 Investment characteristics of partnerships arrangements
Referring to the table, above:
11.3.1 Who: Civil Society Organizations and Government Ministries
11.3.2 What: Implementation and advocacy, not enforcement
11.3.3 Where: Low to middle-level seniority
68
Of course, in varied and multiple contexts UNICEF works with relatively high level officials: director-generals or even deputy
ministers. As such, the finding reported here may be an artifact of evaluation reporting which, as discussed, is not specific on
such matters per se.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
132
Child Protection Themes, Sectoral Good Practices
Synopsis
In the original terms of reference for the meta-evaluation, the analysis was set to monitor for
correlations between good practices and child protection themes. While the sample size of any area of
good practice is too limited to call out correspondences with much certainty, some trends can be
observed. For example, psychosocial programming emergency contexts is aligned to good practices
making family based care a reality (as an alternative to institutionalization for children without
caregivers). Psychosocial programming in emergency contexts is also well aligned with good practices in
working with professionals and relevant government ministries. A good practice has also been identified
linking girls’ livelihoods and child participation.
FINDINGS
However, a more pronounced issue is that Child Protection
investments are more likely to succeed when they are done in
collaboration with investments in other basic needs (e.g., water and
environmental sanitation, povety alleviation). This theme is sounded in
16 reports, with all but two possessing a good evidence base.69 Child
Protection investments in these instances included supports such as
vocational training (six reports) and basic needs like food or water
(nine reports).
One-quarter (27%) of the
reports suggest that CP
investments are more likely
to succeed in collaboration
with investments in other
basic needs (e.g., Water and
Environmental Sanitation).
In the majority of these reports (11), the evaluator is making the observation. In 5 of the 16 reports, the
evaluator is conveying stakeholders’ point of view.
However, Child Protection does not stand to gain a substantial insight into how this matter affects
outcomes and impacts because in the majority of the reports (12), the topic is mentioned more or less in
passing, as if the matter is widely taken for granted. The four reports where the evaluators provide more
than passing reference are: Tanzania 200470 and 2007; Ghana 2003, and Ethiopia 2003.
11.4 Inter-sectorality blind spots
11.4.1 Child Protection needs to consider socio-economic contexts
69
Searching askSam on 12a* brings up 23 reports, with 13 highly rated and 10 poorly rated. Searching on 3d* brings up 13
reports, with 9 highly rated and 4 poorly rated. Some of the same reports came up in both searches
70
Impact Assessment of the Most Vulnerable Children (MVC) Community Based Care, Support and Protection in Musoma Rural
(2004/021)
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
133

Report on Child Justice (Zambia, 2005)
o

Street Children and Street Mother, Partnership Programme (Ethiopia 2003) — Supports include
dwelling, skills training, and a micro-credit scheme:
o



On a general level it can be concluded that an increased awareness and sensitivity
towards children’s rights in the criminal justice system were observed in nearly all the
sites… This has, however, not translated into a general improvement in service delivery
… The problems experienced by the Zambian society make it increasingly difficult to
advance the rights of a relatively small group (child offenders) when poverty and
suffering is so vast and pervasive.
The sustainability of the outcomes of the component seems ambivalent in that it is only
a third of the street mothers that confirmed having the confidence of continuing with
their business after the withdrawal of external assistance while fifty-eight per cent
categorically stated they could not carry on their small business after the cessation of
the support. (Reporting stakeholder point of view)
Evaluation of Lihlombe Lekukhalela Child Protectors (Swaziland 2005) — Supports include basic
needs:
o It must also be understood that these issues are taking place in an environment of
poverty and marginalisation. People who have scarce access to basic resources at times
find it difficult to see how they can make a difference to the whole range of hardships
they witness in their community.
Bam Emergency Programme (Iran [Wiles et al] 2007) — Post-disaster Family Reunification,
offered food assistance:
o Economic empowerment is not part of UNICEF’s mandate, but it may be able to
advocate with the SWO [State Welfare Organization] to identify agencies (governmental
or NGO) that could work on underlying livelihood issues.
Evaluation of the program “'Improved Mechanisms of Child Protection” (Bosnia Herzegovina
2003) — Supports include mixed social welfare strategies:
o
Weak economic activity in almost all core municipalities aggravates already feeble social
systems.
11.4.2 Protection must address economic vulnerabilities

Protecting Street Girls and Ending Child Migration (Ghana 2003) — Supports include vocational
training; life skills:
o

Basic shelter needs to be met first. (evaluator reporting on focus groups with target
stakeholders)
Evaluation of the Prevention of Child Abandonment Project (Romania 2003) — Supports include
community services:
o
Child abandonment or neglect, (which) is determined mostly by the lack of certain
resources at the level of the family.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
134

Psychosocial Interventions in Gujarat (India 2002) — Post-disaster psychosocial support, teacher
training:
o Reduced economic options could also increase the risk of child labor and trafficking.
11.4.3 Child Protection should increase synergies with other areas

Thematic Evaluation of UNICEF’s Contribution to Juvenile Justice System Reform (Serbia,
Romania, Tajikistan 2006) — Supports include health (substance abuse programs); vocational
and life skills training; training social workers:
o

Grassroots Peace-building (Sudan 2004) — Support includes water:
o

It is perceived as necessary that UNICEF makes a greater effort to increase the synergy
of its work in juvenile justice with its work in other areas.
Provision of many water sources reduces conflict.
Education as a Preventive Strategy Against Child Labour (UNICEF Global 2003) — Supports
include vocational and life-skills education:
o
Education was viewed as the key strategy within a broader multisectoral approach
needed to break the cycle of poverty and disadvantage that maintains child labour.
11.4.4 Poverty negatively affects community participation

Mahzel Project for Social Reintegration and Protection of Disadvantaged Children (Eritrea 2004)
— Supports include basic needs of food, shelter, and clothing:
o

An Assessment of the Impact of Implementing the MVC Programme and the Potential for Scaling
Up (Tanzania 2007) — Supports include basic needs, education:
o

Drought and poverty have negative impact on community participation.
Community-based Child Protection Programme Evaluation and Review (Somalia 2006) — This
report is poorly rated for evidence quality:
o

Surprising is the fact that in those districts with highest community based support also
show highest support levels through other organisations.
Impact Assessment of MVC Support Programme Implementation (Tanzania 2004) — Supports
include secondary education, vocational training:
o

Impossible to earn community support and cooperation sans economic support.
(stakeholder point of view)
The evaluation found that in locations where the CPCM programme had been
implemented in conjunction with other UNICEF (also see p 21) interventions such as
water, education or health, or where communities had access to other services, there
was a higher level of participation and commitment to the programme.
Report on the Evaluation of the UNICEF-Supported Chikankata CBOSP & OVC Training Projects
(Zambia 2002) — This report is poorly rated for evidence quality:
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
135
o
Community participation has, however, been affected by the high poverty and hunger
levels caused by close to eight years of poor rain in the area. People have no basic
income, which they used to derive from various farming activities.
Scale
Synopsis
To the degree that credible evidence is based on action taken at certain scales, the meta-evaluation
attempted to track the scale of the programmes and projects, to avoid false implications about where
an investment is known to work. The more the lesson is visible across different scales, the more
confidence can be placed on it.
Meta-evaluation scale data was tracked in three categories:

Micro: any scale from district or less. Includes projects at community and municipality level
unless they cross a major size threshold. Covers most pilot projects.
o Meta-evaluation includes three reports at micro scale.

Meso: any scale between district and national, but most important when the scale of multiple
provinces is reached, which brings qualitatively different demands on project management,
cultural adaptation, etc. Can also apply to major urban projects.
o Meta-evaluation includes 35 reports at meso scale.

Macro: when the entirety of the target population is covered, including projects covering entire
nations but also those that cover entire groups even if geographically limited.
o Meta-evaluation includes 21 reports at macro scale.
The “meso” investments carry significant management accountabilities and strategic planning burdens.
With just a few exceptions, there are no stand outs-for good practices with well supported evidence
quality in this category.71 In the “macro” category, the meta-evaluation finds the following distinctions:
71

Direct government buy-in and involvement—the eight emergencies-related reports fall into this
category, as do well rated investments such as deinstitutionalization (Belarus), juvenile justice
reform (CEE/CIS, Yemen, Zambia) support for reintegration (Eritrea, Liberia) and mine education
(Global).

Greater management buy-in and involvement—in the majority of the well rated investments,
the technical and/or political demands are driven by a noticeable degree of
political/international immediacy in combination with direct government buy-in.
Exceptions are: Bangladesh Girls Livelihoods (2005), Cambodia Disabilities Project (2006), Laos Street Children (2006).
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
136
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
137
Prevention or Response
Synopsis
At least on a conceptual level, investments to strengthen the Protective Environment can distinguish
between prevention and response. According to the meta-evaluation data:

This distinction was found to co-exist in both concept and practice in evaluation reports of 12
Protective Environment investments. Thematically, the co-existence is noticeable with respect
to:
o Juvenile justice reform (CEE/CIS, Thailand, Yemen, Zambia)
o Gender and child protection (Bangladesh, Ghana, Guyana, Egypt, Ethiopia)
o Street working children (Ghana, Ethiopia)

In another 40 evaluation reports, the programs were presented as response-oriented, although
some contained such preventive elements as livelihoods trainings. In diverse instances these
may pave the way for preventive investments, such as in the case of alternatives to
institutionalized care and care for unaccompanied children or street children.

Prevention was an explicit objective in seven programme evaluation reports.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
138
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
139
Chapter 12:
Conclusions
Overview
In shaping the terms of reference for the meta-evaluation, the authors noted that while almost all
UNICEF-supported contexts are addressing child protection issue programmatically,
Not all country offices work confidently in child protection, and the evidence base remains small,
including in verifying to what extent and how the elements of the protective environment are
understood and addressed in the field by UNICEF and others.
The meta-evaluation affirms this outlook. With respect to both evaluation reporting emphases and
programming concept/design, the Protective Environment Framework is in the early stages of
becoming Child Protection’s second skin. This is predictable, given that the evaluations were
conceptualized and implemented at roughly the same time as the initial dissemination of the
Protective Environment Framework.
The retrospective application of the Protective Environment Framework to evaluations of its own
birth era, 2002-2007, shows how several areas of the framework’s programming theory are ripe for
further development. For example:

The framework’s foundation, systematization of national and societal protections, is
un/under-reported, as indicated by: (a) the minimal reflection on this topic in the evaluation
reports, (b) in programming concept/design, an insufficient analysis of pervasive inequalities
such as gender, and no evidence of contingency planning to address the difficulties of
obtaining baseline studies; (c) a consistent finding that poverty constrains community level
buy in to protection issues. Theorizing the causes and enablers of inequality, essentially, and
having programming strategies emanate from that, would appear to be a missing link.

With systematization at its core, Child Protection is a direct stakeholder in the global
networks that are forming to press the case, largely to donors, to deliver the means to
measure impact over the long term. Both the Center for Global Development Evaluation Gap
Working Group and the Network of Networks Impact Evaluation Initiative (NONIE) have
come into existence to address the “disincentives to financing impact evaluations such as the
lack of long-term funding and the difficulty in getting just-in-time funds for design.”72 These
disincentives hit home for Child Protection because 64% of the reports had no longitudinal
72
DESIGNING A NEW ENTITY FOR IMPACT EVALUATION: MEETING REPORT, Center for Global Development, 2007
http://www.cgdev.org/doc/eval%20gap/Bellagio_07_Meeting_Report.pdf The core NONIE network is comprised of the DAC
Evaluation Network, The United Nations Evaluation Group (UNEG), the Evaluation Cooperation Group (ECG), and regional
evaluation networks.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
140
data and the majority of the evaluators complained that there were no baselines in place for
the programmes that they were evaluating.

While rights based approaches to programming are implicit in the Protective Environment
Framework, in practice these are hardly robust: there are only four instances of verifiably
working partnerships with younger stakeholders, for example, and, using the Convention on
the Rights of the Child to guide programming rationale (for example, as a tool for
communities to improve their lobbying capacities with state actors) is almost non-existent.
The place of rights based programming in the Protective Environment –at the level of
concept/design and strategy/implementation -- is a rich topic for further development.

Important achievements that represent the basic rationale of the Protective Environment
have yet to be marketed as such, including (a) the significance to systematization of
professionals’ capacity building, (b) contributions to strengthened social safety (e.g. shifts
towards community-based care for children), and (c) a strong inter-generational alliance
combined with community education opportunities to facilitate an end to harmful traditional
practices. A strategic communications plan for making these valuable investments very
visibly identified with the Protective Framework, that is, as central to its raison d'être, will be
helpful in building general literacy with the framework’s programming theory.
Picking up from the introductory chapter’s discussion of limitations and biases, in what sense is the
meta-evaluation a baseline for Child Protection? Inasmuch as the fundamental requirement of a
“baseline” is to show a specific performance measure, findings for the programming practices and
evidence quality meet this requirement. The meta-evaluation also provides lessons about the Protective
Environment Framework in practice, for instance: quality technical inputs such as situation analyses
cluster around government more than community-based stakeholders. There is also enough evidence of
a repetitive nature to suggest that the findings concerning monitoring can be generalized with
confidence.
For individual thematic areas of Child Protection, for instance, child labor or early marriage, findings
must be scrutinized for their applicability.
Organization of this chapter
Conclusions are presented for each element of the Protective Environment, Emergencies, and CrossCutting issues. Only findings that serve to illustrate themes and issues that are common throughout the
meta-evaluation are brought forward into the conclusions discussion.
Child Protection stakeholders met with the evaluator for a one-day workshop at UNICEF global
headquarters. The purpose was to add depth and context to the conclusions and to generate an initial
set of recommendations. Stakeholder inputs on the conclusions are noted, below. Initial
recommendations may be found in Appendix Nine of this report.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
141
Conclusions
Programming and Evaluation Practices
12.1.1 Crippling programming and evaluation practices are taken for granted
For reasons that are not novel among development professionals, organizations like UNICEF are obliged
to accept that some of the time, programming planning and implementation must transpire on a
contingency basis. Political instability, donor fiduciary delays, natural disasters, any of these and more
can shift the frame of carefully planned investments and suddenly strategic planning turns into the ship
that has to be sailed while it is being built.
That said, the meta-evaluation finds a surprising absence of urgency around certain programming and
evaluation practices that, although fundamentally crippling in terms of the operational intelligence they
bury, are nevertheless treated on the order of “business as usual.” Three can be named: (a) the scarcity
of time series information (e.g. 64% of the reports have no longitudinal data), (b) an alarming overreliance on Key Informant Interviews, and (c) an under-emphasis on sampling and scale (e.g. 80% lack a
sampling plan and most are quite small in reach).
It should be emphasized that these customs are not characteristic of Child Protection or UNICEF alone,
indeed, the whole of the global evaluation establishment is wrestling with similar irregularities and
irrationalities. Still, if the meta-evaluation contributes to any one change, it would be to press Child
Protection donors and partners to engage with fresh eyes on the causes of and fixes for these three
areas of programming and evaluation practice.
12.1.2 De facto operations assumptions may be limiting the ambition of Child Protection evaluations
The meta-evaluation is inconclusive about what drives the three programming and evaluation practices
discussed just now. For instance, there may be a working assumption that Child Protection impacts can
be tracked only within a localized frame. Or there could be a bias against statistical analysis, as if
qualitative issues cannot be tracked at scale. Or, funding could be limiting the scale of the evaluations.
Whatever the reason, the ambition of the evaluations seems unnecessarily limited.
12.1.3 Improving analysis of institutional contexts is a necessity
The reports provide insufficient analysis of institutional contexts (e.g. donor requirements, host
government sensitivities). Without information about upstream and institutional factors, such as UNICEF
global managerial priorities and donor commitments/expectations, the frame of reference for
interpreting programming decision making, for example, the absence of baselines and monitoring, is
narrow and un-illuminating.
12.1.4 Need to re-examine quality control, leverage, at the at the sub-programme level
The meta-evaluation finds that the closer to centralized political power, the more “clout” UNICEF Child
Protection appears to find and utilize and the better its programming design inputs and technical
outputs. UNICEF’s downstream work shows less “clout” and quality. Most probably this is a by-product
of serial outsourcing, from one civil society organization or sub-regional ministerial office to another. It
does raise a question as to whether sub-contracting arrangements taking place within a context of good
strategic information on where/how best to apply leverage.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
142
12.1.5 Difficult to reconcile gender findings with corporate commitments
The evaluation reports indicate that programming is not differentiating between the needs and
opportunities of boys and girls. The disadvantages that girls face across diverse institutional and
political environments are buried, the findings suggest, under a one-size-fits-all programming template.
This finding perhaps can be considered in the context of the findings regarding the relatively weaker
quality of technical inputs and outputs at the sub-programme level. However, the meta-evaluation is
largely inconclusive as to how such a gap can be reconciled with UNICEF’s corporate commitments to
gender sensitive programming and with Child Protection’s clearly articulated commitments in that
regard.
12.1.6 The askSam database has been an indispensible tracking tool
The significance of askSam to the meta-evaluation has been overshadowed somewhat by the overall
focus on the main report and findings. Yet, askSam has been indispensible in keeping track of any of the
100+ unique characteristics that were set for analysis (see the Protective Environment Framework Metaevaluation Classification System, Appendix Two). And used synergistically with Excel, where ratings
assigned to the classification system characteristics were stored, askSam allowed for statistical and
textual evidence to be contrasted side by side. It has been a straightforward process, for instance, to
map a trend discovered through the power of AskSam textual search, find the data points in Excel, and
perform further statistical analysis as evidence. This allows a field operator to know how well a
programme did on a specific criteria and then to read about what they did. The opportunity for learning
is greatly enhanced.
A qualitative assessment of askSam is planned as a separate deliverable for this project. askSam is a
simple, searchable, database. It should not be difficult to deepen technical capacities at Headquarters as
a bridge to introducing a wider field utilization of askSam.
Strengthening Government Commitments
12.2.1 How the convening organization convenes
It would appear from the findings that for Child Protection UNICEF acts as a convening organization at
the national level with (a) a good track record in creating contexts for new partners to come together,
and (b) excellence in profiling emerging issues. In this latter regard, providing information and technical
analysis through situation reports has been an area of effective investment. This is a strong area of
leadership for UNICEF Child Protection.
12.2.2 Disparity in provision of technical information inputs
Anticipating a theme that has manifested itself in several contexts in the meta-evaluation, it can be
noted here that the technical inputs mentioned in 12.2.1, above, have no analogue where civil societylevel and community-based stakeholders are concerned. Of course, work with government actors is
work with government actors, hence, what is the rationale for raising the matter of non-alignment in
information-related technical supports to civil society? The rationale is simply this: to take note of a
disparity that seems to constitute a distinctive difference in the way that UNICEF does business with
varying types of stakeholders.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
143
12.2.3 Need to clarify the reasons for dis-investments in enforcement
Although UNICEF finds and uses its “clout” most effectively with government partners at the national
level, the meta-evaluation finds consistent gaps in enforcement and monitoring. These may reflect an
undercurrent of instability in government commitments to the Child Protection agenda, but the metaevaluation is not conclusive on this point. It raises the question of whether enforcement weaknesses are
basically the price of doing business on Child Protection, or whether UNICEF could better leverage
international agreements on aid effectiveness (e.g. Paris Declaration) in forging these agendas with host
governments.
Legislation and Enforcement
12.3.1 Evaluators do not address institutional contexts, re constraints to enforcement
Mirroring the findings for “Government Commitments:” (a) UNICEF is doing a good job in driving
technical inputs for legislative change, and (b) UNICEF almost completely de-emphasizes monitoring
legislative investments and there are no instances of investments in enforcement mechanisms at the
state level. The evaluations do not provide information as to how institutional factors such as
government sensitivities and donor preferences could be in play here.
12.3.2 A de facto “backdoor” enforcement strategy is in operation
In the void between legislation and enforcement, UNICEF is investing in building the capacities of
professionals and community level advocates to recognize and respond to Child Protection issues.
According to the meta-evaluation findings, this allows Child Protection to work around host
governments’ limitations on enforcement or around other institutional constraints that may be in play.
12.3.3 Lopsided supports to duty bearers versus rights holders
There is no example of an investment to support duty bearers in developing protection legislation being
combined with capacity building of rights bearers to know and to claim their rights. That is, technical
inputs are clearly aligned with legislation whereas enforcement is largely at the level of moral
persuasion. The meta-evaluation finds that investments in informational efforts such as situation
analyses are a priority for state-level actors in shaping a legislative agenda, but that these are not
perceived to be forceful, or potent, in shaping an enforcement climate.
Elsewhere (see below), a similar operational apartheid appears to be in play, where the duty bearers at
the state level receive specialized, precise, technical, inputs, while rights holders receive looser,
generalized, and nontechnical “awareness raising.”
Attitudes, Traditions, Customs, Behaviors, Practices
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
144
12.4.1 A wholly replicable methodology is not seeing much marketing
Exemplary programming addressing two difficult issues, ending child marriage and female genital
cutting, does not seem to be recognized in Child Protection as just that – exemplary. Yet, this
programming utilizes a methodology that is both replicable and that sits near the top of the metaevaluation for the quality of the evidence base. There would not seem to be a mechanism in place as
yet to call out the singularity of these accomplishments and to use them to further understandings of
the programming theory that underlies the Protective Environment Framework.
The methodology pairs a community level intergenerational alliance with educational opportunities
on basic rights-related issues. It is implemented in communities over extended periods, with a core
issue at stake (e.g. ending child marriage). The sample size for this exemplary programming is exactly
two reports.
12.4.2 Models/theories of societal change may be missing?
The remaining two reports in this category carry vague assertions about changing practices, and both
have a weak evidence base rating. Moving from descriptive to interpretive models of societal change
may be an area where evaluators need to be pressed for specifics and where training on approaching
these matters may be useful to Terms of Reference designers.
Open discussion/engagement with Child Protection issues
12.5.1 Need to clarify practical/theoretical differences: “information” and “awareness”
As discussed above, the primary finding is that Child Protection –in programme concept and evaluation
reporting—is not delineating differences in terms that are meant to play distinctive roles in building a
protective environment: “knowledge,” “advocacy,” and “awareness,” are used interchangeably. One
hypothesis as to why these distinctions are blurred is that the basic place of information in triggering
chains of knowledge and awareness has been downplayed or otherwise not specified. As discussed
earlier, however, this distinction seems more apt to be blurred with respect to civil society as opposed
to state actors, for whom delivery of information through situation reporting is a high priority.
12.5.2 Rational and place for programme communications (“C4D”) are unclear
Strategic communications (Communication for Development) is a missing link in the programme
descriptions in all 59 evaluation reports. This absence is also aligned with findings for building capacities
of those closest to the child, which tracked 31% more activity in the “response” category than in the
“recognition” category
According to participants in the meta-evaluation workshop, the absence of investments on C4D reflect
sampling biases, and not, for example, (a) a weak emphasis on strategic communications in Child
Protection, and/or (b) a need for technical support for design, monitoring, and implementation. It is
noteworthy that the findings for building capacities of those closest to the child.
12.5.3 Rights based approaches have an unclear status
Good outcomes are seen where the Convention on the Rights of the Child (CRC) has helped to legitimate
programming investments, but in an equal number of evaluations the CRC’s status as more than a
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
145
ceremonial role is weak. It is possible that, at the end of the day, rights based approaches and the rights
framework are not widely applied and that these are also less well developed areas of the Protective
Environment Framework programming app.
Children’s Life Skills, Knowledge, and Participation
12.6.1 Rights based approaches are not a central emphasis
The findings suggest that child participation, a basic principle of rights based approaches to
programming, is not an elemental component of Child Protection investments as represented in the
universe of materials that was available for the meta-evaluation. Programming is found to be top-down
and particularly exclusionary at the design stage; evaluation reporting on this topic is almost accidental.
In just four reports of 59, child protection stands to obtain information on what works and what doesn’t
regarding child participation in the service of building a protective environment.
A call for information on child protection programs with participatory components for younger
stakeholders would be helpful in clarifying the actual status of child participation within Child
Protection. It would also be helpful to review evaluation terms of reference to find out whether
participatory efforts have been dropped inadvertently as topics for impact assessment.
12.6.2 Evaluating Life Skills investments has not been a priority
The findings show that Life Skills investments are unmonitored, missing, or un-ambitious; just about
one-quarter of the evaluations described these investments in any case. The status of these
investments in an overall evaluation plan seems to indicate that their status vis-à-vis programming is not
central.
Building capacities of those closest to the child
12.7.1 Professionals get more professional inputs from UNICEF?
As noted, across both “response” and “recognition” categories, professionals’ capacity building saw the
highest programme practices quality, with 19% more overall quality than investments in communities’
capacity building. One hypothesis is that training requirements for professionals are more cut and dried,
technical, and hence more easily deliverable (and evaluable) than, say, the requirements for community
based child protection advocates. Or perhaps the delivery mechanisms are at issue, whereby UNICEF has
more direct control in respect of professionals’ capacity development than communities. This is a topic
for further research.
12.7.2 There is no consistent standard for working with volunteers
Findings concerning community engagement and capacities development/training suggest that when
these work well, they are powerful tools. A consistent standard about how best to utilize volunteers,
however, is missing, and it is unclear why trainings –when they can be so powerful and additive—might
ever be under resourced or poorly orchestrated. As noted, the capacities building component of the PEF
points up the importance of training programme content and delivery, such that length and/or quality of
training/support make the difference, not just a difference
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
146
12.7.3 The front end information needs of civil society partners/stakeholders are taken for granted
As noted, data addressing “Major actors are facilitated in recognizing child protection needs” suggests
that raising awareness about rights and needs of vulnerable children is often an afterthought, secondary
to actionable objectives of strengthening the capacity to respond to needs. This is substantiated further
by the evidence of 31% more investment activity in the “response” category than in the “recognition”
category. What is not clear is how Child Protection came to this, what understanding of information
needs led to the secondary emphasis on recognizing child protection issues as opposed to responding to
them.
12.7.4 Protective Environment programming theory is unclear about the status of building capacities
of families
As noted, data addressing “Strengthened capacities for responding to child protection needs” suggests
that “top-down” (vs. “bottom-up”) strategies predominate in capacity-building CP programmes. That is,
families are the least frequently facilitated in the “response” category, professionals are the most
frequently targeted. This could be a simple artifact of the evaluation reports that were available for the
meta-evaluation, or it could be that the place of families in the Protective Environment Framework is
still somewhat under-theorized.
12.7.5 Evaluators do not address the systematization of capacities building
As noted, out of 59 reports, 41 addressed systematization and/or integration of the programme under
evaluation. Note therefore, that nearly 20% of the evaluations failed to even address the sustainability
of capacity development. And, of the 41, only 24 are rated with credible quality of evidence. At the very
least it would thus appear that at the level of evaluation reporting this elemental component of the
protective environment has not yet been sufficiently internalized. It is unclear whether this conclusion is
applicable also to programme design/concept.
12.7.6 Best investment towards sustainability: training professionals
As noted, professionals’ response capacity building is linked to integration/systematization of Child
Protection. This may be a de facto area of emphasis in Child Protection or a coincidence of the reports
that were available for the meta-evaluation. The flip side is notable: where communities’ response
capacities are concerned, issues concerning ownership and volunteer fatigue constrain sustainability.
Delivery of necessary, child friendly, services
12.8.1 Evaluation terms of reference: out of sync with the PEF
The delivery of child friendly services is the most densely represented area of investment in the metaevaluation, with 62% of the strategic planning-related practices rated as good quality. However,
evidence quality is a significant constraint: of 27 reports possessing information on service delivery
efforts only 15 rated credibly in terms of their evidence base. Evaluation reporting generally focused on
strategic planning irregularities such as gaps in management responsiveness to bottlenecks and an
inability to gather reliable contextual information. Disentangling who was supposed to do what took
more precedence than who accessed what services to what impact. This could be expressive of an
evaluation culture that is essentially “glass half empty.” It could also be a reliable indicator of the sea of
contingencies and improvisational realities of service delivery.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
147
Misunderstanding programming contexts, regularly it would seem with respect to gender differences, is
found to undercut service delivery on a consistent basis.
12.8.2 Child Protection vastly understates its successes
Though relatively limited in number, the five instances of shifts towards strengthened social safety nets
have an arguably muted presence in Child Protection communications and marketing. They are treated
as business as usual when, in fact, they would appear to demonstrate programming and planning
practices that work: taking the long view on capacities building of government and professional
partners, good stewardship of and investments in situation analyses.
Monitoring and Oversight
12.9.1 No explanation for monitoring findings, they are totally inexplicable
Monitoring and Oversight is the most deficient area of the Protective Environment investment portfolio.
At least 90% of the reports that were available for the meta-evaluation contained complaints about
monitoring having been insufficient or absent altogether. Investments that align to centralized political
power –for instance, relating to enforcement of a national plan--are among the least likely to be
monitored for quality assurance. But the meta-evaluation also finds a lack of depth in evaluation
reporting on the matter of causation: evaluators offer no interpretations at all as to factors that might
explain the absence of programming addressing monitoring and baselines.
Monitoring capabilities as represented in these reports do not differ significantly from the global norm
in that they are processes in need of support. A concern, then, is not only about knowledge
management (how does the Child Protection community learn about and build on previous efforts), but
also about pragmatism (how CAN evaluation X bring our practice forward?) and even marketing (what IS
new and innovative about this programme?). The meta-evaluation finds this area of programming
practice to be critically sub-standard.
Child Protection in Emergencies
12.10.1 Gender and Psychosocial findings merit further study
The meta-evaluation had just a handful of emergency-related reports to go on – eight in total. The
findings are generally consistent with those cited already for the main components of the PEF. And (a)
psychosocial investments seem to be delivering a depth and uniformity of operational information, with
consistent findings that teachers are overburdened and out of school children are underserved, and (b)
gender analysis is insufficient throughout. The cornerstone of good response capacity seems to be
UNICEF’s cultivation of government partners and other significant stakeholder in non-emergency times.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
148
Cross-cutting issues
12.11.1 Lack of clarity on what makes a cross-cutting issue cross-cutting
Two cross-cutting issues, gender and poverty alleviation, were more consistently at issue in the metaevaluation than all five that were preset. As both address the foundational principle of the Protective
Environment, moving towards systemic changes in national and societal policies, this reinforces the
meta-evaluation’s general impression that systematization is not explicitly enough positioned as a driver
of programming logic. On the matter of partnerships, evaluation managers are not prioritizing analysis
of partners’ seniority levels as these are vastly un/under-reported.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
149
Appendix One:
1.
Terms of Reference
Purpose of Assignment: Conducting a Child Protection Meta-evaluation
UNICEF’s work in the area of child protection has evolved rapidly and considerably for over two decades.
Programme design in child protection has been shifting from projects for specific groups of children in
especially difficult circumstances to a more comprehensive approach of supporting change in social and
institutional environments. The approach entitled ”protective environment for children” was introduced
during the 2002–2005 MTSP for UNICEF’s work on child protection. The use of this approach leads child
protection programmes to become more strategic and better integrated in all UNICEF work.
All but one of the 157 UNICEF-supported countries, areas, and territories address child protection issues
programmatically. At the same time, not all country offices work confidently in child protection, and the
evidence base remains small, including in verifying to what extent and how the elements of the
protective environment are understood and addressed in the field by UNICEF and others.
In order to expand and strengthen child protection programming towards achieving the targets of the
current MTSP 2006–2009, an authoritative basis for confirming effective protection interventions is
needed.
The aim of this consultancy is therefore to determine which Child Protection programming strategies
and project interventions have been proven to work. Proven to work means that the combination of
good design, monitoring, and quality evaluation allows UNICEF to state that certain approaches—if well
executed—are likely to result in measurable improvement in resisting violations of children’s rights to
protection, and to enhancing the resiliency and social networks that will foster strong child and
community development. The evidence will be examined against the goals contained in the Protective
Environment Framework (PEF).
This evaluation is intended to contribute to UNICEF in three main ways:
1. To provide an evidence base for: a) global lessons sharing with partners and b) global, regional,
and national programming and advocacy efforts.
2. To inform any potential revisions of UNICEF’s Child Protection goals, especially those contained
in the Medium Term Strategic Plan 2006–2009, which is scheduled for a Mid-Term Review in
early 2008.
3. To support the management accountability for capturing lessons learned and examining the
impact of UNICEF’s work in child protection.
Other direct and indirect expected benefits include the following:
 Identification of programmatic elements for inclusion in child protection training and guidance
 Refinement of programmatic indicators/proxy indicators
 Identification of specific needs in knowledge management strategies, policies, and management
of financial and human resources
 Better internal and external collaboration on child protection
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
150

2.
Generating lessons learned for any future reviews of child protection research and evaluation
methodologies, with a long-term goal of standardizing methods where possible to ensure valid
and reliable data is always generated.
Details of SSA
Section:
Level:
Duration:
Supervisors:
Location:
3.
Child Protection/Programme division
Comparable to P5 level
21 May 2007 – 21 December 2007
Shirin Nayernouri, CPS
Sam Bickel, Evaluation Office
Consultant is expected to work from home with not more than 2 visits to UNICEF NY
Major Tasks and Deliverables
1. Mobilization – A compilation of electronic copies of Child Protection evaluations conducted by
UNICEF from 2002–2007. This deliverable will be provided by UNICEF.
2. Evaluation Quality Review – A completed evaluation report rating form prepared electronically for
100–120 evaluations.
3. Tools Development and Piloting – An inception report to include:
 An agreed and refined set of themes to be examined
 An agreed method for recording and/or coding information found within the evaluations
 An updated workplan for completing the consultancy
4. In-depth Annotations of the Evaluations – The deliverables will include:
 Estimated 70 evaluations completely annotated per protocols developed in Stage 3. The
annotations will be in electronic form within the documents, and will be linked in a
database that allows users to browse a hyper-linked index that sends them to their topic
of interest.
 Draft gap and thematic coverage analysis to circulate for comment to CP.
5. Analysis and Draft Report – the deliverables will include:
 Early draft summaries of findings; the findings should include feature boxes, a synthesis of
evidence, and a summary of evidence gaps.
 A comprehensive draft report prepared following feedback from UNICEF on the draft
summaries.
6. Final Report – The deliverables will include:
 Presentation of draft report, day long workshop, HQ
 The final evaluation report to include the thematic chapters, as described in phase 5, and
the following additional elements:
 A complete record and copy set of all prior deliverables not already transmitted to
UNICEF, including relevant notes and data files.
A report on the process of the evaluation, strengths and weaknesses of the
methodology, and other tentative process recommendations to ensure better
knowledge management of CP evaluations and evidence in future years. This report
will be from 5–10 pages long, and is an elaboration of the normal chapter in a Final
Report on Methodology. The goal is to self-analyze the methods used as a Lessons
learned for UNICEF to consider in the construction of future exercises of this type.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
151

Stand-alone Power Point presentation of no more than 20 minutes, including all speaking
text as noted. This is to permit UNICEF staff to deliver the findings in subsequent venues in
the words of the consultant.
4.
Qualification or Specialized Knowledge/Experience Required

Advanced university degree in Social Sciences

Proven ability to conduct detailed research or meta-evaluations

Strong technical evaluation skills

Good knowledge of child protection issues

Excellent analytical, planning, and writing skills

Experience with project management

Familiarity with UNICEF programmes

Good communication and interpersonal skills

Teamwork
5.
Timeframe: This work is expected to take eight months starting 29 May 2007.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
152
Appendix Two: Protective
Environment Classification
System
Introduction
As stated by UNICEF, “The Protective Environment is a web of interconnected elements which
individually and collectively work to protect children from violence, abuse, and exploitation.”73
As stated in the main body of the report, the Protective Environment was introduced as a
programming framework for Child Protection in the context of the 2002–2005 Mid-Term
Strategic Plan of UNICEF. This appendix lists and annotates where necessary the eight elements
that make up the Protective Environment, plus six additional categories that the UNICEF Child
Protection Section and the UNICEF Evaluation Office wished to include in the meta-evaluation
analysis.
Part One: Protective Environment
1. Government commitment to fulfilling protection rights of children
a. National plans of action, policies, budgets
b. Initiatives to strengthen capacities for partnerships, advocacy, implementation74
c. Commitments emphasize prevention, response, or remediation
prevention = pr
response = rsp
remediation = rmd
d. Other
2. Legislation and enforcement
a. Legal development/reform reflecting child rights principles
b. Stengthened enforcement
c. UNICEF understands and has capability to address bottlenecks and opportunities75
d. Other
3. Addressing harmful attitudes, customs, behaviors, practices
a. Insights into local power structures, decision making networks
b. Tactics are contextualized and stratified
73
http://www.unicef.org/peflash/pe2.html
Implementation-related advocacy was a rare investment.
75
As there is no code addressing funding constraints, ”10c” alone is not sufficient, 2c is coded with 10c in that
instance (e.g., resource found to be constraint that has not been addressed)
74
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
153
c. Changed attitudes and practices
d. Other76
4. Open discussion and engagement with child protection issues
a. Increase in public knowledge about the issue
b. Open discussion on sensitive issues
c. Use of media for advocacy work
d. Formal linkages to CRC are visible, part of the program
e. Children feel able to speak and/or act more freely about protection concerns
f. Other
5. Knowledge and skills of children to protect themselves
a. Insights into what children/young people know/need to know
b. Relevant information/services are provided (e.g., rights trainings, free counseling,
condoms distribution, school curricula, child media, peer-to-peer activities)
c. Children/young people are facilitated to get involved (e.g., establishment of youth
networks)
d. Other
6. Capacity of those closest to the child
a. Major actors are facilitated in recognizing child protection needs
Family
Community/CSOs
Professionals
b. Strengthened capacities for responding to child protection needs
Family
Community/CSOs
Professionals
c. Capacity development is becoming systematized, integrated, not ad hoc
d. Other77
76
3d is used:

To tag observations that shed useful/important light on the local context, but that don’t necessarily come
from the project itself, but rather from the evaluator (although it’s not always possible to disentangle the
two); or,

To tag observations that indicate how the local socio-political context might complicate project goals (as in
the case of Somalia and Swaziland, where extreme poverty overwhelms children's rights in the mind of the
community).
In other words, 3d refers to the general socio-political context, rather than specific power structures/decision-making
networks (such as Sudanese tribes), the appropriateness of tactics (such as targeting tribal elders), or the change the
project has created.
77
6d has been used to refer to capacity building “management” issues that aren’t specific to UNICEF (8f). This is used
primarily for the Cambodia Social Work Training Evaluation (2005) to tag text that discusses the problems of the MSST
programme (Management Social Service Training): problems relating to curriculum, staff, pedagogical methods,
funding, etc. In other words, this is a category of logistical/programmatic issues that are different from mainstreaming
and sustainability (6c) and that aren’t just to do with UNICEF (8f).
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
154
7. Access to services
a. Provision of necessary and child-friendly services
b. MVC & Families facilitated to access these services
c. Strengthened social welfare systems
d. Other
8. Monitoring and oversight of child protection issues
a. Monitoring systems including data collection on CP issues
b. Establishment of participatory and locally-based mechanisms
c. Reliable reporting systems are established
d. Establishment of community surveillance systems
e. Information on sensitive subjects, e.g., closed
f. Other78
9. Child protection in emergencies
a. Ability to execute rapid assessments that yield necessary info in a timely manner
b. Overall management to deliver a holistic response on the CP-related Core Commitments
for Children in emergencies
c. Protection and care of separated/unaccompanied children and children associated with
armed groups
d. Psychosocial support interventions are targeted and quickly established
e. Capacities for prevention and response to gender-based are quickly in place79
f. Other
Part Two: Cross-Cutting Issues
10. Costs
a. Ability to calculate any cost-benefit or cost-effectiveness ratios
b. Ability to assess CP program cost ‘drivers’ (e.g., what propels or restrains costs)
Yes = y
No = n
c. How important is the known level of resources invested to program success?
Yes = y
No = n
11. Partnership Arrangements
a. Who
Family = aa
Community = ab (CSO, religious leaders, tribe)
78
There’s no code for key programme design issues: how does UNICEF identify objectives, how does it identify
specific indicators, what do these indicators measure, how does it specify attributibility in a diffuse landscape of
providers and partners, how does it monitor and evaluate its own activities and organisational challenges, etc. 8f has
thus been usurped as much as possible to flag these issues.
79
As the occurrence of specifically gender-related materials became more frequent, 9e was usurped to tag nonemergency as well as emergency-related gender information.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
155
Youth = ac
International = ad
Ministry = ae
Professionals = af
b. What
Facilitate = ba
Implement = bb
Enforce = bc
Advocate = bd
c. Where (decision-making seniority, influence)
Lowest = ca
Middle = cb
High = cc
d.
When (at what point in the project are partners coming on)
Beginning = da
Middle = db
End = dc
12. CP themes (sectorality)
a. Correlation between PEF good practices and child protection themes 80
13. Scale81
a. Planned to work at ______ level
b. Had success at _____level
Micro = mi
Mezzo = me
Macro = ma
14. Prevention or Response
a. Program objectives account for differences among prevention, intervention, and
remediation
Yes = y
No = n
Can’t say = cs
80
12a has been usurped to track evaluation data addressing another kind of intersectorality: where evaluators find
that Child Protection investments are more effective when orchestrated in tandem with, for example, water and
environmental sanitation, or poverty alleviation.
81
Some of the background questions on the matter of scale were set as follows:

What level of decentralization is workable for what kind of CP interventions?

Have we shown that we CAN work at scale?

If not, what’s the reason we’ve worked at lower scales?

Is CP programming most functional at smaller contextual levels by nature?
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
156
Figure 4: Average Quality, Protective Environment Elements (Disaggregated)
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
157
This figure shows the average programming quality of each sub-component, or building block, of
the Protective Environment Framework. The vertical axis shows the eight components plus
emergencies with all the sub-components. The average quality of the investment strategy, that
is, the appropriateness and effectiveness of the programming practices that are attached to
them, is shown on the horizontal axis. Programming was rated on a scale of 0 (unacceptable) to
3 (excellent). Details on the ratings framework may be found in Appendix Four.
Again, these quality ratings are not measures of evidence quality: they are measures of the
quality and effectiveness of the investment strategies, as facilitated or constrained by
programming practices.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
158
Appendix Three: Evidence
Quality
Reporting Framework
Introduction
The UNICEF Evaluation Office applies evaluation standards quality ratings to each evaluation
that enters into the UNICEF Evaluation Database. For this meta-evaluation, however, the UNICEF
Evaluation Office designed an instrument to assess evidence quality in depth. Its purpose was to
assist in distinguishing evaluation report contents not on the basis of general evaluation practice
but, rather, on the basis of the integrity of the evidence base.
Thus, the meta-evaluation evidence quality ratings framework (see Figure 5, below) focuses
specifically on activities relating to methodological good practice—for example, the use of
comparison groups and longitudinal data. It has also been a means of identifying evaluation
reports that analyze results against globally accepted standards as opposed to what individual
stakeholders say (“customer satisfaction”).
This appendix presents the measurement framework and a short overview of the average report
ratings. The numbers in Figure 5 (below) are the values assigned for the level of quality. For
instance, if a report used ample longitudinal data it might receive a 2.5 or above rating; if there
were an over-reliance on what Key Informants said (as opposed to an equal or greater emphasis
on other less subjective variables) the report would read more as a customer satisfaction survey
and receive a poor rating (0.51- 1.5).
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
159
Figure 5: Evidence Quality Reporting Matrix
Variable
0–0.5
0.51–1.5
1.51–2.5
>2.5
Results Based
Management level
Can’t say
Activity
Output
Outcome or
impact
None
Selected at time
of evaluation
Matched during
program design
Random
assignment
End (eval only)
End and middle
End and baseline
Baseline, middle,
end
None: presented
without analysis
Client
satisfaction
Against project
goals
Against globally
accepted
standards
No analysis
Unclear82
Intended only
Intended and
unintended
None
One variable
(e.g., gender)
Multiple single
variables
Compound
variables (e.g.,
rural boys vs.
rural girls)
None
Simple
tabulations of
qualitative data
or n/a
Simple
tabulations of
survey data
Complex analysis
of survey data
Unacceptable
0–0.5
Poor
0.51–1.5
Good
1.51–2.5
Excellent
2.51 and above
Comparison
groups
Data time series
Standards used to
analyze results
Consequences
analyzed
Data/analysis
disaggregation
Statistical analysis
Overall ranking
82
Consequences unrelated to project goals = unclear.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
160
Evidence Quality Findings
The average overall rating for the meta-evaluations was 1.44. This is the higher end of the
“poor” category. The evaluations were fairly equally divided in terms of evidence quality with
almost half in the “poor” evidence category and half in the “good” category:
-
Excellent:
4 reports (6.78%)
Good: 24 reports (40.68%)
Poor: 29 reports (49.15%)
Unacceptable: 2 reports (3.39%)
The average ratings for individual variables show that evidence quality was generally good with
respect to the analysis of intended and unintended consequences and attention to results-based
tools and methods. The evidence quality was generally poor in the areas of statistical analysis,
data time series, and the use of comparison groups.
The following average ratings per variable were calculated:
Evidence quality variable
Average rating,
59 reports
Consequences analyzed 2.2
Standards used to analyze results 2.1
Results Based Management level 1.83
Data analysis/disaggregation 1.56
Statistical analysis 1
Data time series 1
Comparison groups 0.47
Figure 6: Evidence Quality, Average Per Variable
A few of these findings can be understood in relation to general UNICEF evaluation performance
standards. Drawing from a 2006 annual review of evaluation quality ratings: 83
83
http://www.unicef.org/evaldatabase/files/UNICEF_Evaluation_Report_Quality_Review_2006.pdf
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
161
-
Whereas in the meta-evaluation evidence ratings, the attention to expected and
unexpected consequences is quite good, according to the UNICEF Evaluation Ratings
system this element was missing from 30% of reports between 2004 and 2006 and is
thus considered to be an area of evaluation practice in need of strengthening.
-
In both systems, the Results Based Management standards are satisfactory; however,
the meta-evaluation evidence quality tool also specifies that outcomes and impacts are
analyzed. This is an area of evaluation practice that, according to the UNICEF Evaluation
Standards, is poor or missing in 28% of the evaluations that were rated between 2004
and 2006. Thus the Results Based ratings for the meta-evaluation are better than those
measured for UNICEF evaluations as a whole. A look at which evaluations did well in
that category (Appendix Five) would suggest that investments in Tier One evaluators are
also investments in Tier One evaluation reporting.
Limitations and Biases
The evidence quality reporting framework is useful as a means of bringing consistency to the
reports in terms of measuring evidence quality. The UNICEF Quality Ratings Standards are too
generalized to accomplish this important task.
One size does not fit all
Generally, evaluations of activities with an on-the-ground presence fared somewhat better
when subjected to this rating tool. That is, evaluations of efforts addressing, say, street children,
did better than evaluations in areas such as legislation. The reason for this is that evaluations of
street children investments could relatively easily utilize longitudinal data and comparison
groups, whereas these methods have less bearing on the evaluation of legislative efforts, at
least in the short run.
Once this limitation became apparent with respect to evaluations addressing legislation,
however, there was no retrofit. This was an emergent finding and a retrospective system of
counter weights and measures was, in the opinion of the evaluator, not called for.
Does not ask “why”
The framework was also largely unresponsive to evaluation findings relating to UNICEF’s own
corporate strengths and weaknesses, that is, to issues revolving around programming practice
that might militate for or against good evidence quality. For example:
-
A relative de-emphasis on monitoring ran through report findings and constrained what
evaluators could do with regard to data time series and comparison groups, yet this tool
records only that these were poorly rated categories, not why that was the case.
-
UNICEF’s robust capability as a broker across often widely varying and unfamiliar
partners, a corporate strength that was seen to mediate the viability of many
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
162
investments, could not be situated except by using the “consequences analyzed” in a
the very widest possible sense and often inferentially, as well.
To bridge this gap, during the Inception Phase of the meta-evaluation, the evaluator analyzed
and synthesized programming practices by applying the Protective Environment Framework to
the first 47 reports (or five/sixths of the entire meta-evaluation content). From this content
analysis, the evaluator designed a programming tracking and measurement tool (see next
appendix).
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
163
Appendix Four: Programming
Practices Reporting Framework
Introduction
The main report has detailed how the programme practices reporting framework came about,
how it was applied, and how it relates to the matter of evidence quality. This appendix presents
the framework and statistical tables to support the discussion in the main text.
Figure 7: Programme Practices Reporting Framework
Variable
A. Context
In practical terms,
how well does the
program design fit
the political,
cultural, and
economic climate?
Unacceptable
0–0.5
Poor
0.51–1.5
Generic, cookie cutter
Good
1.51–2.5
Excellent
>2.5
Basic understanding
of the environment
Taps into relevant
formal and informal
decision making
networks
Activates new,
unusual, partnerships
B. Intersectorality
None; should have
had but didn’t
Limited, but properly
so (wasn’t the scope
to become
intersectoral)
Convenes partners;
some logical or
desired partners
Integrated
C. Programme
communications84
Missed opportunity
Limited; unconnected
to the CRC
Evidence based
advocacy; media plan
Activates
commitments to the
CP program and CRC
Excellent use of
Unicef convening
power
Poor or missing
situation analysis
D. Facilitative role
Moral persuasion
Missed opportunity
Limited
Knows who/how to
lobby
E. Achieved level
of community
ownership
None or symbolic
Expectations are not
realistic
Develops buy-in
Community defined;
community driven
F. Child
participation
Missed opportunity
Mostly rhetorical
Incorporates CRC
trainings
Integral; forged at
design stage
84
Refers to Communication for Development, or “C4D.”
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
164
Figure 7: Programme Practices Reporting Framework
Monitoring plan with
capability to make
adjustments
None
G. Strategic
planning and
technical
assistance
How the work will
get done, by
whom, with what
resources, etc.
H. Knowledge
management
Non-identification or
non-mobilization of
partners
Non-identification of
the Protective
Environment
Framework
None
OVERALL RANKING
Unacceptable
0–0.5
Duplication of effort
Weak technical
assistance
Protective
Environment
Framework drives
planning
Future strategies can
be informed by this
one
Human rights lens in
place
Limited
Poor
0.51–1.5
Design takes good
practices into account
Good
1.51–2.5
Proactive,
comprehensive
Excellent
2.51 and above
Programming Practice Findings
Strategic planning is defined as how the work will get done, by whom, and with what resources.
As discussed in the main report, this was the paramount programming issue across the whole of
the meta-evaluation. The density of evidence around “Strategic planning,” which is four and
one-half times higher than “Child participation,” carries a freight of information about the
professionalization of development practice. What is the nature of this evidence? Good? Bad?
Some of both?


“Strategic planning” was cited most often in relation to the very weakest link in the
Protective Framework, “Monitoring and oversight,” which had ratings of poor-to
unacceptable three-quarters of the time.
On the other hand, in another area of the Protective Framework, “Building capacities of
those closest to the child,” three-quarters of the strategic planning ratings were good or
better. With respect to “Service delivery,” almost two-thirds of the time the ratings
were at least adequate.
These and other statistic can be seen in Figure 8, below. This figure shows programming
practices that were rated as “Good” to “Excellent” divided by all the ratings to show the
percentage of good or better quality programming practice. Readers can see that, for instance:

With respect to UNICEF’s use of its facilitative role, 64% of the time the efforts were
rated as good or better with respect to legislation and almost 70% as good or better
with respect to strengthening government commitments.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
165

On the other end of the spectrum, only a quarter of the strategic planning efforts saw
adequate ratings with respect to monitoring and oversight; only a third of the instances
relating strategic planning to legislation were rated as adequate.
Figure 8: Program Practices/Protective Environment Elements
Programme practice
Government
commitments
Legislation
Harmful
attitudes,
practices
Public
engagement
Child
participation
Building
capacities
Service
delivery
Monitoring
&
oversight
0.00%
0.00%
58.06%
100.00%
82.14%
87.50%
85.71%
0.00%
Percentage (1.5 or
higher-rated divided by
all values)*
Context
Intersectorality
50.00%
Programme
communications
100.00%
Facilitative role
68.42%
Community ownership
66.67%
Child participation
Strategic planning
43.75%
40.91%
50.00%
39.39%
64.10%
100.00%
100.00%
50.00%
91.30%
100.00%
55.56%
37.50%
66.67%
50.00%
50.00%
45.45%
100.00%
55.56%
40.00%
44.00%
66.67%
75.00%
0.00%
53.85%
Knowledge
management
0.00%
33.33%
40.00%
0.00%
50.00%
72.00%
61.54%
0.00%
* Some cells are empty because the value in the cell was zero. Some cells show 100% when the sample size is
negligible
Limitations and Biases
The incorporation of Programming Practices Reporting Framework into the meta-evaluation
makes the analytical framework responsive to the actual content of the reports, which is
generally more reflective of programming design and execution than of evidence based good
practices. However, it is a blunt instrument: each rating of zero to three represents the average
for the report, and so can only be partially illuminating.
On the other hand, to test the tool’s ability to deliver replicable results, two individuals applied
it to the same reports during the pilot stage. The ratings and the investments that were tagged
as evidence addressing the meta-evaluation objectives were largely identical.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
166
25.69%
2003
2006
2006
2004
Sheeran Consulting, Inc.
Final Report, 15 May 2008
3.49
2008 UNICEF Child Protection Meta-Evaluation
167
Overall ranking
q7
Statistical analysis
d8
Evaluation of the
Deinstitutionalisation Project,
Including Alternative Schemes and
Family Support
Thematic Evaluation of UNICEF’s
Contribution to Juvenile Justice
System Reform in Tajikistan
Data/analysis
disaggregation
Tajikistan
2.56
Consequences
analyzed
q7
Standards used to
analyze results
d4
Data time series
Belarus
1.85
Comparison groups
q7
RBM level
d8
A Multidisciplinary Approach to
Combat Domestic Violence and
Other Form of Violence Against
Women and Children
Assessment of Effectiveness of
Juvenile Justice Trainings for Police
Workers in Azerbaijan
2006/006 Report sequence
Azerbaijan
2.85
3
0
0
2.8
2.8
2
0
1.514
2003/008
h3
Evaluation of the UNICEF Impact on
the Juvenile Justice System Reform
in Three Countries in the Regions
(Serbia, Romania, Tajikistan)
1.2
1.5
1.5
1.5
1.5
1.5
1.5
1.457
2006/007
d7
3.49
0
0
0
1.5
2
2.5
2.5
1.214
2006/006 2004/007
Bosnia &
Herzegovina
Reports/Evidence Ratings
UNICEF
Evaluation rating
q7
Date
d8
2006
CP second theme
Serbia and
Montenegro
Country
CP theme
CEE/CIS
CEE/CIS
CEE/CIS
CEE/CIS
CEE/CIS
Region
Appendix Five: Meta-Eval
2
0
2
3
2.5
2
2.5
2
3
0
0
2.8
2.8
2
0
1.514
1.66
UNICEF Project on Preventing
Trafficking of Women and Children,
Gender-Based Violence, and
HIV/AIDS in BiH
2006
2003
2003
2005
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
168
Overall ranking
h3
Statistical analysis
d7
Evaluation of the program
“Improved Mechanisms of Child
Protection”
Data/analysis
disaggregation
Bosnia and
Herzegovina
2.64
Consequences
analyzed
q1
4
Standards used to
analyze results
d1
2
External Evaluation of the Day Care
Center “TISA” for Children with
Disabilities, Bijelo Polje, Montenegro
Data time series
Bosnia and
Herzegovina
2.31
Comparison groups
q7
RBM level
d1
0
Mid- Term Evaluation of the
Efficiency of UNICEF
Deinstitutionalisation Project
2002/001 Report sequence
Serbia and
Montenegro
1.64
2
1.5
1.5
0.5
0.5
1
2
0
1
2
2
0
0
1
1
0
0.5
1.5
1
1.5
0
0.786
1.5
0
0.5
1.5
2
1.5
0.5
1.071
2.5
0
1.9
1.5
2
1.6
1.6
1.586
1.5
0
0
1.5
3
0
0
0.857
2006/003
q7
Mid-Term Evaluation of the
“Children’s Chance for Change”
Project —A Juvenile Justice Initiative
in Serbia and Montenegro
0
2005/003 2006/008
d4
2.23
1
2003/005
Tajikistan
Mid Term Review UNICEF Romania
Report on Child Protection
Programme
2003/013
q7
UNICEF
Evaluation rating
d8
Date
Serbia and
Montenegro
1.79
2002
CP second theme
q1
4
2006
CP theme
d1
2
Country
CEE/CIS Region
CEE/CIS
CEE/CIS
CEE/CIS
CEE/CIS
CEE/CIS
Romania
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2.62
Evaluation of Training of Social
Workers
2008 UNICEF Child Protection Meta-Evaluation
169
Overall ranking
Statistical analysis
Data/analysis
disaggregation
Consequences
analyzed
Standards used to
analyze results
Data time series
q1
4
Comparison groups
d1
2
Institution Building & Mainstreaming
Child Protection in Indonesia UNICEF
Supported Child Protection Bodies
(LPAs)
RBM level
Cambodia
2.1
2003/010 Report sequence
q1
4
2.6
1.5
0.5
1.571
2004/007
d1
2
Evaluation of Disability Project
3
1.5
0
0
1
2
1
0
0.786
2004/030
Indonesia
2.84
0.9
1.5
0
2.5
3
2.5
2
1.5
1.857
2005/16
q1
4
0
3
0
2.5
3
3
3
0
2.071
0
0
0
0
2.5
0
0
0.357
2.6
1.5
2
3
3
2
1.9
2.286
2003/023
d1
0
2.8
2.5
2005/17
Cambodia
UNICEF
Evaluation rating
q1
4
Date
d1
2
Evaluation of the Development of
Integrated Community Based Child
Protection Services project (in
Ungheni)
External Evaluation of the Child
Protection Network
2003
Cambodia
1.66
2004
q1
4
Evaluation of Projects Aimed at the
Prevention of Child Abandonment in
Maternities
2004
d1
2
2005
Moldova
2.08
2004
q1
4
2005
CP theme
EAPRO
CP second theme
Country
Region
CEE/CIS
CEE/CIS
EAPRO
EAPRO
d4
EAPRO
Romania
Psychosocial Interventions,
Evaluation of UNICEF Supported
Projects (1999–2001)
2006
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2003
2001
2003
EAPRO
EAPRO
EAPRO
2.54
2008 UNICEF Child Protection Meta-Evaluation
170
Overall ranking
Statistical analysis
Data/analysis
disaggregation
i1
0
Consequences
analyzed
d1
Support Systems to Reintegration
Programmes for Children and
Women Needing Special Protection
Standards used to
analyze results
Indonesia
2.07
Data time series
h3
Comparison groups
d7
Child Rights in Post Conflict Areas
(Access to Health, Water &
Sanitation), Aniong Veaeng and
Trapeang
RBM level
Cambodia
1.86
2006/006 Report sequence
q8
Lessons Learned Review on the Law
Enforcement Against Sexual
Exploitation and Trafficking of
Children Project
0
1.5
3
3
3
2.5
2.214
1.5
0
0.5
2
1
0.5
0
0.786
0.5
0
1
0.5
0
1
0
0.429
1
0
0
2.5
2.5
0.8
0
0.971
0.2
1
0.5
3
1.5
1.5
0.2
1.131
2006/035
d1
1.72
2.5
2003/011
Cambodia
Review of the Street Children Project
2001/007
h3
3.54
2003/800
d7
UNICEF
Evaluation rating
Cambodia
Date
q1
4
2006
CP theme
CP second theme
Country
Region
EAPRO
d5
EAPRO
Laos
2002
2006
EAPRO
2007
2006
EAPRO
EAPRO
EAPRO
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2.33
Documentation and Evaluation of
UNICEF Thailand’s Post-Tsunami
Psychosocial Initiatives Report
2008 UNICEF Child Protection Meta-Evaluation
171
0
0
1
1
0.2
2.3
0
1.5
1.5
2,5
Overall ranking
Statistical analysis
Data/analysis
disaggregation
Consequences
analyzed
g1
Standards used to
analyze results
d1
7
Preliminary Assessment of the Use
of Family and Community
Conference Group as an Alternative
Approach in Juvenile Justice
Data time series
Thailand
1.8
Comparison groups
q7
Thao Dan Street Children Program
RBM level
d8
Report sequence
Thailand
2.57
None given
q1
4
Evaluation of District-Based
Initiatives on the Prevention of CSEC
and Child Labour (in Indramayu,
Surakarta and Probolonggo,
Tulungagung)
2.5
1.5
1.5
0
1.144
1.5
0.2
0.2
0.2
0.614
0
2
2
1.5
1.5
1.329
0
0
3
1
0.5
0
0.86
1
0
3
2.5
0
0
1.143
2006/018
d5
2.39
2002/816
Vietnam
Evaluation of Psychosocial
Rehabilitation Opportunities,
Training and Enhancement for
Children Along the Thai/Burma
Border
2007/801
d2
2.07
2006?
d7
UNICEF
Evaluation rating
Indonesia
Date
i1
0
2006
CP theme
CP second theme
Country
Region
d1
EAPRO
Thailand
2.72
Mahzel Project for Social
Reintegration and Protection of
Disadvantaged Children as Future
Development Stakeholders
2005
ESARO
ESARO
2004
ESARO
ESARO
Report on Child Justice in Zambia
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
172
Overall ranking
Statistical analysis
Data/analysis
disaggregation
Consequences
analyzed
Standards used to
analyze results
d1
3.9
Data time series
d4
Evaluation of Lihlombe Lekukhalela
Child Protectors
Comparison groups
Eritria
1.94
RBM level
q7
Report sequence
d8
Link Association for the Relief of
Children (LARC) Community Action
Support Project Evaluation
1.5
0
2
2.5
1.5
1.5
1.644
3
0
0
0
2
0
0
0.714
0
3
0
3
3
1.5
3
1.929
3
3
3
3
3
3
3
3
2.4
1.2
2
2.3
3
2
2
2.129
2004/021
Zambia
2.34
2.5
2005/002 2005/005 2002/011
j2
Impact Assessment of the Most
Vulnerable Children (MVC)
Community Based Care, Support,
and Protection in Musoma Rural
2004/008
d1
2
UNICEF
Evaluation rating
Swaziland
Date
j2
2005
d1
2
2003
CP second theme
Zambia
3
2005
CP theme
q1
4
Country
d1
2
ESARO
Region
Tanzania
2003
2007
Report on the Evaluation of the
UNICEF-Supported Chikankata
CBOSP & OVC Training Projects
2004
2005
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2002
ESARO
ESARO
ESARO
ESARO
2.08
2008 UNICEF Child Protection Meta-Evaluation
173
Overall ranking
Statistical analysis
Data/analysis
disaggregation
j2
Consequences
analyzed
d1
2
Impact Assessment of MVC Support
Programme Implementation in
Karagwe, Kisarawe, and Magu
Standards used to
analyze results
Zambia
2.29
Data time series
q1
4
Evaluation of Data Collection Project
on the Impact of Armed Conflict on
Children (SSRC/UNICEF)
Comparison groups
d1
2
2.26
RBM level
Tanzania
Report sequence
g9
Street Children and Street Mother,
Partnership Programme for 2004 in
Addis Ababa
2.5
1.5
0.5
2.5
3
1.5
1.5
1.857
3
3
2.5
3
3
2.5
3
2.857
3
0
0.5
3
3
0.5
0.5
1.5
3
1.5
2.5
2.5
3
2.5
0.5
2.214
3
0
1
1.5
1
0
0.957
2007
d1
2.74
None given 2003/040
Angola
An Assessment of the Impact of
Implementing the MVC Programme
and the Operation of the MVC Funds
and the Potential for Scaling Up to
Provide National Coverage of Social
Protection for Children
2004/020
q1
4
2.39
2002/010
d5
UNICEF
Evaluation rating
CP second theme
Ethiopia
Date
CP theme
q1
4
Country
d1
2
ESARO
Region
Tanzania
.2
d1
Iran
d1
7
g1
Sudan
d1
q8
Evaluation of Emergency Child
Protection Programs in Bam
3.24
Evaluation — Grassroots PeaceBuilding Project
2004
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
174
Overall ranking
Statistical analysis
Data/analysis
disaggregation
Consequences
analyzed
Standards used to
analyze results
Data time series
Comparison groups
RBM level
UNICEF
Evaluation rating
2003
2006
2.15
2007
Global
MENA
MENA
3.87
2005/811 Report sequence
d3
2003/801
Global
3.67
2005/005
p1
None given
d2
Report of the Organizational
Development Assessment and
Capacity Building of UNICEF Child
Protection Partners in Somalia/land
Education as a Preventive Strategy
Against Child Labour: Evaluation of
the Cornerstone Programme of
UNICEF's Global Child Labour
Programme
Evaluation of UNICEF’s Support to
Mine Action
2004/006
Global
2.73
2005
d8
Date
CP second theme
CP theme
Region
Country
d1
2
Global
ESARO
Somalia
3
0
0.5
3
3
3
3
2.214
2.5
0
1.5
2.5
2.5
2.5
1.5
1.857
3
0
2.5
3
3
3
0
2.071
1.5
0
0
2
2
2
2
1.357
3
1.5
0.5
3
3
2.5
0.5
2
2007
2006
2006
Psychosocial Care for Children
Traumatized by Terrorist Violence
2004
2002
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2003
MENA
MENA
MENA
MENA
MENA
2.68
2008 UNICEF Child Protection Meta-Evaluation
175
Overall ranking
i1
0
Statistical analysis
d1
Urban Child Protection in Cairo and
Alexandria: An Assessment of
Achievements and Prospects
Data/analysis
disaggregation
Algeria
2.33
Consequences
analyzed
q1
4
Standards used to
analyze results
d1
2
Report on Appreciative Assessment
of Psychosocial Interventions in
Gujarat
Data time series
Egypt
2.81
Comparison groups
g1
Bam Emergency Programme
2003–2006
RBM level
d1
7
Report sequence
India
2.87
3
3
2.5
1.5
1.929
0
0
0
0
3
1.5
0
0.643
2.5
0
2
3
3
2
2
2.071
2.5
0
1
3
2.5
1.5
0
1.501
0.5
0
1
2
1
1
0.5
0.857
3
0
2
3
2.5
1.5
2
2
2006/005
g1
0.5
2006/010
d1
7
Community-Based Child Protection
Programme Evaluation and Review
0
2007
Iran
2.05
3
2002/800
q1
4
Evaluation of the ELMAN-UNICEF
Partnership in Support of the
Reintegration and Rehabilitation of
Children Associated with Militia
2004/002
d1
2
3.18
2003/002
CP second theme
Somalia
UNICEF
Evaluation rating
CP theme
f5
Date
Country
Region
d1
MENA
Somalia
Final Evaluation of the
Empowerment of Children in Need
of Special Protection Project
2004
2002
MENA
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2005
2005
ROSA
ROSA
2.14
2008 UNICEF Child Protection Meta-Evaluation
176
Overall ranking
Statistical analysis
f5
Data/analysis
disaggregation
d7
Endline Report on Kishori Abhijan:
An Intervention Research Study on
Adolescent Girls’ Livelihood
Consequences
analyzed
Bangladesh
2.31
Standards used to
analyze results
f5
Data time series
d9
Protection of the Girl Child and
Eradication of Malpractices
Comparison groups
Bangladesh
1.61
RBM level
h3
2004/029 2006/007 Report sequence
d9
Evaluation of UNICEF Support to
Juvenile Justice System
2004/005
Egypt
2.17
2002/001
q7
Mid term Review — Gender and
Child Protection Programme
2005/834
d8
2.03
2005/83
6
Yemen
UNICEF
Evaluation rating
q1
4
Date
d1
2
Evaluation of the Social WorkCourse Within the Faculty of Arts
University of Sana’a
2006
CP second theme
Ethiopia
1.7
2004
CP theme
q1
4
Country
Region
MENA
d1
2
MENA
MENA
Yemen
1
0
0
1
2.5
1
0.5
0.857
2
0
0
0
1.5
1.5
0
0.714
0.5
0
0.5
1.4
1.5
1.5
0.5
0.843
0
0
0
0.5
1.5
1.5
0
0.504
2.6
2.6
2.6
2.6
2.7
2.6
2.5
2.6
1.8
0
0.5
0.5
0.5
0.5
0.5
0.614
Evaluation of Alternative Care
Programmes
2006
2003
2003
WCARO
WCARO
1.92
2008 UNICEF Child Protection Meta-Evaluation
177
Overall ranking
Statistical analysis
Data/analysis
disaggregation
Consequences
analyzed
1.5
2
0.5
0
1.001
3
1.5
2.5
3
3
2.5
2.2
2.529
1.5
1.5
0.5
2.5
2.3
2
2.5
1.829
3
0
3
3
3
3
2.4
2.486
0.5
0
0
1
1
1.5
0.5
0.644
111.1
27.8
56.1
124
130
92.26
57
85.47
59
59
59
59
59
59
59
59
1.883
0.471
0.95
2.11
2.2
1.5637
0.97
1.449
Totals
Counts
Averages
Sheeran Consulting, Inc.
Final Report, 15 May 2008
Standards used to
analyze results
d4
Evaluation of the Protection
Component of the Rights Promotion
and Protection Programme
Data time series
d1
2.14
Comparison groups
Sierra Leone
Community Based Reintegration:
Programme Evaluation
RBM level
h3
2002/800 Report sequence
d5
2.48
2
2007
Ghana
Evaluation of the Reintegration
Component of the DDRR Programme
for Children
0
2006/002
f5
2.68
1
2003/011
d1
Participatory Evaluation for the
“Women and Children at Risk”
Project of St. Francis Xavier
None given
Sierra Leone
UNICEF
Evaluation rating
f5
Date
d1
1.92
2002
CP second theme
Liberia
2007
CP theme
q1
4
Country
Region
TACRO
d1
2
WCARO
WCARO
Guyana
Tsunami reports
These reports were reviewed for the “Emergenices” chapter. They were not included in the formal meta-evaluation, however, as they came too
late. The 2004 Indian Ocean Tsunami Disaster: Evaluation of UNICEF’s Response (Emergency and Initial Recovery Phase)



Indonesia (May 2006) by Alexander et al.
Maldives (May 2006) by Allaire et al.
Sri Lanka (May 2006) by Lewis Sida et al.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
178
Appendix Six: Frequency of
Evidence, Protective
Environment Elements
Figure 9, below, shows the number of instances that any of the eight components of the
Protective Environment or any of their sub-components addressed the meta-evaluation
objectives in a meaningful way. “Meaningful” was measured by the level of instructional
information that any investment carried, negative or positive, addressing the objectives of the
meta-evaluation.
In Figure 9, the horizontal axis shows the number of occurrences, or frequency, of evidence; the
vertical axis shows the Protective Environment Framework components and sub-components.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
179
Figure 9: Frequency of Evidence, Protective Environment elements
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
180
Appendix Seven: Thematic Overview,
Meta-Evaluation Reports
Child Protection Primary and Secondary Themes
Count of Overall
D 12 Multi-thematic child protection
Juvenile justice
Orphans and affected children
Social protection; social safety nets; conditional cash transfers (CCT)
18
1
3
14
D 01 Armed conflict, child soldiers
Children without caregivers; institutionalization
CRC/Child rights promotion, reporting, and monitoring
EMOPs M&E/data systems
Non-formal, vocational and life skills
Mental health
10
1
2
1
3
3
D 08 Juvenile justice
Child rights legal and policy/administrative reform; CRC compliance; policy and law analysis
7
7
D 07 Sexual exploitation, trafficking
Child labor
Gender-based violence; sexual exploitation; gender and justice
Non-formal, vocational and life skills
6
1
4
1
D 17 Psychosocial support
4
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
181
Family separation and reunification; physical protection; child protection; psychosocial issues
4
D 04 Children without caregivers; institutionalization
Armed conflict, child soldiers
Child rights legal and policy/administrative reform; CRC compliance; policy and law analysis
Social protection; social safety nets; conditional cash transfers (CCT)
4
1
2
1
D 05 Street children
Gender-based violence; sexual exploitation; gender and justice
Social protection; social safety nets; conditional cash transfers (CCT)
4
1
3
D 09 Harmful traditional practices
Gender-based violence; sexual exploitation; gender and justice
Non-formal, vocational and life skills
2
1
1
D 10 Disabilities
Child rights legal and policy/administrative reform; CRC compliance; policy and law analysis
Social protection; social safety nets; conditional cash transfers (CCT)
2
1
1
D 02 Child labor
Multi-thematic sectoral, including supply as a program strategy, capacity building and training; HRBAP
1
1
D 03 Land mines/Unexploded ordinances
Armed conflict, child soldiers
1
1
Grand total
Sheeran Consulting, Inc.
Final Report, 15 May 2008
59
2008 UNICEF Child Protection Meta-Evaluation
182
UNICEF Evaluation and Research Database, Child Protection Classifications
D
D1
D2
D3
D4
D5
D6
D7
D8
D9
D10
D11
D12
D13
D14
D15
D16
D17
D18
Child protection
Armed conflict, child soldiers
Child labor
Land mines; unexploded ordnance (UXO)
Children without caregivers; institutionalization
Street children
Violence and abuse
Sexual exploitation; trafficking
Juvenile justice
Harmful traditional practices
Disabilities
Program communication, media analysis CP
Multi-thematic CP
Child protection surveys and status; data systems; Situation Analysis
Child protection sectoral or partnership review/evaluation
Other
Migration
Psychosocial support
Birth registration
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
183
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
184
Appendix Eight: How
Large Are the
Programs and the Evaluations?
Introduction
The credibility of evaluation evidence can be measured in relation to the evaluation’s plan for
surveying a subset of the program’s target population. This is the evaluation’s sampling plan: of
the universe targeted for an intervention, what portion of that universe can be tapped for
research purposes to get a generally accurate picture of what is going on?
Target Population and Sampled Population
With a few exceptions, the reports that were available for the meta-evaluation provide limited
information on the relationships between target and sampled populations on which their
analyses are based. To reprise from the main report:


Less than half the reports discuss the number of stakeholders consulted for the
evaluation in relation to the number of stakeholders targeted by the program.85
Of these reports, only 20% describe the rationale for how the stakeholders consulted for
the research were selected (sampling plan).86
These information gaps leave readers wondering how to value the evaluation findings, how
much they might be applicable to the broader target demographic and in what circumstances,
and whether an evaluation’s reach was suited to the scale of the programme or project being
evaluated. This state of play is illustrated in the following chart:
85
Further, the number of stakeholders targeted by the program is more commonly given than the number of
stakeholders consulted for the evaluation (30% more often).
86
In most cases, those consulted are duty bearers, not rights holders. Significant exceptions to this are the Ghana,
Tanzania, and Viet Nam reports.
Sheeran Consulting, Inc. 2008 UNICEF Child Protection Meta-Evaluation
Final Report, 15 May 2008
185
Stakeholders
Targeted by
evaluation:
- Known
- Unknown
Number of Reports
Percentage
43
16
72.88%
27.12%
Targeted by
program:
- Known
- Unknown
33
26
55.93%
44.07%
Targeted by
Evaluation AND
Program
- Both
unknown
- Both known
- One known
10
27
22
16.95%
45.76%
37.29%
The meta-evaluation analysis of sampling practices also raises a question about the scale of the
evaluations that CP is investing in. Fig. 10 (below), is a side-by-side view of the numbers of
stakeholders targeted for the respective evaluations and programmes. The stakeholder numbers
are shown on the horizontal axis; the vertical axis lists programmes by country and sequence
number (e.g. Somalia 2005/811).87 As shown, there are plenty of instances when this
information was not supplied in the evaluation reports. But what might also stand out is the
relatively small scale of the evaluations.
87
Appendix Five lists all reports by Country, Title, Year, and Sequence Number.
Sheeran Consulting, Inc. 2008 UNICEF Child Protection Meta-Evaluation
Final Report, 15 May 2008
186
Figure 10: Scale of Evaluations versus Scale of Programmes/Projects
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
187
Appendix Nine: Stakeholders’
Initial
Recommendations
Introduction
These recommendations were generated by members of the Child Protection Section, UNICEF
Global Headquarters, during a one-day workshop at which the meta-evaluation methodology
and findings were presented (April 2008). Stakeholders marked them as preliminary pending a
phased roll out and deeper study of the meta-evaluation findings and conclusions. There was
some discussion of a management response being developed as part of this longer-term
process.
The evaluator contributes two recommendations relating to improving the evidence base and
using askSam.
Recommendations
a. Ensure that the Child Protection Strategy and Meta-evaluation are understood to be
complimentary
Any presentation of either the Strategy or the Meta-evaluation should reference the other as
context.
b. Tailor presentations of the findings to different target groups
Task force is forming to oversee this process, which will reach out to internal and external Child
Protection partners. This will include a seven-week e-conference, each week addressing specific
elements of the Protective Environment Framework findings. A 2-3 page synopsis of the report,
beyond the Executive Summary, may be developed for different target groups.
Evaluation Office focal point will also share findings on the evidence base issues with Regional
Monitoring and Evaluation Officers during a global meeting in early May 2008. Child Protection
Division is circulating the findings immediately to Child Protection Regional Advisors for
interpretation and recommendations.
c. Establish a Child Protection Monitoring and Evaluation Resource Group (MERG)
Use the meta-evaluation as a platform for developing a MERG (inter-agency technical groups) by
sharing findings with selected partners. MERGs can also work together to standardize indicators
and become a technical nucleus for Child Protection.
d. Establish a Model Terms of Reference for Child Protection evaluation reporting
Also the work of the task force, to identify the reporting issues of first importance. An evaluation
toolkit is also part of this recommendation.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
188
e. Develop a results framework
To operationalize the Protective Environment Framework, Evaluation Office focal point made
this recommendation.
f. Follow up on four specific findings
Certain findings were called out for further examination: gender bias, missing child participation
at the program design stage, that buy-in to Child Protection issues is constrained by a lack of
basic needs, and the gap in investments addressing programme communications
(Communication for Development, or “C4D”).
Regarding C4D, since there is a perception among staff that Child Protection invests widely in
programme communications-related activities, it was discussed that there could be a need to
consolidate Child Protection-related programme communications knowledge-practice.
From the evaluator:
g. Ask managers for the back-story on missing pieces
The broad institutional contexts of investments (e.g. donor financing, host government
sensibilities, unworkable levels of abstraction in evaluation reporting requirements or in
results based approaches) are absent in the evaluation reports; without them, matters such
as the neglect of baselines, indicators, monitoring, and enforcement, are practically
inexplicable. Focus groups with Child Protection and Evaluation managers in the country
offices would be useful now in shedding light on these institutional factors. How are
managers’ hands tied, for example, on the matter of baselines? Where are the bottlenecks
on monitoring? Why are examples of rights based approaches to programming, particularly
with respect to stakeholder participation, so lean?
h. Use the askSam database for ongoing programme evaluations
askSam is essentially ready for use as an ongoing program evaluation tool. New evaluations
could be added annually, coded and analyzed just as the first 59 have been, so that areas that
are flagged for development or improvement now can be tracked over the next five years. In
other words, additions to the database moving forward would allow trendline and other types
of statistical analysis. Consistent inputs over the next five years are simple and cost-effective for
two reasons. First, 80-85% of the work in establishing this (and any) database is already done:
the taxonomy has been established, codes have been set, and ratings scales have been designed
for (i) the quality and effectiveness of programming design and execution, and (ii) evidence
quality. Second, if 50% of each year’s harvest of Child Protection evaluations continued to be
entered into the database per year –roughly akin to the meta-evaluation percentages-- this is an
average of 10 reports per year. The process of coding, assigning ratings to 10 reports, and
assessing trends, is probably on the order of three to five days of work.
This use of the initial investment might proceed in collaboration with (a) UNICEF Evaluation
Office, a stakeholder particularly on the matter of improvements to evidence quality, (b)
Knowledge Management, a stakeholder in the systematizing and managing and leveraging of
UNICEF’s knowledge stores, and (c) Human Resources, a stakeholder in maintaining and
enhancing programming skills.
Further reflections on using askSam are provided in a separate technical note.
Sheeran Consulting, Inc.
Final Report, 15 May 2008
2008 UNICEF Child Protection Meta-Evaluation
189