Plan International UK
M&E Framework
DRAFT
30.05.2013
DRAFT M&E Framework Plan International UK
Table of Content
Table of Content ............................................................. 1
Introduction .................................................................... 2
Section 2: Evaluation Plan............................................. 32
Section 3: Monitoring Strategy ..................................... 48
Appendices.................................................................... 54
1
DRAFT M&E Framework Plan International UK
Introduction
Plan International UK has been asked by DFID and
PWC to put together an M&E Framework specific
to our GEC funded project and to incorporate the
data required for this in to our reporting schedule.
Our M&E Framework provides an overview of our
approach and methodology for
monitoring and evaluating our project Supporting
Marginalised Girls in Sierra Leone to Complete Basic Education with Improved Learning
Outcomes.
This framework is presently in draft form and has been put together by Plan International UK
and Plan Sierra Leone with support from the GEC consortium partners and our independent
baseline research team.
2
DRAFT M&E Framework Plan International UK
Section 1. Baseline Plan
Plan International UK has contracted GK Consulting LLC to undertake independent and thorough
baseline survey research ahead of the GEC funded project – Supporting Marginalised Girls in Sierra
Leone to complete Basic Education with improved Learning Outcomes.
The plan for undertaking this piece of work will be finalized after their pilot phase during April 8-12th.
However to date, based on a) both GK and Plan’s experience of carrying out similar pieces of work b)
the information provided by PWC to guide this process and c) the discussions we have been having
with our local team and consortium partners, we have drafted the following plan:
Background
Plan has approved the following team structure for the purposes of the baseline survey. (CVs and
experience summaries of the below can be provided on request, as can proof of our competitive
process and rationale behind choosing GK Consulting):










Gwendolyn Heaner, PhD: (GK Consulting) Baseline Team Leader
Sybil Bailor, Chief of Party (Plan Sierra Leone) Plan Sierra Leone Team Lead
Maggie Korde, GEC Programme Manager, (Plan UK) Plan UK Team Lead
Nils Gardek (GK Consulting)– Quantitative Research Lead
Samuel Byrne, Head Evaluation and Research Coordinator, (Plan Sierra Leone) Plan Technical
Review Team Lead
Jake Phelan, Learning, Impact and Assessment Officer (Plan UK) Plan Technical Review
Support
Stan Peabody, PhD (EEA)– Senior Technical Reviewer
Carmen Aldinger, PhD (GK Consulting) – Child Education Specialist
Sharron Kelliher, MA (GK Consulting)- Field Coordinator
Marissa Van, MPhil (EEA)– Qualitative Research Analyst
Our proposal to conduct a baseline study on girls’ education in Sierra Leone is developed with focus
on our 4 main project objectives:




Increasing access and retention in grades 5 and 6 of primary school and grades 7,8.9 in
junior secondary school, and of children with disabilities in primary school
Strengthening girls’ learning in grades 5 and 6 of primary school and junior secondary school
Ensuring that girls are learning in an inclusive environment and protected from harm
Ensuring girls’ voices and needs are listened to and responded to and that girls participate in
decision-making concerning their education
Education in Sierra Leone
Sierra Leone faces great challenges with respect to education. One consequence of the war was
destruction of schools: over 70% of all schools (1270 primary schools) had been destroyed (IMF,
2001); this led to almost 70% of school-age children being out of school. In 2001 Sierra Leone
3
DRAFT M&E Framework Plan International UK
started implementing a policy of providing free primary education (FPE) for all pupils in governmentowned and government-assisted schools. During 2001-2005, the government committed almost 20%
of its budget towards education; almost half was devoted to primary education (UNDP 2007); by
2005 there were at least 4300 primary schools. Each district received new funding for education and
the number of students doubled from 0.6 million to 1.3 million between 2001 and 2004; in 2004
Sierra Leone had the highest primary school enrolment rates amongst all Sub-Saharan countries
(World Bank 2007). However, statistics also show that girls, particularly those at JSS level in rural
areas, have much lower rates of enrolment than boys; in addition, the disabled have strikingly lower
rates of enrolment.
Analysing data from before and after Sierra Leone’s 2001 implementation of free and compulsory
primary education, Cannonier and Mocan (2012) found that “an exogenous increase in education
triggers a change in attitudes that is empowering for women”, indicating that not only does
education contribute to technical skills for girls’ to compete in the workforce, but it also gives them
the confidence to do so. However, gaining access to and continuing education is not necessarily easy
for girls. Maclure and Denov (2009) highlight the huge importance of educating girls, and call for
more targeted interventions to affect girls’ education in Sierra Leone:
“In post-war contexts, education is widely regarded as essential not only for civic
reconciliation, but also as a key force for gender equity. In Sierra Leone, however, despite
enhanced educational opportunities for girls, much of the emphasis on post-war educational
reconstruction is unlikely to rectify gender inequities that remain entrenched within
mainstream schooling and in the broader social context”.
Given this, the need for the GEC project is clear and from that, the need for a solid baseline in order
to measure the projects outcomes.
Baseline Study Approach
A. Sampling Strategy
Qualitative and quantitative data will inform all facets of the project (i.e. not just data on targeted
population [girls] to measure the program’s effects over time, but also data from those around them
regarding their perceptions of targeted population in order to better understand the reasons behind
girls’ low retention), but it is key to select representative communities/schools/respondents in order
to obtain an accurate baseline. A good cluster sample will ensure that the size of the sample (n) that
is randomly chosen from the sampling frame (or population) is closely representative of the
population as a whole.
Thus the baseline team will utilize 3-stages of cluster sampling with randomized selection and
stratification, and ensuring some quotas are met, throughout the five districts. The first cluster’s
sampling frame will be made up of villages in which there are intervention schools stratified by
district; the second cluster’s sampling frame will be a list of the communities (village/town) that lie
within the catchment area of those schools; the third cluster will be made up of the households
within that particular community. The total number of communities and number of households
4
DRAFT M&E Framework Plan International UK
surveyed in each depends upon the total population size of the intervention group, as detailed
below.
Sample size
This sampling strategy starts by looking at the ‘ideal case’ for the sample size. This gives a total
sample size of 5,280 girls. In this ‘ideal’ case, the treatment and control groups would be equal,
2,640 in each, and individuals would be randomly assigned. Calculations for this ideal case are set
out in Appendix 1.
There are a number of reasons why this ‘ideal case’ cannot be achieved: the project design does not
allow for randomisation; it will not be possible to identify a substantial and appropriate control
group; the project has multiple different interventions within it at different levels (school, individual
and community based) and is too complex to break down into separate treatment arms; project
resources are not sufficient for a survey on this scale and there are more efficient ways of
approaching the issue.
Accordingly, the proposed approach outlined below will split data collection into two separate datacollection exercises. A household level survey will be administered to measure changes in attitudes
and perceptions at the household level. A school survey level will administer learning tests to girls to
track learning of study subjects over time and provide data to gauge impact. Since expected changes
in learning performance are significantly smaller than changes in attitudes and perceptions,
separating the learning tests from the household survey allows a significant reduction of the latter
sample. Instead of 5,280 households, the proposed case therefore suggests a household level
sample of 1,440. This helps to significantly reduce survey costs while maintaining a minimum level of
statistical power to assess the programme performance.
Power calculations have been undertaken to inform the sample size of both the school and the
household survey. Calculations for the MDES have been made for a power of .80 and a significance
level of 0.5 using the following formula:
MDES = (t(1-k)+ta) * sqrt(C/(P(1-P)J))*sqrt(1-C/(P(1-P)NJ))
Whereby:
(t(1-k)+ta) = 2.8
C = Intra cluster correlation
P = Portion of girls allocated to treatment
J = Total number of clusters
N = Data points within each cluster
The household survey will be administered in a total of 1,440 households in 144 villages. This
includes a control group of 240 households in 24 villages and 120 intervention villages. This control is
not ideal but is constrained by a number factors (see section below on control villages). However, on
certain outcome variables it will be effectively increased by additional control schools, which are
significantly less problematic to identify (see below).
5
DRAFT M&E Framework Plan International UK
Table 1.1 Household Survey Power Calculations
Household survey
effect size
intra cluster correlation
Multiplier
Allocation ratio
Total # clusters
Intra-Cluster Sample Size
Response rate
sample size
Treatment cluster
control cluster
MDES
0.3
0.2
2.8
0.83
144
9
90%
1440
120
24
0.34
The household survey will be administered in a total of 1,440 households in 144 villages. This
includes a control group of 240 households in 24 villages and 120 intervention villages. For the
household survey the parameters used are specified in the table below. The household sample size
given allows the measure impact on the household level of 34%. The allocation between treatment
and control does not follow a 50/50 split. The treatment sample comprises 90% of the total number
of clusters. Following Bloom (1995) one can argue, however, that substantial deviation from the
50/50 treatment/control group mix does not significantly reduce the statistical power of an
experiment. Nonetheless, the MDES is slightly higher than absolutely desirable given an expected
effect size of around 30%. However, this is the best that can be achieved given constraints on control
villages. The table above specifies the parameters chosen for calculating the survey sample size.
Table 1.2 School Survey Power Calculations
Primary School
JSS
effect size
0.2
effect size
intra cluster
0.2
intra cluster
correlation
correlation
Multiplier
2.8
Multiplier
Allocation ratio
0.7
Allocation ratio
Total # clusters
200
Total # clusters
N
21
n
Response rate
90%
Response rate
N
24
N
sample size
4800
sample size
Treatment cluster
140
Treatment cluster
control cluster
60
control cluster
MDES
0.210819 MDES
0.2
0.2
2.8
0.714286
84
21
90%
24
2016
60
24
0.329983
Combined
effect size
intra cluster
correlation
Multiplier
Allocation ratio
Total # clusters
n
Response rate
N
sample size
Treatment cluster
control cluster
MDES
0.2
0.3
2.8
0.704225
284
21
90%
24
6816
200
84
0.210185
They key outcome indicator with regards to the school survey is learning. It is more efficient to
administer the learning tests separately from the household survey, so this will be analyzed at the
school and individual level. Learning assessments at school level will be carried out with a randomly
selected number of girls per target grade (at P5, P6 and JSS1 for the baseline). These girls will form a
6
DRAFT M&E Framework Plan International UK
cohort that will be tracked throughout the project. Re-contact details will be taken at baseline, and
at subsequent waves information on girls exposure to specific elements of the project (such as
bursaries, involvement in study groups, children’s clubs etc.) will be recorded.
At Primary level, 140 treatment schools will be randomly selected and 60 control schools. Within
each school, 12 girls per sampled grade (P5 and P6) will be tested, giving a total sample size of 4,800.
At JSS, 60 treatment schools will be sampled and 24 control schools with 24 girls tested in the first
target grade of the cohort (JSS1), giving 2,016 girls in total. Details of sample size calculations are
given in the table 1.2. Oversampling within clusters is used in expectation of attrition within the
cohort. The minimum detectable effect size at JSS is not ideal, but the gains from increasing the
number of treatment clusters when it is not possible to increase the number of control clusters is
minimal.
Within each class, girls will be randomly selected by counting off the nth girl (where n equals the
number of girls in the class divided by 12 (or 24 in JSS) and rounded down), starting at the girl
nearest the door and working along each bench in turn. In cases where there are not enough girls in
a grade in a sampled school, the team leader will keep track of the shortfall and randomly sample a
larger number of girls in the next available school with more than 12 (or 24 girls for JSS) available to
ensure the total sample size is met within the same number of clusters. In addition, all disabled
children benefitting from CBR activities will be administered the test, a total of 2,125 girls and boys.
The table below summarizes the number of girls who will be assessed at each wave in control and
intervention groups.
Table 1.3: Overall Sample Composition
Intervention
HH questionnaire
1,200
School based testing
4,800
Beneficiary testing
2,125
TOTAL
8,125
Control
240
2,016
0
2,256
Total
1,440
6,816
2,125
10,381
Using ASER tests will provide internal validity within the project (and control) over time, but not
external validity. The additionally of learning outcomes will be measured through changes pre to
post-test within control and intervention groups. At mid-term and end-line the learning assessment
will be re-administered to the same project beneficiaries and the same girls interviewed through the
household survey. Within control schools the same cohort of individual girls will also be followed
throughout. Re-contact details will be recorded at baseline. If girls in the cohort drop out of school
their replacements will be selected randomly (the sample allows for a 10% attrition rate). Total
sample sizes for learning assessments will obviously be increased by the 1,440 girls assessed through
the household survey but this is not factored into these calculations.
The priority in this sampling strategy is put on measuring learning outcomes. The school level control
will not provide a larger counterfactual for the household questionnaire administered at the village
level. However, this is counter-balanced to some extent as key indicators measured through the
household survey (such as changes in attitudes and behaviours, some based on binary ‘yes’ or ‘no’
questions) expect to see a larger effect size.
7
DRAFT M&E Framework Plan International UK
Enrolment and retention will be measured in 100% of intervention schools and the 80 control
schools. Attendance will be measured in a randomly selected sub-sample of intervention schools and
the 80 control schools.
This strategy is designed to measure one treatment arm only, that is, the cumulative effect of all
project activities (see Evaluation strategy below). Given that it is not possible to achieve the ‘ideal
case’ of treating disabled children as a sub-group (or separate treatment arm), an alternative
approach will be taken. Enrolment of children with disabilities can be measured directly though
school surveys for all control and intervention schools. A quota sample in the household survey will
ensure a small but insignificant number of disabled girls are surveyed in control and intervention
groups. To supplement these two measures, all children registered through CBR activities (2,125
boys and girls) will be administered the learning test and a very short questionnaire (questions from
which will be the same as those used in the household survey that relate directly to key indicators).
Although there will not be a robust control group for children with disabilities, changes in their
learning, attitudes and perceptions, disaggregated by sex, will be measured pre and post-test and
compared internally to girls in control and intervention groups. Table 1.4 summarises the main
sources of data and sample sizes for key outcome variables.
8
DRAFT M&E Framework Plan International UK
Table 1.4: Summary of Data Sources and Sample Composition
Indicator
Survey
School
Project beneficiaries
Notes
Intervention
Control
Intervention
Control
1200
240
720
80
785
Intervention Schools comprised of 180 JSS and
540 PS (100% of target schools). Disaggregated
for PS/JSS and disability
Retention
720
80
785
Attendance
120
80
177
Intervention Schools comprised of 180 JSS and
540 PS (100% of target schools). Disaggregated
for PS/JSS
Spot checks in one school closest to the randomly
sampled village (as per household survey)
4800
2016
8256
Enrolment
Learning
1200
240
Individual attitudes and
behaviours i.e. % of marginalised
girls who feel their parents and
community members support
their education
1200
240
Disabled children
(attitudes/perceptions of and
learning outcomes)
Unknown - Est.
10% of
surveyed = 120
Unknown Est. 10% of
surveyed =
24
Intervention
Total
2,125
Control
-
Data collection consists of both household and
school survey
1,440
Questions relating specifically to the attitudes
and perceptions of children with disabilities (i.e.
% of CwD who feel their parents and community
members support their education) will also be
asked alongside learning assessments during
registration of CwD
2,269
Survey data will relate to girls only; project
beneficiaries will be boys and girls
9
DRAFT M&E Framework Plan International UK
Sample selection
This sample selection process is based on the practicalities of the sample frame and village level
data available in Sierra Leone (see section below) and the need to ensure all villages within the
catchment areas of target schools have an equal chance of being randomly selected.
In summary, sampling of intervention communities will be based on a multi-stage process as
follows:
1. Randomly select x number of villages (where x is the total number of sample points
required, and where all villages in the first stage are also the location of a target
schools, PS or JSS), stratified by the number of schools per district and by PS and JSS
within each district
2. Per randomly selected village, randomly select one village per school from a list of all
villages that feed into the school, including the village where the school is located
3. Per randomly selected village, randomly select x number of households (where x is the
sample size within one cluster) as outlined below
Selection of sites in which to collect data (stage one cluster)
A total of 144 sample points will be selected at the first stage in the target areas, which is a
sufficient number to use in order to represent each strata (each level of school) according to the
actual target population, as detailed above. Figures have been rounded up/down to the nearest
whole number to obtain the given numbers in Table 1.2.
Table 1.2 indicates the number of schools that will be targeted after the sample has been
stratified relative to a) school level (JSS or PS) and b) the total GEC project intervention
population of each targeted district by school level, so that the sample is representative of the
GEC project intervention population as a whole (i.e. so that each district has an equal chance of
being included in the sample, even though their project intervention populations may be
greater/far less than others). Ensuring representation from all districts is important because it
will allow for various features that may be different across districts such as ethnicity, languages
spoken, different leadership (country and district-based), geography and relative socioeconomic status (all of which may play a role in the outcomes of GEC) to be represented in the
sample and thus in the collected data.
Table 1.5 below indicates the total number of communities that should be sampled based upon
the above criteria. The total number of JSS intervention schools is 180, or 25% of the total. The
numbers can be calculated for the PS once we have the total number selected in each district.
10
DRAFT M&E Framework Plan International UK
Table 1.5: District-level stratification
District
# of
intervent
ion
schools
(JSS)
# of
interventio
n schools
(PS)
% of
interventio
n schools
per district
total #
of
villages
to be
sample
d per
district
# of
treatme
nt
villages
to be
sampled
# of
control
PS to
be
sample
d per
district
# of
control
JSS to
be
sample
d per
district
31
# of
control
villages
to be
sampled
per
district*
*
6
Port Loko
47
141
0.26
37
16
5
Kenema
27
81
0.15
22
18
4
9
3
Kono
29
87
Kailahun
34
102
0.16
23
19
4
10
3
0.19
27
23
5
11
4
Moyamba
43
129
0.24
35
29
6
14
5
Total
180
540
1
144
120
24
60
20
The number of control school sampling points per district may need to vary depending on actual
number of available control villages per district but the total number will remain the same i.e.
as there are no eligible control JSS in Moyamba more will be sampled from Port Loko instead
(see selection of control villages and control schools below).
Community (town/village) cluster – (Stage 2 cluster)
After village/school clusters are selected at the first stage, numbered lists of the communities
within the catchment area of that school will be used to randomly select one community to visit.
The list of villages using is given as the second stage tab within the sampling frame (columns O
to W list the villages to be randomly sampled at the second stage according to which row (first
stage village/school has been sampled).
Households/girls cluster – (Stage 3 cluster)
Within each of the communities that have been selected, the adults residing in 10 households
will be administered a questionnaire first. Households will be randomly chosen by going to the
town center, spinning a bottle on the ground and walking in the direction of the bottle, then
choosing every Xth house (X determined by total households in town/10) while walking in a
straight line. A degree of quota sampling will be employed in this method in order to ensure that
socio-economic variations within the communities are represented. In this case it may be
necessary to over-sample particularly poor or particularly wealthy households in order to ensure
they are represented. At a minimum, one household per community should be relatively better
off than other households.
Within each household in which eligible girls reside (ages 9-16) as targeted by the Plan Sierra
Leone GEC project logframe, adult heads (the primary caregiver in each household) will be
interviewed based upon convenience sampling (i.e. whether they are there and willing to talk
when the team arrives). Quota sampling will ensure that there is an even split between male
30.05.2013
11
DRAFT M&E Framework Plan International UK
and female caregiver interviews (i.e. if after 5 houses, only men have been contacted, the team
will continue to every nth household until a female is contacted). In total we will obtain data
from a minimum of 1440 households (10 in each community).
We will also collect quantitative data from a total of 1,440 targeted girls aged 9-16 who reside in
each randomly selected house that also has an eligible girl; thus, 10 girls in each of the
communities will be surveyed. Also important is that girls’ with disabilities are represented in
the sample, again obtained through quota sampling. Where possible, quotas should be met in
each community according to the following table:
Table 1.6 – Quotas to fulfill in quantitative surveys
Persons
to
be Rural SL % Female SL
interviewed
M/F (assume rural and
females
urban %
fewer)
Female Caregiver
Girls with disabilities 2.7
Child-headed
households
Quota
of
meet (per
community)
2
2
1*
Quotas within households will only relate to specific sub-groups (female caregivers, girls with
disabilities, and child-headed households) and not related to socio-economic characteristics or
educational/learning attainment.
If there are no girls in a household who fulfil the quota requirements or the quota has already
been fulfilled in that community, but there is more than one eligible girl, then the girl to be
interviewed will be selected randomly. Pre-prepared numbered tokens numbered 1 to 9 will be
pulled out of a bag (as a more obviously transparent random process for household members
than a Kish grid). If a quota cannot be fulfilled in a village the process will carry on regardless
and the quota will not be rolled over to any subsequent community.
If there are no girls in the household who match the target age and/or do not fulfil quota
requirements, the next Xth household will be chosen until twenty targeted girls are reached. If a
household is reached and there are no primary caregivers available, then the next house should
be visited (i.e. even if children are available, they must not be interviewed without contacting
primary caregiver first). If the village boundary is reached in attempts to fulfil the quotas, then
the team should return to the centre and walk exactly the opposite direction and continue as
before until the desired sample is obtained. If there are not enough households in that
community to fulfil the quotas, then another community within the school’s reach will be
randomly selected.
30.05.2013
12
DRAFT M&E Framework Plan International UK
School Data
In each selected village/school selected at the first stage, school data will be obtained by visiting
the school itself and finding key informants to provide data and/or directly measuring or crosschecking certain criteria (i.e. physical location, number of students present, size of school, etc.).
Therefore we will collect 120 surveys with school data from intervention schools. In addition to
these surveys, the team will conduct ‘spot check’ attendance counts on the days that the school
is visited. If data collection occurs on a weekend or holiday, then another team member will
travel to the school while it is in session in order to complete the spot check and collect any
addition school data that was not possible during the household visits.
Control
In order to best measure the program’s effects, research will be conducted with control groups
that will not receive the treatment of Plan’s programme, and will also be outside of the
intervention area for similar programs run by DFID and BRAC. Using a quasi-experimental design
(implementing the same pre-tests and questionnaires to both control and intervention girls
throughout the various stages of data collection including baseline), we will collect data in
communities with similar demographic, geographic, socioeconomic and cultural features to
those of our intervention communities. These will ideally be within the five intervention
districts, but this is dependent upon the project’s actual reach with JSS and PSS (i.e. if any of
these schools will be outside the project’s reach).
Reflecting the sampling strategy outlined above, there will be two distinct control groups:
control villages (providing a counterfactual for the village level household survey) and control
schools (for the school/project beneficiary level data collection). Given the constraints
identifying appropriate controls in either groups, selection of controls will not be random.
Constraints on identifying controls
Finding sufficient control schools for the collection of school level data presents a challenge.
Plan and partners will be working in 77% of all JSS in the five districts. In some districts, we have
selected 100% of eligible JSS schools (where those not selected are either Private schools or
Boy’s schools, and so not suitable for a control group). There will be slightly more eligible
primary schools not selected in each district, but it is difficult to isolate the communities that
feed into different schools. Given the nature of the inter-relationships of schools and
communities, and the fact that children from one community may attend a number of different
primary schools (in one district for example there is an average of 8 villages that feed into one
primary school) and children from one primary school will attend different secondary schools
(an average of 3 JSS that children from one primary school may go on to) it is difficult to find a
community in the district that will not experience a high degree of spill-over and contamination.
We have to date identified a small number of schools and associated communities that we feel
are geographically distant enough from intervention areas, but there are not many to choose
from.
30.05.2013
13
DRAFT M&E Framework Plan International UK
Turning to districts outside the project target districts is one possible solution, but also has a
number of issues. One is in the regional differences of the 5 target districts, with the Eastern
region (comprised of 3 target districts) being geographically and economically quite different to
other parts of the country, and Northern and Southern regions (the two remaining districts
being one in each) socially and politically different. More importantly, the GEC project
implemented by BRAC is in 13 districts, so all overlapping with Plan’s target districts. The DFID
funded Education Consortium, a similar intervention to the GEC, will be targeting 7 districts.
Details of exact areas of intervention for this project will not be available for some months but it
too will limit possible control sites in other districts.
In sum, having a larger number of control clusters is not a viable option financially, nor
practically. The easiest way of identifying controls, reducing the number of schools and
communities the project intervenes in, is not seen as an option. Nor do we believe that in a
three year project it is appropriate to take a phased approach to establishing a control group
and do not have time or resources to redesigning the project to achieve this and meet
contractually agreed targets.
Table 1.7: Location of girls’ education projects in Sierra Leone, 2013
Plan-led GEC
Port Loko
Moyamba
Kono
Kenema
Kailahun
Brac-led
GEC
Port Loko
Moyamba
Kono
Kenema
Kailahun
Pujehun
Tonkolili
Koinadugu
Bombali
Freetown
Kambia
Bo
IRC Education Consortium
Kono
Kenema
Kailahun
Pujehun
Tonkili
Koinadugu
Bombali
Freetown
The only district not covered is Bonthe, a small costal district in the far south west of the
country, relatively distinct from target districts.
Approach to selecting controls
To overcome the challenges of identifying suitable control clusters within Sierra Leone control
sites will be selected at two levels: control schools and control communities. Control schools will
be identified in intervention areas that meet the same selection criteria as intervention schools
but which have not been selected for project activities. Numbers will vary between districts, but
will total 80. These schools will be located in areas close to intervention schools, and at the
Primary level will likely be feeder schools for intervention JSS. This makes them unsuitable as
sites for the household survey, as at the household level there will likely be spill-over and
30.05.2013
14
DRAFT M&E Framework Plan International UK
contamination with other schools located close by. Data collected from control schools will be
on enrollment and learning outcomes only. This will allow us to increase the effective sample
size for the control group relating to outcome indicators on enrollment, retention and learning,
but not for indicators relating to changes in attitudes and behaviors at the
household/community level.
Control communities (which in some instances may overlap with control schools) will be
communities where no effects of the intervention will be felt. Due to the challenges of
identifying such areas, they will be smaller in number and will be purposively sampled to ensure,
first and foremost, their separation from intervention areas and, to the greatest extent possible
to be broadly matched with the socio-economic characteristics of intervention communities. In
each control community, the same number of households will be surveyed as with intervention
communities.
Respondents will be randomly selected using the same method as that for the intervention
group and clusters will be the same size and fulfill the same quotas; the same questionnaires
and assessments will be used. While the data collected form control groups will not have the
same reliability as that collected from intervention groups, because this approach selects
representative respondents at random, it means that the results of each data unit gathered can
be favourably compared between the experimental group and the control group.
B. Quantitative tools: Questionnaires and interview guides
Because the purpose of a baseline is to collect data that specifically measures the key indicators
that will be tracked over time, the instruments designed will be those that give us information
about these particular project outputs. (For any additional data collected because it seems
interesting, it would make sense for mechanisms to be put in place in order to collect this data
over time, in which case these additional indicators should be added to the log frame in relation
to additional outcomes. At the least these indicators need to be more explicitly related to the
outcomes as indicated in the log frame). A draft list of all quantitative and qualitative
instruments can be found in Attachment 1.
Exclusively measuring only those items that relate specifically to the log frame ensures that the
actual procedure of baseline data collection is as succinct as possible, which further ensures the
validity of the data collected. Any additional data collected simply out of interest for the time
being will add to the time that each respondent gives to the research team and if these findings
will not be used constructively, it is best to not ask the question in the first place. Validity is
ensured with a succinct questionnaire, especially when working with children.
Three different questionnaires will be used for collecting quantitative data: For adolescent girls;
for household adults; for schools. For households and for girls, DFID questionnaires to which
additional questions (not covered by this template) have been added specifically to inform the
Sierra Leone GEC. These additional questions will be added throughout the DFID questionnaire.
30.05.2013
15
DRAFT M&E Framework Plan International UK
For schools, only a Sierra Leone-specific survey will be used. All survey questions will be aligned
with log frame indicators to allow midline and end line measurements of outcomes. The
following components must be included in all questionnaires for girls and for adults:
 Identifiers: geographic area, household ID, date of interview
 Background characteristics: gender, age, socio-economic status, educational level,
educational status, disability status (how the project defines disability is outlined in
Attachment 6.)
 Outcome indicators – (to measure what the program intends to change, i.e. girls’
education)
 Impact indicators (measures of program’s achievements – for future questionnaires in
particular, but at the baseline to determine whether any proposed activities/resources
provided by the program already exist to any extent)
The Sierra Leone-specific questionnaires will be relatively simple and short (as preliminary field
testing and refinement will yield only those items that are useful for measuring our variables).
They should be easily understood by those who will be interviewed (i.e. those as young as age 5
and those having no education at all). Consideration should be given to those ‘hard to measure’
indicators such as marginalization and thus measured indirectly using scales or indexes that
have been previously validated based upon an agreed upon definition of that indicator.
Once developed, the Sierra Leone-specific questionnaires will be pre-tested with appropriate
respondents (i.e. individuals with similar traits but who will not be in the program) to ensure
that questions are complete and accurate, that response options are relevant, exhaustive and
mutually exclusive, and that respondents understand the questions being asked. A first-round
test of the questionnaire can be conducted with existing contacts in Freetown, which will also be
useful for training the field team; a second-round of testing will be conducted in a rural
community during which all fieldwork activities will be performed. Any necessary refinements
will be made prior to the beginning of data collection; the Team Leader will determine whether
an additional round of testing and/or training is necessary prior to the beginning of data
collection.
Questionnaires for girls also include a learning assessment component to be completed by all
girls who have been interviewed. The ASER test will be piloted in Sierra Leone and adapted
accordingly so that it is appropriate in this context.
Technology
Because of the high volume of questionnaires that will be collected, the research team will use
Asus Google Nexus 7 Tablets with EpiSurveyer or Magpi software. This serves to conserve paper,
ensure that data is not lost/ruined by weather, that data can be uploaded every day, that
manual data entry is minimal, and to allow the research team to remotely analyse data while
collection is still on-going.
30.05.2013
16
DRAFT M&E Framework Plan International UK
An IT consultant will be contacted in order to assist in programming finalized questionnaires into
the device prior to launching, assist in uploading data to computers, and to ensure the survey’s
integrity with the software. Data will be uploaded into SPSS for maintenance and analysis.
Database Design and Data Processing
In developing the database into which the gathered data will be uploaded, the IT consultant will
ensure the structure and coding are fully compatible with earlier databases so that cross
analyses can be easily achieved. The database developer in the research team must closely
collaborate with all aspects of the statistical design and particularly the review and modification
to the questionnaires. The database developer will work with Plan in-country staff to help them
account for the longitudinal aspects of the project M & E activities so that changes can be clearly
measured over time. The mechanisms for data capture and transfer to the databases has to be
fully compatible with the database structure and needs of the end-users of the information
contained within each.
The databases should be in SPSS file format and would be accompanied by a basic guidelines
handbook on how best to use them, which will require some basic training on how to use SPSS;
this training will be a component of the field training prior to data collection. The team leader
will provide further training after data collection has completed, if necessary.
C. Qualitative Methodology
Qualitative research will complement quantitative data and enable rich longitudinal studies for
the duration of the project. There will be three main components of the qualitative research,
which will be carried out in one community in each district (5 total communities). To ensure at
least one JSS is represented, all JSS schools in the sample will be listed and one will be randomly
selected; then each primary school in the sample will be listed according to district (excluding
the district of the JSS), and one will randomly be selected.
The aim of the qualitative data is explore the lived-experience of marginalized girls in Sierra
Leone and to examine this in relation to educational attainment and specific interventions and
changes the project intends to bring about. It seeks to expand upon the projects theory of
change and begin a process of testing the underlying assumptions of the project and
understanding of how and why change does or does not happen that will continue throughout
the project. This qualitative research, as part of a mixed methods approach, forms a crucial part
of the baseline. It aims to tell us not only if we have met our targets but why change has or has
not happened.
Similarly, as part of a mixed methods approach, the qualitative data will inform and strengthen
quantitative data. Qualitative research will begin before the household survey so as to identify
areas of particular concern to marginalized girls in Sierra Leone, which in turn can be used to
strengthen the survey to ensure we are measuring – and learn from – what is important to
those we are primarily accountable to, as well as what is relevant for our logframe.
30.05.2013
17
DRAFT M&E Framework Plan International UK
Qualitative research is of particular importance in relation to Output 4 of the project. The
impact of complex empowerment and accountability processes are not easily, or even always
usefully, solely measured through quantitative measures. The case studies based on process
tracing methodology, alongside focus group discussions, will offer much more appropriate data
on the impact of this part of the project (see for example McGee and Shutt, CDI Practice Paper,
2013).
Key informant interviews
Purposive sampling will be used to identify key informants in five communities who will provide
comprehensive and ‘insider’ information surrounding girls’ education. The interviews will be
semi-structured (guided by a key informant questionnaire, but conducted with a trained
facilitator who will probe when necessary). The interview should last no longer than thirty
minutes. Examples of key informants include:
o School teachers (male and female)
o Head teachers (male and female)
o Other school stakeholders (board members, etc.)
o Other employees of school (counselor, maintenance)
o Members of parent-teacher associations
o Local representatives of civil society who deal with children’s/women’s rights
o Children with disabilities (age 10-16)
Focus group research:
The field team will also conduct focus group research in one community per district (5 total
communities, the same as those selected for FGDs) with five groups of people as indicated
below (30 total focus group discussions).
o Girls in school (age 10-16)
o Girls out of school (age 10-16)
o Boys (age 8-16)
o Female adults
o Male adults
The community that will be selected for a focus group in each district will be selected randomly
(numbering each community and then randomly selecting one number out of n communities
per district). Attempts will also be made to select a random sample of participants for each
focus group; this will be done in a similar manner as that for the household survey: a bottle will
be spun in the centre of town and every nth household will be chosen, from which one family
member (the first encountered that fits the requirements of any focus group) will become part
of the focus group. The process will continue until 6-8 representatives from each group are
present. It is also important, in addition to this process, to ensure that various minorities are
included in the groups (because they have a lesser chance of having been chosen at random),
30.05.2013
18
DRAFT M&E Framework Plan International UK
especially those who are disabled or especially wealthy/poor so purposive sampling will need to
be used when necessary.
The focus group will involve a combination of close-ended questions (responses collected by
eyes closed/hands up method and tallied on a flip chart for participants to see) followed by
discussion surrounding the results, and also through a series of open-ended questions. This
focus group guide will have been previously developed using with attention to relevant
questions that are both within a logical sequence and will also generate conversation, as the
goal is to get the group to talk more than the facilitator. The facilitators will be trained to follow
the script to an extent, but also to probe further when necessary or to let conversation continue
if it is particularly rich. Visual aids will be utilized to help focus group participants visualize
certain questions (i.e. color-coded time use charts, tally marks under each response for closeended questions, flow/causal diagrams, pie charts, timelines, ladders of importance). Each focus
group will last 1.5 hours long, 2 at a time, in separate venues). A trained facilitator will conduct
each; a trained note taker will take exhaustive notes for the duration.
Case study
In each focus group/ KI community in addition to other communities where the opportunity
clearly presents itself, one case study will be conducted. This will involve identifying a person,
family, place or event that directly relates to the theme of the study, and will be explored indepth using process-tracing methodology. This involves attempting to highlight the particular
event(s) that contributed to some reported outcome. Certain case studies will be useful as
longitudinal studies that can be explored over the duration of the project and will help in
elucidating other subjects for potential longitudinal data collection. The team leader will make
the final decision on what constitutes a worthwhile and appropriate case study (virtually as
needed) .
All of the above activities will be repeated in both of the control groups. Thus we will have
collected qualitative data from the following:
 Key informants – 5 per community * 5 communities = 20 from intervention groups
 Focus group participants – (60 per site) * (5 intervention sites) = 300 from intervention
groups
 Case studies – 5 from intervention groups
30.05.2013
19
DRAFT M&E Framework Plan International UK
20
Deliverables and Activities
Tentative Work Plan:
The timeline given below has been updated but remains provisional. It will be updated once the
sampling strategy is agreed with the EM and the baseline can go ahead.
Table 1.8: Baseline Data Collection Work Plan
Date(s)
23 March
25 - 26 March
26 - 27 March
27 March
27-31 March
04 April
7 April
10-11 April
Task
virtually meet with project
manager and relevant other
staff for initial briefing and
discussion of tor
review project
documentation; write
evaluation protocol and
workplan; discuss with plan
review existing data
collection tools from plan
projects and adapt
begin recruitment of local
innumerators (virtually) if
needed. identify suitable
pilot site close to freetown
design and review data
collection instruments and
questionnaires with input
from plan staff; test
instruments with contacts
on the ground in liberia and
sierra leone
virtual meeting with plan to
review and revise sampling
strategy; selection of pilot
site
discuss and coordinate
research methods with
partners and gec evaluation
manager for final agreement
international travel days;
early arrival in country and
firming up logistics;
preparing for training events
Key Personnel
G Heaner and N
gardek
Deliverable
G Heaner and N
gardek
evaluation protocol and
workplan for baseline
G Heaner and N
gardek
inception report and
draft instruments
M Chea; G Heaner
resumes for potential
staff
G Heaner; N
Gardek; C Aldinger
draft questionnaires
and interview GUIDES;
all questionnaires, all
qualitative tools for
each target group, data
entry processes for
plan’s review
RECCOMENDATOINS
ON strategy FROM em
G Heaner; N
Gardek
G Heaner; N
Gardek
beginning of
DISCUSSIONS and
revisions to m&e
strategy
G Heaner; M Van
Epp; M Chea
30.05.2013
DRAFT M&E Framework Plan International UK
11 April
12 April
13 April
14-17 April
18 -19 April
20-22 April
22 April
22 April – 14
May
22 April – 14
May
15 May – 22
June
team meeting; kick off
meeting with local office;
device update w/ surveys
and testing; meeting with it
consultant
field team training in
methodology and data
entry/upload with it
consultant
first pilot (full) in moyamba
for training purposes and
final refinement of
instruments
additional training and pilots
as needed; refinement of
instruments and sampling
strategy
qualitative data collectionin
moyamba and port loko
further recruitment and
training for additoinal
enumerators and team
leaders;
finalization of qualitative
and quantitative
instruments; final upload to
tablets
refinement of baseline
methodology and m&e
framework with
pwc/coffey/dfid
further traiing, recruitment
and prepartion for
additional tasks for
household surveys and
learning assessments
data collection with support
from staff – oversee the
start up of PROJECT. field
coordinator transition to
local team leaders
21
G Heaner; N
Gardek (virtual); M
van Epp; M Chea
G Heaner; M van
Epp; M Chea with
local field team
G Heaner; M van
Epp with local field
team
G Heaner; N
Gardek (virtual); M
van Epp; M Chea
training and capacity
building for data
collection and entry;
guidance document
finalized instruments to
be used for duration of
data collection;
uploaded into devices
finalized qualitative
instruments
G Heaner; M van
Epp; M Chea with
local field team
G Heaner; S
Kelliher; M Chea; S
Tolley
G Heaner
finalized instruments
G Heaner; Plan
staff
approved m&e
framework
G Heaner, S
Kelliher, Plan staff,
local field teams
G Heaner; S
Kelliher; M Chea; S
Tolley; local field
team
progress reports from
field coodinator (by
phone) every other day
to team leader;
uploaded to computer
nightly; and internet
when possible but
minimum 2x per district
30.05.2013
DRAFT M&E Framework Plan International UK
03– 21 June
05– 21 June
15– 25 June
25– 30 June
26– 10 July
05– 15 July
15– 2 August
31 July – 01
August
2 August
12– 16 August
data collection for school
data and learning
assessments in kono and
kenema
data collection for school
data and learning
assessmnets in kailahun,
port loko and moyamba
project tl returns for final
baseline data collection /
upload with assistance from
local field team;
transcription of field notes
finalized; data entry for
cover sheets, learning
assessments, paper
questionnaires
data entry for school survey
data from all districts
data cleaning; analysis of
household baseline data;
preparation of summary
data for household survey
data cleaning; analysis of
learning assessment and
school data
preparation of reports: draft
evaluation report; m & E
monitoring strategy
ethical and child protection
protocols; child friendly
summary of report
submission of draft report
to plan uk
review of draft reports by
plan
22
S Kelliher, local
field teams with
remote support
from G Heaner
Plan Sierra Leone
staff supported by
Plan UK
G Heaner; local
field team, local
data entry
assistants
all raw household data
uploaded for analysis
Plan Sierra Leone,
local data entry
assistants, remote
support from g
heaner
N Gardek; G
Heaner; M Van Epp
all raw school survey
data uploaded for
analysis
raw data and summary
data for key project
logframe indicators
shared with plan
N Gardek; C
Aldinger
N Gardek; G
Heaner; S Peabody;
C Aldinger
C Aldinger; G
protocols and child
Heaner; M Van Epp friendly summary
G Heaner,
Plan UK
draft report on 2
august including m&e
monitoring strategy,
school data analysis,
child friendly version of
report
plan uk comments on
draft reports
30.05.2013
DRAFT M&E Framework Plan International UK
16– 20 August
20– 31 August
As Needed
consideration of plan
comments and finalization
of report
final review of all reports by
plan
G Heaner, C
Aldinger, S
Peabody
Plan UK
verbal presentation of
findings in
workshop/meeting with
plan (virtual or in person
possible)
G Heaner
23
submission of gk final
report by 20 august
final report and all
supporting materials
submitted by plan uk
presentation of findings
Fieldwork Logistics and Quality Assurance
A field team of local and international researchers will conduct the baseline study. Three days of
preparation in Freetown will be needed in order to refine methodology, train local field teams,
hold meetings with the local Plan office, and prepare devices for data entry. Following that, the
team aims to visit a minimum of one site per day to conduct research activities. All efforts will
be made in order to ensure validity of data collection while completing all quantitative OR
qualitative research in that community during one day. If it appears, following the first days of
data collection, that it is impossible to complete all activities in a single day, then more
enumerators will be located and hired such that this can be done, given the short timeline for
data collection (maximum of 45 days including 1 day off each week as indicated in the work
plan).
If it appears that data collection can be done with 4-6 enumerators in one day, then the team
leader will explore the possibility of completing two communities in one day, either back to
back, or splitting the team in half, always ensuring that there is one supervisory team member
present to provide on-the-ground. In order to ensure quality in the baseline data collection, the
research team will be made up of suitable and well-trained individuals; at every stage in the
data collection, at least one senior team member will be present to provide quality assurance
and oversee field logistics. One qualified and experienced Sierra Leonean will fulfill this role for
at least one week of the data collection upon being trained by the international staff; the Team
Leader will return to finalize the data collection and ensure quality in the upload of data. Draft
field activities are provided in Appendix 1 (Will be adapted accordingly when sites are selected).
Quality assurance of data collection and entry
Certain measures will be taken in order to assure quality of the data being collected by
enumerators who will, at times, be working unsupervised. We will ensure that not only are
enumerators following procedures properly in terms of randomly selecting houses and
respondents, strongly following ethical guidelines, and not rushing through
questionnaires/making up answers just to get them finished, but also when conducting
interviews, are accurately phrasing certain questions that are less straightforward and require
30.05.2013
DRAFT M&E Framework Plan International UK
explanation, and correctly coding/recording those answers when responses may not be
provided verbatim on the questionnaire answer choices, and thus require a bit more thought.
To ensure that data collectors' accuracy in phrasing questions and coding/recording answers,
each field team will have a team leader who, while he/she will be collecting data
himself/herself, will also periodically visit households in which interviews are being conducted,
and observe the enumerator at work, looking over his/her shoulder. This will occur early on in
the data collection (and has been ongoing during training activities) to ensure that each
enumerator understands exactly how to ask each question, and proper ways in which to probe if
that is necessary. Throughout the duration of data collection, 'spot checks' will be conducted in
order to observe this as well. If it is found that an enumerator is repeatedly misunderstanding
answers or questions, then he/she will be re-trained until corrected and, failing that, replaced.
The in-country Field Coordinator, Sharron Kelliher, has good experience training and working
with field teams, and has been working with the enumerators from the start. She will be present
during the launch and also for the first half of data collection activities with select teams,
providing on-the-ground QA.
The local Team Leaders (one for each field team) will be present in the villages at the beginning
of each field activity in order to ensure that households are being selected randomly, and that
certain households are not simply passed-by for a short interview because they do not fulfill
some requirement (i.e. girls aged 9-16). This will be assured because prior to interviews, the TL
and team will select all houses that will be interviewed, and tape the 'cover sheet' on the door
with the name of the first contact in that household, such that the team knows that it is a house
to be interviewed during the day. Additional houses will be randomly selected during this time
that will be visited in the event that one or more of the selected houses did not have a girl aged
9-16 to interview, or if nobody was present/willing to be interviewed throughout the entire day.
At the end of the day, cover sheets will be collected by the TL and matched with the data
entered on the devices; where the names do not match, the TL will follow up. Thus it is expected
that more than ten cover sheets will be collected at the end of each day because it is likely that
at least one of those households will not have a primary caregiver and/or a girl of eligible age
present for interview on that day. If teams repeatedly claim that all persons were present in the
first ten houses selected, then it will be necessary to investigate further to confirm (i.e. to
ensure they've not moved the cover letter to a different house where they knew persons were
present).
It will also be important to ensure that enumerators are not entering false data and/or rushing
through questions such that respondents cannot answer appropriately. In most cases, TLs will
not be present to regulate this while in the field; however, it will be their duty every evening
when backing up the data to their computers to look at a few questions that will flag instances
where data collectors may not have asked questions appropriately. One example of this will be
the question to the girl that asks, 'How often are you happy' and the question immediately after
'How often are you unhappy'. An answer that indicates 'always' for both, for example, will
suggest that either the data collector did not ask the respondent the question at all, or that
30.05.2013
24
DRAFT M&E Framework Plan International UK
he/she was not careful in explaining it such that we obtain valid data. Other questions are
strategically repeated throughout the questionnaire, using the exact same wording, so as to
confirm that a prior answer matches and again, to ensure data collectors are actually speaking
to people to collect data. If the TL notices such answers following a day of data collection,
he/she will personally visit that household the following day to find that person who was
claimed to be interviewed, and determine whether he/she was interviewed at all and how long
the interview took. If he/she was interviewed for the amount of time the interview usually
should take, then the interview will be repeated by the TL and answers will be matched. If the
answers are different for certain factual questions (i.e. not dealing with perceptions or
good/bad, etc), then the previous enumerator will be talked to and possibly released.
Also, every week, raw data will be emailed to the Evaluation TL, Gwen Heaner for detailed
review. Because each enumerator's name is linked to his/her completed questionnaire, this will
enable her to observe certain areas where answers don't make sense and need to be followed
up on. This will provide a level of oversight to the TLs as well, who, while GK Consulting has
worked with them before with excellent results, will likely benefit from another set of eyes on
the numerous questionnaires coming in.
Because the teams are using electronic devices to enter data and then upload that data through
a hard-wired connection to a computer, we can be certain that the data as it will be exported
into SPSS will be correctly entered. Therefore, we will not need to do any double-entry for those
questionnaires completed using devices. In the event that paper questionnaires are completed
(a contingency plan for each field team), then 5% of those will be double-entered. If we find
numerous errors on each questionnaire (+3%) in this method, then all paper questionnaires will
be double-entered.
Finally, in some cases it will be possible for back checking of data randomly, through having a
different enumerator re-visit a previously surveyed household to conduct the interview again,
and then match the answers to those of the previous enumerator. While this will be difficult to
do often, given the constraints on time that the teams are already facing, whenever a team or
individual finishes the targeted number of respondents in a day, then he/she will be tasked by
the TL to return to one previous community nearby, to randomly select one of the households
from the 10 that were interviewed within each, and to conduct the interview again, in full.
Again, because time is limited, and because we have other backstopping mechanisms in place
(above), this will not be a frequent occurrence. At this point, we are setting a goal of conducting
random back checking in 2% of households per district (29 households in total). As this will occur
while data collection is ongoing, any instances where back checking demonstrates inconsistency,
then the responsible data collector will be released.
30.05.2013
25
DRAFT M&E Framework Plan International UK
Recruiting and Training a Suitable Team
Recruitment
The local field team will be made of full-time consultants who will be present for the duration of
the project in addition to temporary day-hire research assistants (enumerators/interpreters).
The full-time team will consist of individuals who are able to work as enumerators and also are
able to assist in key informant interviews and focus group discussions. In order to assure
respondents will feel comfortable speaking to the research team, both genders must be
represented:
- Two facilitators, one male and one female, who will also act as enumerators
- Two note takers, one male and one female, who will also act as enumerators.
Key criteria to consider in recruitment of field team members are: fluency in both English and
Krio, in addition to skills in at least one of the regional languages (Temne, Mende and/or Kono)
(written and spoken); experience conducting field research including delivering face-to-face
questionnaires with people of limited literacy; experience taking part in focus group discussions
or at a minimum having leadership roles during group meetings; age (it is critical to include at
least one young female to talk to the younger girls who may feel uncomfortable speaking to
adults and/or men); basic computer literacy (to assist in uploading data and/or data entry if
necessary). CVs are being collected at this stage and there is a strong pool of candidates from
which we can choose; final decisions will be made prior to arrival in Freetown.
Additional day-hire research assistants (4-6, depending on how long the questionnaires take to
administer during the pilot) will be recruited at the district level and will be high school
graduates at a minimum, fluent in English, Krio and the regional language, and able to translate
questionnaires from English into the appropriate dialect of the respondent, and then record
answers back in English. They will be familiar with electronic devices, field research, and dealing
with children and adults.
Training
Training will take place over one half-day in the office and half-day in nearby communities, and
will continue in the field during the first rural pilot. In the office, the research team will be
briefed on the basics of the project and the goals of the baseline study, and then will be
introduced to the research instruments one by one. A ‘run through’ of all survey questions,
interview questions and focus group questions will take place, allowing each member of the
team to comment on strengths and weaknesses of certain questions in case they need to be
removed, rephrased or elaborated upon.
Then the local research team will conduct ‘mock’ data collection activities using the
international research team as subjects. The international team will give the local team a
number of challenging responses in order to enable them to learn how to deal with issues they
may face in the field. Special attention will be given to the ethical guidelines that every member
30.05.2013
26
DRAFT M&E Framework Plan International UK
of the research team is required to follow and also which they will have to convey and oversee
when working with the district-based research assistants.
Finally, the local research team will be trained as to the best way to select key informants and
identify case studies for further research. While in the field the Team Leader will be responsible
for making the final decision via phone or in person when possible, the local team is expected to
report to the coordinator their ideas.
Pilot
Two pilot activities will take place in communities with schools relatively close to the capital (so
Plan staff can visit and international staff travel less in their shorter time in-country) during
which all field activities will take place (focus group research, household data collection, school
data collection, key informant interviews, case studies). The first will be at one targeted rural
site on the way to Moyamba or in Moyamba (depending on selected school sites); all collected
data will be thrown out. This will serve as both an extended training for the enumerators and
also to enable the international research team to observe how questions are received and
answered by targeted respondents. After this pilot, further training that will be delivered as
needed; the research team will then complete further refinement of questionnaires and
interview guides. The second pilot will employ the finalized instruments and will take place in
one of the randomly selected sites in Moyamba. If Plan project staff would like to observe the
activities, then the site closest to Freetown (Moyamba) can be visited first.
While we intend for the second pilot to yield valid data collection, until we are sure that the
collected data is reliable and accurate, data will be thrown out and more schools will be
randomly selected to obtain the desired sample size as detailed below.
Data entry / maintenance while in the field
It is expected that throughout fieldwork, quantitative data will be uploaded and qualitative data
will be manually entered (if devices are not used) every night after fieldwork activities by the
local research team and field coordinator, assuming electricity is available to power
computer(s). This includes all quantitative and qualitative (i.e. typed transcripts of interviews /
focus group discussions). The field coordinator will be responsible for ensuring this happens and,
when possible (though it is not assumed that this will be frequently possible while in the field)
uploading the data to an online database that the team leader can access. The data will be
reviewed as it comes in for any obvious errors and dealt with accordingly. In the event that it
seems the research team requires more guidance, the Team Leader will return early to ensure
activities go according to plan and data collected is valid.
The field coordinator will also be responsible for keeping track of all paper items through the
duration of fieldwork. Whatever data is not entered after fieldwork has been completed will be
entered from Freetown so that the team leader can access all data prior to the Team Leader’s. A
30.05.2013
27
DRAFT M&E Framework Plan International UK
one-week cushion has been factored into the schedule to allow extra time for data entry or data
collection given the inevitable logistical challenges faced during such work.
Research ethics plan
In any research that deals with vulnerable or marginalized populations, it is imperative to pay
close attention to the potential to do harm through asking questions or eliciting conversation.
The research team will follow ethical guidelines as prescribed by the British Sociological
Association. The team will be acutely aware of the need to obtain vocal informed consent from
EVERY participant in the study; each participant will be told that his/her name will not be used in
any project documents without their explicit permission; pseudonyms will be used in any
narratives, and that we will give a unique ID to each participant, which will be linked to their
names but those names will be kept separately in a password protected document. The
importance of us keeping track of their individual progress will be explained to them such that
they understand its import and can give informed consent.
If, at any point in a conversation, it appears that the participant no longer wants to speak, then
it is imperative that the researcher a) can identify this easily and b) stop the research
immediately. The participant should never be coerced to take part in the first place or to ‘keep
answering’ while taking part.
Certain questions might deal with sensitive topics; therefore it is important that the researcher,
while obtaining informed consent, explain the types of questions that will be asked on the
survey, and assure the participant that a) his/her answers will remain totally anonymous (if it
will be) and b) that he/she can choose to not answer a question if he/she chooses; c) he/she can
stop the interview at any point without question.
It will also be important that only females conduct interviews with girls. The team will consist of
males and females who will conduct household questionnaires; however, for the interviews with
girls, one or two females on the team will be conducting only interviews with girls which are
much shorter; therefore, they can visit houses specifically for these interviews.
We would also like to stress that we expect DFID and PWC to uphold these ethics as a minimum
standard when they are undertaking their own baseline in country. Since they will be considered
a ‘project associate’ of Plan’s during this time, it will also be required that Plan’s Child Protection
Policy, Attachment 5, is signed by anyone visiting our project sites and communities for the
purposes of this project.
Data Analysis
Data will be analysed using SPSS and disaggregated according to, at a minimum, gender, age,
ethnicity, district, degree of isolation from district centre, education level, number of children,
disability status, school enrolment, participation in children’s clubs, feelings of support from
family and community, exam pass rates, learning assessment results, teachers with sufficient
30.05.2013
28
DRAFT M&E Framework Plan International UK
resources, participation in study groups, perception of violence in school, incidence of
mistreatment from teacher, hours spent studying, grade girls’ expect to complete, perception of
importance of school, and girls’ degree of marginalization, [using marginalization index that
indicates parents level of education, disability status, occupation, house type, expendable
income]. The marginalization index has been developed in order to gauge girls’ relative level of
marginalization, and to track how their opportunities change over time (i.e. not whether they
become less marginalized, as this will be recorded at the baseline, but how they are able to have
‘life chances’ despite this marginalization.
Likely variables for regression analyses will consist of that are indicated in the logframe and
based on tentative intermediate outcomes are:
 Disability status vs. school enrolment
 Gender vs. school enrolment
 Gender vs. school success
 Gender vs. supportive parents
 Gender vs. confidence in school to ask questions
 Gender vs. importance of school
 Gender vs. grade expected to complete
 Gender vs. perception of violence in school
 Marginalisation status vs. school enrolment
 Marginalisation status vs. supportive parents
 Marginalisation status vs. school success
 Mistreatment in school vs. feeling comfortable to report it
 Disability status vs. enrolment
 Disability status vs. supportive parents
 Disability status vs. school success
 Disability status vs. perception of violence in school
 Disability status vs. grade expected to complete
 Disability status vs. importance of school
 Disability status vs. confidence to ask questions in school
 Perception of violence in school vs. school success
 Perception of violence in school vs. enrolment
 Supportive parents vs. enrolment
 Supportive parents vs. grade expected to complete
 Supportive parents vs. school success
 Participation in study groups vs. school success
 Degree of isolation vs. school enrolment
 Degree of isolation vs. supportive parents
 Degree of isolation vs. good teachers
 Marginalisation status vs. grade expected to complete
 Marginalisation status vs. importance of school
30.05.2013
29
DRAFT M&E Framework Plan International UK
This list is by no means exhaustive bit is intended to provide an indication of some of the likely
areas of significant interest.
Qualitative data analysis
Further variables for regression analyses will be informed by qualitative data collection (ie when
it becomes clear that certain variables may be linked). Qualitative analysis will also become
important in understanding any trends that are found in regression analyses, or in case expected
trends are not found. In these cases, narratives surrounding these relationships will be explored
in order to add rich data to the findings, and provide more nuances to the reported correlations.
These findings will also be used to verify reported trends that will ensure that the quantitative
data collected is valid. In the event that qualitative data and quantitative data do not agree, this
will be presented in the final reports and flagged for follow-up.
Other qualitative findings will be coded and logged, and analysed for trends. This is particularly
useful when dealing with certain hard to measure indicators (marginalization, goals,
confidence), which require a more significant level of explanation and subsequent discussion in
order to get valid data. Close-ended questions will also be analysed in order to highlight trends
and associate those findings with rich narratives.
Finally, qualitative data will be used to highlight significant stories of change for girls, parents,
teachers and others; these individuals will be followed over the duration of the project for
follow-up interviews in order to provide more data into how their lives are changing, and why.
These stories will be compared with quantitative data from those same individuals in order to
confirm that findings are valid.
Final report outline
The final report will be outlined with a dual purpose: a) to provide all the data that is important
for the specific log frame indicators that will be measured over the duration of the project; b)to
provide background information surrounding the differences in certain project sites, so as to
better inform project staff for programme planning. The report will be thus first include a
section on the literature surrounding girls’ education globally, and specific information
surrounding Sierra Leone. Then the report will describe the methodology for the baseline and
the problems encountered and thus any limitations in the data. Next the report will highlight the
major correlations that were found that relate specifically to project log frame indicators and
enrich the discussion with qualitative findings, including relevant case studies. Then the report
will highlight any trends that were not found (but were expected to) and explain reasons based
on qualitative data.
The report will then discuss any discrepancies within the data. In the second major section of
the report, other relevant trends that were found will be highlighted that may be useful
information for project planning purposes (i.e. not specifically related to the baseline and/or log
frame as it presently exists). Finally, the report will outline broad recommendations for
30.05.2013
30
DRAFT M&E Framework Plan International UK
measuring these indicators over the duration of the project, including the mid-term and final
evaluations. A comprehensive appendix will provide further tables that may be of interest to
project staff.
30.05.2013
31
DRAFT M&E Framework Plan International UK
Section 2: Evaluation Plan
a. Key Evaluation Questions
Project monitoring and evaluation will seek to answer the GEC programme evaluation questions,
as well as those posed by Plan’s Programme Accountability and Learning System and Evaluation
Standards.
 To what extent was the GEC successfully designed and implemented?
 What works to increase the enrolment, retention and learning of marginalised girls?
 What impact did the GEC funding have on the retention and learning of marginalised
girls?
 To what extent did the GEC represent good value for money?
 How sustainable were the activities funded by the GEC and was the programme
successful in leveraging additional interest and investment?
In addition to using GEC questions and OECD-DAC evaluation criteria, the following are crosscutting questions for Plan’s evaluations:
 Child-centeredness – to what extent were children involved in the project, how were
they selected, what was the impact on boys and girls of their participation in the project
and how did the project affect girls and boys, directly or indirectly, positively or
negatively?
 Non-discrimination and inclusion – who benefited from the project and who was
excluded, and why? How were marginalised/ vulnerable groups included?
 Gender – to what extent did the project contribute to increased equality between boys
and girls, women and men? To what extent was the project gender transformative?
During the first quarter of the project Plan and partners will also be looking at a broader learning
strategy for the project with specific research questions.
b. Logframe Indicators
It is recognised that the project logframe needs revising to ensure the appropriate level of
outcome and output indicators. This will take place together with further revisions to indicators,
targets and assumptions following the baseline (and within 30 days of the baseline report being
submitted). The project will measure the additionally of the intervention through comparing
treatment groups with control groups. This is discussed in section 1. Outcome level indicators
are dealt with in more detail below:
Retention: Data on enrolment and repeaters per grade for the 2012/13 academic year and the
2011/12 year will be collected at baseline for project and control schools, allowing calculation of
retention. This will be repeated annually at the beginning of each academic year. Data will be
collected directly from schools by project staff (as data available from District Education Offices
30.05.2013
32
DRAFT M&E Framework Plan International UK
varies in reliability) using the School Selection and Survey Form (attached) and stored on a
central school-level EMIS database.
Thus baseline figures for enrollment will be set from numbers of girls and boys per grade
enrolled in all target schools in the 2012/13 academic year. The same data will also be collected
for the 2011/12 academic year, along with numbers of male and female repeaters per grade in
2012/13 allowing calculations of retention. Data on the numbers of children with disabilities in
all project schools will also be calculated for the 2012/13 academic year as part of the baseline,
disaggregated by sex and disability type though not by grade. More detailed information on
children with disabilities enrolled in schools will be collected through HI’s monitoring activities.
The same data as above will be collected for all intervention and control schools. The
additionally of the project on numbers of girls enrolled and retention, and number of children
with disabilities enrolled, will be measured through analyzing changes pre to post-test in
intervention schools against pre to post-test changes in control schools. National level EMIS data
will provide a further reference point for analyzing changes in enrollment.
Attendance: Attendance data was not collected during school selection as requiring
substantially more time per school, and historical attendance data is not always available from
schools, but will be the focus of various follow-up activities and the baseline survey. Attendance
data will be collected from the household questionnaire and questions on estimates of the
number of days of school children have missed (in control and target groups). In the five schools
randomly sampled for in-depth qualitative research (and in control schools) detailed attendance
records will be compiled from school registers, giving an average for all classes in targeted
ranges.
Spot checks will be carried out on a regular, rotating basis by project field staff. Headcounts of
girls and boys in school on a given day will be compared with the number of students officially
enrolled to give a snap-shot of attendance. The independent baseline survey team will also carry
out spot-checks in all sampled treatment and control schools at baseline. As with all other
indicators, the baseline process will be repeated at mid and end-line allowing the additonality of
the project’s impact in terms of attendance to be measured in terms of pre to post-test changes
in control and intervention schools to be reported on at each milestone.
Learning: The project will collect data on exam pass rates (NPSE and BECE) in all intervention
and control schools. Such data is only available for the previous academic year and it is
recognised that this is not a sufficient measure on its own. In the current absence of EGRA and
EGMA tests adapted and validated to Sierra Leone, the project will adapt an ASER test for
literacy and numeracy based on the age-appropriate school curricula. This will be piloted as part
of the baseline methodology and tools pilot. This test will be administered to all individual
children surveyed through the household questionnaire (control and intervention groups) and
additional project beneficiaries.
The learning assessment for literacy and numeracy is attached. These literacy and numeracy
tests were developed by Plan for the GEC in Sierra Leone. They have been informed by
30.05.2013
33
DRAFT M&E Framework Plan International UK
international assessments such as the ASER and Uwezo and aim to assess girls’ basic literacy and
numeracy skills. They are based on the national curriculum and national literacy and numeracy
targets have been taken into consideration when designing the tools. Finally, the design has
been informed by an initial pilot with primary school girls in Sierra Leone. The learning
assessment will be carried out with girls as part of the household survey and through a random
selection of girls in target grades in control and intervention schools, as detailed under the
baseline section above. All girls surveyed will form a cohort tracked over time. In addition, all
disabled children benefitting from CBR activities will be administered the test, a total of 2,125
girls and boys. Table 1.3 on page 6 summarises the number of girls who will be assessed at each
wave in control and intervention groups.
Using ASER tests will provide internal validity within the project (and control) over time, but not
external validity. The additionallity of learning outcomes will be measured through changes pre
to post-test within control and intervention groups. At mid-term and end-line the learning
assessment will be re-administered to the same project beneficiaries and the same girls
interviewed through the household survey. Within control schools the same cohort of individual
girls will also be followed throughout. Re-contact details will be recorded at baseline. If girls in
the cohort drop out of school their replacements will be selected randomly, assuming there are
sufficient girls still enrolled in the grade (the average number of girls enrolled in selected schools
in JSS2 is 34, so at JSS level a high proportion of all girls per grade will be tested).
Leverage / sustainability:
Specific indicators to be measured regarding structural and systematic changes are outlined in
the logframe indicators below. Output 4 in particular (the school score-card) is designed to both
bring about and to measure changes in school and district level policies and practices. In
summary, the project will assess:
 Changes in parental / community attitudes to girls education and the value of educating
disabled children, evidenced through household surveys pre and post-test in control and
intervention communities and qualitative research
 Changes in the knowledge, attitudes and behavious or teachers and school authorities in
relation to gender responsive pedagogy and inclusive education, evidenced through preand post-training questionnaires and triangulated with feedback from girls in schools
(through survey, qualitative research and score-cards)
 Changes in the attitudes of other key duty-bearers (local Chiefs, District Education
Officers et.), evidenced through key informant interviews pre and post-test
 Changes in the existence and implementation of Teacher Code of Conducts and in
particular their impact on violence and gender based violence within schools, evidenced
through number of schools with a CoC in place triangulated with girls’ feedback and
interviews with duty-bearers
 Changes in, and extent of implementation of, local by-laws at the Chiefdom and District
levels, evidenced through local policy mapping to be carried out in first year of project,
process tracing, and score-carding activities and follow-up
30.05.2013
34
DRAFT M&E Framework Plan International UK
Changing community and individual behaviour and attitudes: This will be measured by tracking
the number of local by-laws created and implemented to hold teachers accountable to the
teacher Code of Conduct. Additionally, individual and community attitudes and behaviour will
be measured through household questionnaires pre and post-test (and in control) and by
looking at the number of girls and children with disabilities accessing and being retained in
schools. In addition, focus groups at the community level will assess changing attitudes to girls'
education, violence and gender equality. Focus group discussions and KIIs with members of
parent advocacy groups and board of governors at baseline and end line stages will be
specifically designed to assess how attitudes and behaviours have changed through the life cycle
of the project. The data collected by the project will enable us to explore changing attitudes of
girls towards their own education (and towards gender equity) through questions around their
own aspirations and their perceptions of older girls and young women in their communities.
Similarly for parents and teachers of girls, by looking at their assessment of girls' futures at the
start and end of the future (per se and compared to their assessment of boys' futures) we will
be able to comment on the difference our interventions have made towards attitudes.
Influencing policy makers: The score-carding and interface meetings process will highlight issues
of concern at the school level, district and national level. The Project will document what issues
have been acted upon at the district and national levels and who has been involved in the
process. The project will also document what interventions are having a positive impact on
marginalised girls' learning outcomes. For example, we will track the progress of the learning
assistants through the process of attending distance education courses and can thus
demonstrate the effectiveness of this as a model for getting more female teachers into the
education system. Where the project has demonstrated success, we will advocate for these
interventions to be replicated and will track any uptake thus measuring outcome indicator 4.
Measuring changes in relation to disabled children
Definitions of disability as used by the project are given in Attachment 6. The project will
measuring changes in relation to disabled children and the effectiveness of the CBR approach on
a number of fronts. Through the household questionnaire, data will be collected on the
attitudes of men, women and girls (including some girls with disabilities) to the value of
education children with disabilities. This will be strongly supported by the qualitative research.
Focus group discussions include questions around attitudes to children with disabilities, their
education, and the systems of support available for them. A focus group will also be held
specifically with girls with disabilities.
While the number of disabled girls interviewed in the household survey will likely be relatively
low and will not give a statistically significant sample as a sub-group on its own, relevant
questions, and the learning assessment, will be administered to all disabled girls and boys
supported by the project (through CBR activities) pre and post-test. There will not be a
significantly large control group specifically for children with disabilities, but changes in disabled
30.05.2013
35
DRAFT M&E Framework Plan International UK
girl’s learning will be compared to other girls in control and treatment groups allowing a form of
‘internal’ difference-in-difference analysis of changes. The numbers of girls and boys enrolled in
primary and JSS (control and intervention schools) will also be measured at baseline through the
school survey and annually thereafter.
Output Indicators
We acknowledge the phrasing of the outputs themselves is perhaps overly ambitious for output
level and in relation to the predetermined Outcome. We will consider revising these as below
after further discussion with all project partners and prior to the final submission of the
logframe.
- Output 1: Parental and community attitudes support girls education and financial and
disability-related barriers to marginalized girls’ education reduced
- Output 2: Girls have positive female role models and are supported to build the skills
needed for life
- Output 3: School staff and management supported to provide safe and inclusive
learning environments
- Output 4: Girls voices and needs are listened to and responded to in educational
decision-making
Plan has tried to achieve a balance of logframe indicators covering direct outputs of the project
with indicators measuring changes in attitudes. Plan has also developed a set of draft
‘intermediate outcomes’ for the project, which we see as joining the direct deliverables
(measurable annually and not just at base/mid and end line if reliant on the survey data) and the
predetermined outcome indicators. These intermediate outcomes do not fit easily within the
logframe format as given but are areas we are interested in measuring. Not all of these are
suitable for logframe indicators. These changes to the logframe have not been discussed or
agreed with all partner organizations and are in no way final. A final logframe will be submitted
on 16th August 2013.
All relevant indicators will be disaggregated by Primary and Secondary levels. Relevant
indicators will, where possible, be disaggregated by disability status. This will not be possible
through the household survey alone as to measure this sub-group is beyond the resources of
the project. However, the same exact questions will be asked of all children with disabilities
engaged in the project through the CBR activity, increasing the effective sample. The control will
not be significant for this sub-group, but changes can still be referenced to the target population
as a whole.
Table 1.9: The Programme Logframe
Additional Output and Intermediate Outcomes for Output 1
Indicator
Source
Baseline
Mean # of hours girls
spend
learning/studying
Household Survey
-
Milestones/
Targets
+30% [Estimate
based on
assumed
Notes
Added to logframe
output 1 to replace
current output
30.05.2013
36
DRAFT M&E Framework Plan International UK
outside of school
during term-time.
% of girls receiving
bursaries who
complete an academic
year
baseline figure
of 5 plus 2 hours
of children’s
clubs giving a
40% increase for
participants]
95% of girls
receiving a
bursary
completing the
academic year
-30% [bursaries
received by
approx. 40% of
target girls but
cannot assume
all surveyed HH
will benefit]
indicator on NPSE
and BECE pass
rates
Will be
disaggregated by
disability (same
questions to be
asked of CwD
referred through
CBR approach)
Project
Management Data
(individual tracking
of all bursaryholders annually)
Household survey
[C98d]
0
% of girls expressing
positive attitudes to
benefit/right to
education
Household survey
[exact indicator to
be refined when
survey instrument
finalized]
-
+30%
% of parents who take
an active interest in
their child’s learning /
in the performance of
their local school
% of marginalised girls
who feel their parents
and community
members support their
education
Household Survey
-
+20%
# of CWD enrolled in
target Primary schools
Project Monitoring
Data (as relating
to CBR activities)
and School
Survey Data
School Survey
Data of all project
and control
schools
% parents/care-givers
with girls of schoolgoing age (appropriate
for P5, P6 and JSS1)
citing financial barriers
as a reason why their
girl child is not in
school
% of CWD as a
proportion of school
population in PS
-
Added to logframe.
Will be
disaggregated by
girls in/out of school
prior to bursary
Added to logframe
Current logframe
indicator. Will be
disaggregated by
disability status (see
note on disability
above)
-
2,052
Added to logframe
[1.4% from one
district]
No target
Suggest to monitor
but not use as
indicator as
increase in
enrolment of nondisabled girls as
well as CWD so
ambiguous
interpretation]
Milestones/
Notes
Additional Output and Intermediate Outcomes for Output 2
Indicator
Source
Baseline
30.05.2013
37
DRAFT M&E Framework Plan International UK
% of Las passing
WACE or TCC
entrance exam
Project Monitoring
Data
-
% of schools (PS) with
at least one female
teacher (qualified or
unqualified)
School Survey
data
% of girls who believe
they have a positive
female role-model in
their school or
community
# of girls with a
supportive female role
model in their school
or community
Household Survey
[needs to be
added]
[approx.. 45% of
schools with
one or more
female teachers
in one GEC
district]
-
% of girls who believe
that going to school is
important for their
future opportunities in
life
Household survey
Targets
80%
[LAs in 180 of
540 PS so target
change diluted]
Added to logframe
Project Monitoring
Data
Relating to mentees
and girls benefitting
from LAs – note
double counting if
including both. If just
mentees same as
current output
indicator
Likely to be high at
baseline, so difficult
to see large
changes.
-
Additional Output and Intermediate Outcomes for Output 3
Indicator
Source
Baseline
Milestones/
Targets
% of girls agreeing
they have been
'mistreated physically,
sexually or insulted by
a teacher'
% of girls (aged 10 or
above) who believe
that sexual
abuse/harassment is
never appropriate
Household Survey
[composite
indicator from
questions relating
to GBV]
% of girls (by PS and
JSS age) with
knowledge of sexual
Household Survey
[similar
questions in
PPA2 baseline
gave 95% of
girls agreeing
that sexual
abuse not
appropriate]
[in PPA2
baseline,
Added to logframe
+20%
Notes
Currently in
logframe – may be
removed. Overlap
with indicator 3.2
and difficulty in
interpreting changes
in reported rates of
violence. Replaced
by indicator below
May be added to
logframe indicator
(output 3) but very
low baseline rate
makes significant
change unlikely
+30%
30.05.2013
38
DRAFT M&E Framework Plan International UK
and reproductive
health and rights (i.e.
correct answers to
questions about
pregnancy or condom
use)
approx. average
of 35% of girls
able to give
correct answers
to questions
about SRH
knowledge]
-
% of school board
members trained
demonstrating
improved knowledge
of teachers code of
conduct
Pre and post
training
questionnaires
(inc. questions on
appropriate
reporting
mechanisms for
GBC)
% of teachers trained
Pre and post
able to demonstrate
training
understanding of
questionnaires
gender responsive
(inc. questions on
pedagogy and
self-reported
teachers’ code of
classroom
conduct
practice and GRP
techniques)
Additional Output and Intermediate Outcomes for Output 4
Indicator
Source
Baseline
% of schools using the
score-card with
positive change on at
least one indicator as
a result of students’
voices and actions
# of changes in
school/local by-laws,
education policy or
policy implementation
in support of girls’
education and/or
children with
disabilities
+60%
Replacing indicator
3.3
+60%
Replacing indicator
3.4.
Notes
Not easy to
establish targets
based on robust
data but will be
monitored
regardless
Project Monitoring
data and scorecard reporting
0%
Milestones/
Targets
50%
Qualitative
reporting through
score-card
process, interface
meetings, case
studies and
project reports
-
-
To replace logframe
indicator 4.1
c. Evaluation Approach
The evaluation will take a quasi-experimental approach, with a control group (as described in
Section 1) to allow for difference-in-difference analysis as well as looking at pre- and post-test
changes in key outcome indicators. A longitudinal survey will be employed to track changes of
individuals surveyed. Future waves will return to the same respondent using the re-contact data
from the household questionnaire, GPS coordinates and written descriptions. However, a
degree of respondent attrition is inevitable and this has been factored in to the sample size. Plan
is committed to the independent evaluation of the project and the transparent and active
dissemination of the final results.
30.05.2013
39
DRAFT M&E Framework Plan International UK
The final evaluation will be carried out by an external evaluator(s) independent of all partner
agencies. The evaluation strategy will be informed by the GEC programme-wide evaluation
questions, Plan’s Child Centred Community Development (CCCD) approach and Programme
Accountability and Learning System (PALS). In essence, the external consultant will be asked to
evaluate how far Plan and partners have gone towards achieving the outcomes and objectives,
and to assess the programme in line with the principles of PALS and OECD-DAC criteria. The
evaluation will seek to question the assumptions and specific outcome areas of the project in
terms of increased and inclusive access and enrolment, improved quality of learning, safety
within schools, increased voice of girls and strengthened accountability of school systems. It will
evaluate the intervention strategies proposed assessing: (a) whether access was supported
through changing attitudes to girls’ education, violence and gender equality and targeted
financial support; (b) whether girls retention and learning was supported by mentoring, the
presence of learning assistants in classrooms and inclusive, gender-responsive teaching; (c)
whether girls safety in school improved and influenced retention and learning outcomes and (d)
whether school performance and accountability was influenced by the scorecard process and
school, chiefdom, and district level advocacy. In evaluating the influence of project
interventions, the evaluations will consider whether the project represents value for money in
terms of Economy, Efficiency, Effectiveness and Equity.
It will not be possible to consider multiple treatment arms within the impact assessment i.e. to
assess the impact of activities relating to disabled children separately from the project as a
whole. There are too many possible ‘treatment arms’ (including: bursaries; mentoring; study
groups; children clubs; CBR; teacher training; learning assistants; and school score-cards) to
quantifiably asses the relative impact of each in relation to each other and the control. On a
purely quantitative basis (i.e. data generated through the household survey) the final impact
assessment will assess the impact of the project as a whole (where impact is the change pre to
post test in relation to the control group, based on a difference-in-difference approach). That is
not to say that individual elements of the project will not be assessed, but that the data sources
will be more varied. Project monitoring data relating to girls receiving bursaries and children
with disabilities enrolled through the CBR approach etc., learning assessments with specific
beneficiary groups will provide a strong basis for assessing individual strands of work within the
project. In-depth qualitative work (focus groups, key informant interviews, and process-tracing
case studies) will further elucidate this data to more fully understand the results of different
activities individually as well as the synergies between them.
30.05.2013
40
DRAFT M&E Framework Plan International UK
d. Research Methodology
Research will be conducted in line with internal guidance on Child Protection and ethical
standards. An ethical protocol has been drafted by project partners to ensure a shared
commitment and understanding of ethical and child protection considerations specific to the
project.
The external final evaluator will have a large amount of existing data to draw on, and will be
asked to review and assess the quality of data provided, synthesis learning and use it to assess
outcome level changes in the target group over time in relation to their level of marginalization,
compared with internal differences in intervention exposure, the control group and national
level secondary data. Data sources from all MER activities that contribute to the final evaluation
research methodology are detailed in the project logframe on ‘Tab 8: MER Data Sources.’
Specific to the final evaluation will be an independent, representative, quantitative communitysurvey and in-depth qualitative research in schools. Primary data collection for the final
evaluation will be collected by independent data enumerators, not by project staff.
Quantitative household survey from a representative sample of marginalized girls: The
quantitative community survey will use the same representative sampling strategy as the
baseline and will ensure a 95% confidence level with a ±5% interval. This survey will return to
the same communities as sampled in the baseline, though will not explicitly seek to interview
the same respondents, instead using a random sampling method to select households within
the communities. Certain questions on community attitudes to girls’ education and gender
norms, for example, will be the same as in the baseline questionnaire to ensure consistency and
provide updated data for logframe indicators. The survey will measure the degree of exposure
to the intervention and results analyzed in terms of the vulnerability status of respondents and
relative outcome level changes between beneficiaries and non-beneficiaries.
Qualitative research data: As at baseline, quantitative survey data will be accompanied by
qualitative research, though shifting focus from identifying barriers to education to examine
stakeholders’ experiences of the intervention, and intended and unintended, positive and
negative outcomes. Qualitative research will include: KIIs with community and religious leaders,
district education officials and other duty-bearers regardless of their participation in project
activities (and with head teachers of control schools), to examine changes in community
attitudes and the education policy context and the extent to which these are attributable to the
intervention and/or other political or social changes. KIIs and FGDs will also be held with a larger
sample of stakeholders involved in the project. At the community level (in communities already
sampled for the qualitative research at baseline), FGDs will be held with parents, girls and boys
in separate FGDs. These will be focused on providing contextual understanding in relation to the
quantitative data to understand why, or why not, changes in community attitudes and gender
norms have come about. The qualitative element of the community survey will not aim to be
30.05.2013
41
DRAFT M&E Framework Plan International UK
representative of the total population of the target districts but, as at baseline, will involve a
purposive sample for maximum variation against key characteristics.
At the school level, research will focus on specific groups of project beneficiaries already
involved in baseline and mid-term. The final evaluation will thus follow-up on with the same
cohort of groups and so will include girls who have received bursaries (and separate FGDs with
their parents); girls with disabilities; members of study groups and children’s clubs; interviews
with random samples of Learning Assistants and teachers who have received in-service training,
and FGDs with teachers at project schools; sampled Boards of Governors and Parent Advocacy
Groups.
In addition to the primary data collection, the evaluation will also pull together learning from
annual programme reviews (APPRs) and all school and individual beneficiary data, as well as
aggregate results from participatory monitoring tools.
e. Sampling Framework
Details of the sampling strategy to be used at baseline, and repeated at endline, are given in
Section 1. Target groups will be tracked over time through the household questionnaire,
administered to project beneficiaries, non-beneficiaries, and a control group at baseline and at
endline. Project monitoring data specific to girls receiving bursaries, the community based
rehabilitation approach, learning assistants and other beneficiary groups will allow for further
tracking of individuals over the course of the project.
A note on the sampling framework submitted: The sampling frame submitted is complete for all
five districts. Population data is, by and large, available only for communities in which schools
are physically located, not all feeder communities which are often small villages for which data
is not available from the 2004 census (the basis for population data used in the sampling frame).
As sample size and stratification is not dependent on community population (stratification is at
the first stage of sampling based on numbers of target schools per district) lack of population
data does not affect the sampling strategy at this stage. Listings of all feeder communities (for
the second stage sampling) are currently within a different data set to the sampling frame and it
is proposed to keep these lists separate (so the sampling frame will be used for the first stage
sampling, then for selected schools, the second stage sampling will be based on a separate list of
schools and their feeder communities). Further, due to small size of some feeder communities
that may be randomly selected, more than one community may end up being sampled in some
instances to reach the number of households per cluster. In such cases, additional communities
will also be randomly selected.
Construction of the Sampling Frame
The sample frame has been developed from two separate sources of data: a list of all schools
selected by Plan for the GEC project and the communities they are in (and some nonintervention schools and communities); and a list of communities with population data from the
30.05.2013
42
DRAFT M&E Framework Plan International UK
2004 Census. Neither data set can be considered wholly complete nor accurate but are the best
that is available in Sierra Leone.
Selection of schools and control
The list of schools was obtained from the school selection and survey process. In one district
(Port Loko) more schools were surveyed then were ultimately selected for inclusion in the
project as intervention schools. This provides some information on similar schools (that meet
the same eligibility criteria) which are not going to be intervention schools, some of which have
been identified as suitable control clusters. However, this is only the case for Primary Schools as
there are no other eligible JSS in the district (the case in 3 out of the 5 target districts).
In Moyamba district, all surveyed schools have been selected. Within Plan’s areas of operation
there was little choice as to eligible schools and so field staff did not need to survey schools
prior to selection (though all selected schools have been surveyed). This is why there are fewer
JSS than planned in Moyamba (43 instead of 45), and thus fewer Primary Schools (as three PS
have been selected per JSS), the shortfall being made up from an increase in the number of
selected schools in Port Loko.
Please see Section 1 above for a discussion of the issue of control clusters. Once control sites
have been selected and the sampling strategy agreed, Plan will double-check all control sites to
ensure minimal contamination from project areas and investigate more fully to see if other
external education initiatives may have a bearing on the controls.
Geographic data
No data is given for the detailed location or directions to communities in the sample frame.
Distance to each community (or rather the school associated with a community) from the
District center is given in the school data by way of indication of the ‘remoteness’ of the
school/community. Latitude and Longitude are provided but there are a number of errors and
no guarantee is given as to the accuracy of GIS data provided at this stage. Geographical data
will not be completed for all communities in the sampling frame at this stage as this is not seen
to be necessary for sampling nor an efficient use of resources. When sampling points have been
randomly selected (for Plan schools and for the EM’s sample), further details for those points
only will be added to the sampling frame.
Population data
Population data given for each community is based on data from the 2004 census. An annual
change has been estimated from World Bank data available up to 2011 giving a total increase of
21% in population estimates per community. The estimated number of children aged 5 to 15 has
been calculated based on DHS 2008 data giving the percentage of children in the 5 to 14 age
range as 33%.
30.05.2013
43
DRAFT M&E Framework Plan International UK
44
Population data and data relating to EAs is not available for all communities. Community names
are taken from the school selection survey, which verified the location and name of all
communities where there are target schools (and in some cases non-target schools). Not all of
these communities can be identified on the list of communities from the 2004 Census. There are
a number of possible reasons for this, including: some selected communities are small and may
not have been captured on the Census (or have changed since 2004); variations in names and
spellings over time and between official records and names used locally.
It should also be noted that the Census often contains duplicate listings for the same
town/village where a single locale is split across multiple EAs. In constructing the sampling
framework, only one EA has been used to give population data (the first listed for a given
community, where the first is the lowest numbered EA Geocode). Thus, population estimates
drawn from the sampling framework for the total population of all listed communities is likely
an under-estimate. As the sampling framework does not include all feeder communities (but
only those in which schools are located) it gives a further under-estimate of the total population
within all project schools. If such data is needed it would be better sourced from the relevant
Section or Chiefdom sub-totals. Data on the size of schools is available for all project schools.
f.
Value for Money Strategy
During our start up workshop in January, Plan with all consortium partners spent time working
on the concept of ‘value for money’ and how we could best apply this throughout the project.
The presentation for this session is attached in Attachment 3 and a very simple Value for Money
Tool like the table below was drafted, and is being used by every partner, so that every
expenditure can be analysed against the ‘4 Es’.
Ref
No.
Which
E?
Decision being
made
Quantative Unit where possible
(money, people, time …)
How was the decision
made?
Result Comparison
1
2
3
4
5
6
7
8
9
10
11
12
30.05.2013
DRAFT M&E Framework Plan International UK
g. Evaluation Governance
All evaluation processes will be overseen by an Evaluation Steering Group which was established
at our M&E Workshop in-country in March. The Steering Group will comprise of the Chief of
Party, the CCU M&E Coordinator, the Plan SLE M&E Manager, a representative from each
partner organisation, a representative from the Ministry of Education, the Ministry of Social
Welfare Gender Children’s Affairs, and district representatives locally. The Plan UK Learning and
Impact Officer and GEC Programme Manager will be on the group remotely via telephone or
skype. In order to have an independent member on the group, we have invited Gwen Heaner,
our baseline and hopefully mid line and final evaluation lead, to be on the group. The Plan UK
members of the group joined with the Learning and Impact Assessment Manager and the
Education Advisor to form a Baseline Tender Committee in the UK to manage the baseline
recruitment process. Going forward the Evaluation Steering Group will have the following ToR
(as agreed at the above mentioned M&E Workshop in March during a session dedicated to this
issue):
ToR for Evaluation Steering Group
Frequency of Meetings: Quarterly
Membership: Chief of Party, CCU M&E Coordinator, Plan SLE M&E Manager, IRC M&E Lead,
FAWE M&E Lead, HI M&E Lead, Pikin to Pikin M&E Lead, GK Consulting Director, Ministry
Representatives from Ministries of Education, Social Welfare Gender Children’s Affairs, and local
government district representatives. (The Plan UK GEC Programme Manager and Learning and
Impact Officer will use phone or skype to take part in each meeting)
Tasks:
 Learning from partners’ experiences, sharing good practice
 Discussing how to disseminate good practice
 Ensuring child protection and ethical standards in all M&E activities are understood and
adhered to
 Supporting dissemination of M&E findings with project stakeholders and beneficiaries in
appropriate formats
 Organising learning events
 Identify capacity needs in M&E and suggesting ways of addressing gaps
 Drafting ToRs for external research/evaluations etc
 Reviewing project performance information and making recommendations to resolve
under-performance
 Sharing information on key M&E processes across different activities / districts
 Facilitating communications and information sharing
It was also decided that the GEC project partners should have a Learning Plan. Going forward
the Evaluation Steering Group will have the following Learning Plan as a minimum:
30.05.2013
45
DRAFT M&E Framework Plan International UK


Quarterly reviews (through evaluation committee)
Learning events: bi-annually, lasting 1 to 2 days each. First one to be end of August /
start of September*
*A bi-annual reflection and learning event will be held with all project stakeholders. The first of
these is to be scheduled for late-August, to provide a chance to look back at the work in schools
in the first academic year and training carried out over the school holidays, to share experiences
and to feed learning into activities scheduled for the next academic year.
The reflection and learning meetings will be the responsibility of the CCU M&E Manager
supported by the Evaluation Steering Group, who will feed in to the agenda and planning.
External stakeholders will be invited such as relevant government officials and staff from the
other GEC project in Sierra Leone implemented by BRAC. A small number of teachers, mentors
etc. also involved in the project will also be invited (possibly from one district at a time on a
rotating basis – so it may be best if the event is held in the district). Representatives of girls and
boys involved in the project (perhaps members from one children’s club, again changing on a
rotating basis) will also be invited, as long as their meaningful and safe participation can be
assured and is adequately planned for.
h. Performance Management Framework
The Chief of Party and M&E Coordinator will be responsible for collating quarterly performance
reports which will be the basis for reports submitted to DFID and progress will be reported
through quarterly Project Strategic Group meetings to examine data from all monitoring
activities and ensure this is used appropriately to shape project implementation.
Annual participatory programme meetings will be held inviting district and national level MEST
representatives, donors, UNICEF, NGOs and local stakeholders. These meetings will be used to
collectively review and validate annual performance data, as well as share and address project
implementation challenges and successes. The project’s design also emphasises shared learning
at project level. Community engagement and participation are critical to many project activities
and specific budgets have been allocated for sharing events relating to children’s clubs at
chiefdom level, chiefdom and district-level interface meetings, and CBR best practice exchange
workshops in every district. The project will document the processes and findings of the
scorecard tool used, Women into Teaching component and CBR approach to be shared with
local and national level stakeholders.
Learning from evaluations, and all other data sources, will be actively shared with a view to
improving the design and implementation of future girls’ education programmes of Plan and all
other partners. Nationally, evidence from the project will be shared with MEST, UNICEF, other
NGOs and education networks. Internationally, learning from the project will be used to inform
Plan International’s Because I am a Girl campaign, aimed at reaching four million girls globally.
All partners will share learning through their own technical networks and will actively seek
30.05.2013
46
DRAFT M&E Framework Plan International UK
opportunities for disseminating learning to others in the sector. All relevant project details will
be made publicly available in line with Plan UK’s published commitment on IATI.
30.05.2013
47
DRAFT M&E Framework Plan International UK
Section 3: Monitoring Strategy
a. Data Collection Strategy
A detailed Monitoring and Evaluation plan was jointly drafted by all project partners, setting out
what data is to be collected, when and by whom. This document is still in draft, while
standardised data collection forms are being developed for each and every activity. The basis for
the monitoring plan is the detailed implementation plan developed by the project and the
project level logframe output indicators. Per output indicator, the project has established key
activities to be monitored consistently, and ‘intermediate outcomes,’ for example, employing
simple pre and post training questionnaires with teachers to monitor changes in knowledge and
attitudes as a direct result of the training component. When finalised, the complete M&E Plan
and all data collection tools will be shared with the EM.
All data collection tools and approaches are to be jointly agreed by all partners, with
standardised formats and pre-agreed definitions. Responsibility for collecting data using these
formats is for the agency leading a particular activity in a given district, with oversight by the
agency providing technical support for the activity, the relevant M&E Officer in the district and
ultimately the project M&E Manager.









Output 1.1: Number of bursaries provided to marginalised girls
Output 1.2: Number of children's clubs that are established and active
Output 2.2: Number of girls participating in girls' study groups
Output 2.3: Number of marginalised girls supported with Mentoring
Output 2.4: Number of Learning Assistants passing TC entrance exam
Output 3.3: Number of board members trained on teacher codes of conduct and the
value of girls' education
Output 3.4: Number of school staff trained on inclusive education, gender responsive
pedagogy and Code of Conduct
Output 4.1: Number of JSS schools that have implemented child-led score-carding
Output 4.2: Number of events/ interface meetings where members of children's clubs
get to discuss issues of concern
Key data collection M&E events are presented in the table below.
30.05.2013
48
DRAFT M&E Framework Plan International UK
1
Month
J
Academic
Year
Project
Milestones
2
F
3
4
M A
Inception
Period
5
6
M J
7
J
O N
D
J
F
M A
Project review and learning event
Beneficiar School
y selection Y2
enrolm
&
assessme ent
data
nt
A
M J
J
A
S
O N
School Year 2013/14
Milestone 1: End of School Year 2013/14
Key
Reports
Reports
9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36
S
School Year
2012/13 Term 3
Baseline
M&E
Activities
8
A
49
Y1
exam
data
B
D
J
F
M A
M J
J
A
S
O N
D
School Year 2014/15
2015/16 Term 1
Milestone 2: End of School Year 2014/15
Milestone 3: End
of Project
Mid-term
Project review and learning event
evaluation
Y3
enrolm
ent
data
Y2
exam
data
37
J
Final impact
assessment
Y4
enrolm
ent
data
Y3
exam
data
C
D
A: Baseline Report
B: Revised Logframe
C: Mid Line Evaluation Report
D: Final Evaluation Report
b. Management Information System
Data on all project activities will be collected using standardised tools, formats and procedures.
Where appropriate for the most data-intensive activities, this will be aided by using digital data
collection devices, otherwise data from paper-based tools will be entered into Excel in the first
instance at the district level. Excel will be used to collate and store all monitoring data, with
databases and coding established at the central level (with the exception of survey data – see
Section 1). All project data will share the same coding structures for geographic and other
identification information, aiding integration of different activity data-sets. Excel is the most
appropriate format given the common level of IT skills at the field level, and allowing data to be
exported into other formats or software for further analysis and reporting at the central level as
required.
Data sharing protocols are shaped by Plan’s Child Protection Policy and the project’s ethical and
CP standards. From the latter, the following are of particular relevance:
 Respect confidentiality and anonymity: Identity of individual participants must be
protected, records and reports should be kept confidential, and no responses or quotes
should be attributed to an individual without their prior written consent.
 Data protection: all data (including written notes and reports, digitised data, photographs
video and audio recordings) must be handled in such a way as to ensure the confidentiality
and anonymity of children is maintained at all times unless permission is given for its use.
Sensitive data must be securely stored (secure filing cabinet, password protected electronic
files etc.).
 Photographs and video: We will always ask permission from children (or, in the case of
young children, their parent or guardian) before taking images (e.g. photographs, videos) of
them. Respect their decision to say no to an image being taken. Ensure that any images
taken of children are respectful (For example: children should have adequate clothing that
covers up the sexual organs. Images of children in sexually suggestive poses or that in any
30.05.2013
DRAFT M&E Framework Plan International UK
way impact negatively on their dignity or privacy are not acceptable). Stories and images of
children should be based on the child’s best interest.
Monitoring data will be shared between project partners on a regular monthly and quarterly
basis based on agreed templates and reporting lines. The Central Coordination Unit will be the
primary repository of all monitoring data, responsible for collating information across activities
and districts. All partner organisations will be required to pro-actively share data with the CCU
along agreed reporting lines. The CCU will be the primary broker on information sharing and
knowledge management between partners and external stakeholders. No project specific
information or data relating to individual beneficiaries will be released to a third party without
the prior agreement of the CCU.
Key data sets will also be securely stored on Plan UK’s servers to provide data back-up. Plan UK’s
Open Information Policy (in draft) and IATI commitments will further inform the external sharing
of information outside of primary project stakeholders.
c. Data Verification Process
On the basis that prevention is better than cure, project partners have agreed that we will
ensure the quality of monitoring data collected by:
•
Recruit suitable data collectors with the right skills, and provide specific training on all
data collection tools
•
Use, wherever possible, sex and age appropriate data collectors, and take time to build
the trust from those we are collecting data from in line with child protection and ethical
research standards
•
Having a clear understanding of the data that needs to be collected, its use and value to
those from whom it is being collected, and appropriate means of data collection
•
Having a clear and common understanding of the definitions of all specific terms and
units of measurement used
•
Having clear lines of communication and data flows
•
Having user-friendly and clearly explained data collection tools
•
Wherever possible, and for all large scale data collection processes, pilot data collection
methods before full roll-out
During the course of the project, we will check and monitor data by:
•
Carry out peer reviews through joint monitoring visits (at least one per quarter) to
support cross-organisation learning and to double check on progress of reported
activities
•
Each organisation, on visiting a school where others are also active, will enquire as to
the progress of all GEC activities in that school and will send a report to the CCU for
cross-checking with existing records
30.05.2013
50
DRAFT M&E Framework Plan International UK
•
•
•
•
•
•
Organisations leading on specific technical activities of the project or training other
organisation staff will retain responsibility for ensuring the quality of the activities
delivered and the results reported on across all districts
Data collected using digital data collection devices will be automatically time and
location stamped
M&E Manger to carry out random spot checks on school to verify data once a quarter
Senior managers (from the CCU and implementing agencies) will regularly carry out field
visits to support operational and project quality
Compare primary data with secondary data, and with data collected by similar projects
in the same areas to look for un-explained differences
External financial auditors will also examine project data as per the payment by results
framework
In verifying data at the organisational and CCU level, we will aim to ensure:
•
Completeness: making sure there are no gaps in data sets; that the data collected is
relevant; and that data is available when needed
•
Correctness: all staff have the responsibility for ensuring the accuracy of data collected
and recorded through, for example, conducting back checks / self-review of own data
compared with previous records, ensuring specific numbers and appropriate
disaggregation etc. Data clerks will carry out an initial check and clean of data. Data
entered will subsequently be reviewed by relevant field staff. M&E Officers at the field
level will periodically check paper and electronic copies of data. The CCU M&E Manager
will periodically review data submitted and follow up as necessary.
•
Consistency: data clerks and M&E staff will look for anomalies or unlikely or strange
patterns in data sets and unusual outliers over time, between districts or data sets or in
comparison to secondary sources
Specific details relating to data verification of survey work (at baseline and subsequently) are
given in Section 1.
d. Stakeholder Engagement and Communication Plan
In all MER activities, the project will be informed by an utilisation-focused approach, with the
explicit aim of identifying the use of, and primary intended users, of all MER outputs to increase
the usability of outputs and increase buy-in from relevant stakeholders. Doing so forms part of
the mandate of the evaluation steering group (see below). For example, the baseline report will
be actively shared with project stakeholders, and a youth friendly summary of it to facilitate
access by different groups and with girls in particular. Results of the baseline will be used to
initiate discussions with duty-bearers at the local level and will feed into Plan and partners
national level advocacy and campaigning at both national and international levels. This strong
emphasis on shared learning will be maintained throughout the project, through participatory
monitoring and local level advocacy for example, as well as with midline and evaluation findings,
and this is built into the Performance Management Framework above. The project will seek out
30.05.2013
51
DRAFT M&E Framework Plan International UK
opportunities for involving and inviting feedback from a range of stakeholders, ranging from
Government actors, national and local women’s and child rights organizations and disabled
people’s organisations, and other relevant members of civil society.
e. Risk Assessment of M&E
The table below summarizes the key risks identified for the early stages of the project. The Chief
of Party will have responsibility for reviewing this list periodically alongside Plan’s project risk
register and updating as necessary.
Risk
Children put at risk by project
MER activities
M&E Framework and sampling
strategy not agreed in time
between
Plan,
baseline
consultant and EM
Plan does not have access to the
Sierra Leone adapted EGRA and
EGMA tests in time to
incorporate into the baseline
study.
M&E needs of the GEC
Programme and the Sierra Leone
GEC project at odds
Inability to identify suitable
control
groups
/
or
contamination of control groups
Impact
Potential or real harm to children
and/or
other
project
stakeholders;
damage
to
relationships between project
partners and individuals or
communities;
reduced
effectiveness
of
project
interventions;
damage
to
organizational reputation of
project; legal repercussions
against project staff or associates
Delay to start of baseline data
collection, further impacted by
start of rainy season and reduced
access to field sites, increased
cost of baseline process.
Delays to the baseline and
project start
Necessary to reduce focus /
length of additional survey
questions to avoid research
fatigue and ethical burden,
reducing ability to measure
context specific barriers to
education / impact of project
specific
activities
and
intermediate
outcomes
/
unintended consequences of
project
No counter-factual; reduced
ability to demonstrate impact
Mitigation
Plan
International’s
Child
Protection Policy signed by all
project partners and associates
Ethical and CP standards agreed
by all project partners and
followed for all MER activities,
with
clear
reporting
and
oversight
All partners and associates
briefed on the CPP and Standards
Request that the EM draw
sampling points in stages by
district
Use ASER type tests for the
baseline (and endline), and
consider subsequent use of EGRA
and EGMA tests with specific
groups of beneficiaries (i.e.
members of girls study groups).
Keep project specific questions to
minimum focusing on logframe
indicators;
compliment
household questionnaire with
qualitative data collection and
project monitoring data
EM to coordinate between Plan,
BRAC and DFID Sierra Leone on
identification of possible control
30.05.2013
52
DRAFT M&E Framework Plan International UK
Research fatigue and /or attrition
of respondents for longitudinal
survey / Hawthorne effect in
sampled clusters
Higher attrition, Reduced ability
to track a cohort; Reduced
validity of findings due to
positive bias within sampled
communities
Incomplete / inaccurate or noncomparable monitoring data
collected
Gaps in knowledge of process;
inability to report against
activities and outputs
Monitoring data
underutilization
/
Research
fatigue
among
respondents; inefficient use of
resources and time; lack of focus
on learning to improve data
collection
Overlap between Plan target and
control areas and Brac and DFID
country programme (ISIS project)
target and control sites
Conducting learning assessments
with girls receiving bursaries
leads to a biased sample
Contamination of control sites
reduces counterfactual and
ability to prove attribution to a
single project
Learning assessments at the
school
level
demonstrate
changes only within a specific
target group of the population
not the totality of all project
activities
f.
overload
groups; survey and qualitative
work to measure exposure to
similar activities
Ensure focus groups do not overrun; keep survey length to
minimum
possible,
collect
respondent ID data to allow recontact, factor in additional
time/resources at endline to recontact;
Comparison
and
triangulation of data sources
Agree
and
monitor
data
collection strategy between all
partners; ensure data verification
process
followed;
ensure
sufficient resources for data
verification
Collect data specifically related
to measuring quality and
effectiveness of the project (and
control);
Establish
project
learning strategy and build in
time for learning and reflection
by project field staff and
managers
Selection of all schools to be
compiled and shared by end of
May, coordinated by DFID Sierra
Leone
Learning assessments at the
school level are complimented by
learning assessments as part of
the household survey, which will
provide a (relatively small)
sample of target girls not
receiving bursaries but involved
in. In addition, the exposure of
individual
girls
receiving
bursaries to other elements of
the project will be measured,
allowing some degree of internal
differentiation.
Budgetary Consideration
Please find attached a revised Costed M&E Plan as Attachment 4
30.05.2013
53
DRAFT M&E Framework Plan International UK
Appendices
Appendix 1:
‘Ideal Case’ Sampling Strategy
The assumptions provided reflect how the impact evaluation should be designed ideally with no
consideration of operational constraints and imperatives are as follows:
1) The ideal case pursues a two-stage sampling strategy. The following overview presents
the different treatments and their main primary unit (i.e. community or school):
Community
CBR
Bursary
School
Student clubs
Parent clubs
Girls mentoring
school amendment
Teacher training
Teacher learning assistance
Scorecards
2) In the ideal case, we would just want to channel treatment to the beneficiaries either
through schools or villages. In the case that is ideal from a purely academic point of
view, we would choose schools as the primary unit.
3) In the first stage, we draw a random sample of schools that will participate in the
experiment
4) During the second stage, we then list all girls that meet the criteria as specified into a
sample frame. Out of this sample frame we then randomly select 3*10 girls that meet
the above mentioned criteria. Within each sub group we further stratify by disability
status (1/0) to be able to detect impact on disabled vs. non-disabled girls.
30.05.2013
54
DRAFT M&E Framework Plan International UK
5) The expected effect size is based on the main outcome variable ‘learning performance’
that is captured through the learning assessment
6) The main vehicle for data collection is a household level survey. Besides collecting basic
socio-economic data, it also administers the learning assessment
7) Ideally, the baseline will be administered before the intervention at the end of the
previous school year
8) Upon completion of the baseline, the intervention will be delivered at the beginning of
school year 1. For that purpose, we randomly assign primary sampling units to both
experimental groups
9) The intervention is followed by the first follow up survey at the end of school year 1
10) A second follow up survey will be administered at the end of school year 2
11) A second follow up survey will be administered at the end of school year 3
12) The final round of data collection is the endline survey that will be administered at the
end of the last school year of the programme. Ideally, this should be once those girls
that were in PS5 have completed JSS3.
13) This panel data approach allows tracking of performance over time.
For this design, the sample size can be calculated using the following formula and values
MDE = (t(1-k)+ta) * sqrt(C/(P(1-P)J))*sqrt(1-C/(P(1-P)NJ))
MDE = Minimum effect size, expressed in standard deviation units of the outcome ( we assume
0.2 SD units)
K = power of 80%
A = significance level of 5%
(t(1-k)+ta) = 2.8
C = Intra cluster correlation = 0.2 assuming a design effect of deff = 2.8
P = Portion of girls allocated to treatment (P = 50%)
J = Total number of clusters (J = 220)
N = Data points within each cluster (N = 10)
The values above reflect the assumption that the 3 cohorts do not differ in terms of effect size
or intra-cluster correlation. Furthermore, the sub samples are weighed equally.
30.05.2013
55
DRAFT M&E Framework Plan International UK
Based on these assumptions, we determine J = 220 with 110 treatment schools and 110 control
schools for both PS and JSS. Hence, the total number of schools would be 440. At the household
level, we incorporate a response rate of 90%. Hence, we assume around 10% of the sample will
refuse the interviews or will be subject to attrition over the course of the programme. This
increases the intra-cluster sample size from 10 girls in each group to 12. Hence, the sample size
is as follows:
- 12 girls in PS5
- 12 girls in PS6
- 12 girls in JSS1, and
The total sample size required is thus 5,280.
Across these sub samples, we will further stratify girls by disability status. The composition will
be then as follows:
Nondisabled
Disabled
Total
P5
8
4
12
P6
8
4
12
J2
12
0
12
This will allow us to detect impact of 25% on disabled and non-disabled separately.
30.05.2013
56