Nuru Kenya Education Impact Assessment Results of the 2013 Education Surveys Authored by: Nuru International’s and Nuru Kenya’s Monitoring and Evaluation teams ABSTRACT In 2013, Nuru Kenya’s Monitoring and Evaluation team surveyed a subset of Nuru Education Program participants (public school children) in Kuria West, Kenya. As part of a larger impact assessment strategy for Nuru Kenya’s Education Program, the main purpose of the survey was to assess the change in English literacy in the schools where the Education Program conducts outreach. Results show that English literacy, among the intervention schools surveyed in Isebania, a division of Kuria West where Nuru works, increased 8.6 percentage points, from 19.4 percent to 28 percent, from beginning of year 2012 to beginning of year 2013. A similar percentage point increase (10.1 percent) was seen in children who can at least read at the paragraph level, from 49.6 percent to 59.7 percent. These results will be compared to the official Uwezo data for Kuria West later in 2014. Based on past performance of Kuria West public school children in Grades 2-‐7, Uwezo results indicate that this increase will be close to 0 percent and will remain near their 2012 rates of 31.7 percent for children at English literacy and 64.0 percent for children who can at least read at the paragraph level. These preliminary results indicate that children in Nuru outreach schools may demonstrate a greater increase in literacy relative to all comparable public school children in Kuria West District. 2 Table of Contents Abbreviations ...................................................................................................................................................................... 3 Overview ............................................................................................................................................................................... 4 Nuru Kenya Education Program ......................................................................................................................................... 4 About Uwezo Literacy Assessment .................................................................................................................................... 4 Methodology ........................................................................................................................................................................ 5 Sampling Frame .......................................................................................................................................................................... 6 Data Collection ............................................................................................................................................................................ 7 Analysis .......................................................................................................................................................................................... 8 Results ................................................................................................................................................................................... 8 Tier 1 Results ............................................................................................................................................................................... 8 Tier 2 Results ............................................................................................................................................................................... 9 Tier 3 Results ............................................................................................................................................................................ 10 Limitations ........................................................................................................................................................................ 13 Conclusions and Recommendations ......................................................................................................................... 13 Appendix 1: Example English Uwezo Survey ......................................................................................................... 15 1/28/14 This information is confidential and is not intended for public distribution. Page 2 3 Abbreviations BOY Beginning of year EOY End of year M&E Monitoring & Evaluation, often referring to Nuru’s Monitoring and Evaluation Program or team NI Nuru International NK Nuru Kenya 1/28/14 This information is confidential and is not intended for public distribution. Page 3 4 Overview Nuru’s Monitoring and Evaluation (M&E) approach is built to produce useful and relevant information that can contribute to key decision-‐making about Nuru’s programs (i.e., whether to continue, iterate, replicate and/or scale an intervention). With this focus on utilization at the center of Nuru’s M&E strategy, the M&E team works to objectively monitor and evaluate the performance and impact of Nuru’s Impact Programs — Agriculture, Community Economic Development, Healthcare and Education. In service to this M&E approach, the Nuru Kenya M&E team administered an end of year survey in November 2013 which built on a beginning of year survey in 2013, complemented a similar data collection in 2012 and answered two questions: 1. What is the impact of Nuru Kenya’s Education’s Program on English Literacy? 2. What is the impact of Nuru Kenya’s Education Program on children’s ability to read at least at the paragraph level? This paper presents the results of data collected throughout 2012-‐2013 to address these questions and will contribute to Nuru’s understanding of children’s progression toward English literacy. Nuru Kenya Education Program The Nuru Kenya (NK) Education Program aims to increase child literacy to the Standard 2 (Grade 2) level among rural public primary school children through student-‐centered teaching and literacy-‐focused interventions. Outreach focuses on the five main components of literacy development: phonemic awareness, phonics, fluency, vocabulary and comprehension. During the 2013 school year,1 the Nuru Education Program conducted classroom outreach in nine primary schools in Isebania Division and three primary schools in Mabera Division of Kuria West, Kenya. Since 2009, NK has been working with farmers in Kuria West, which is among the poorest, and most remote, rural areas in Kenya.2 School selection is based on the Ministry of Education’s recommendation on which schools have the most need. Outreach occurred in Pre-‐Unit (Kindergarten) through Grade 7. The nine primary schools in Isebania received similar outreach during the 2012 school year for two hours per month per class. Workshops two hours per month continued in term one of 2013 (January-‐March), then increased to four hours per month per class in the second term of 2013 (April-‐onward). The three primary schools in Mabera received outreach starting in January 2013 for four hours per month per class. Outreach was cancelled when schools closed for two weeks due to the presidential election and for four weeks in July due to teacher strikes. Based on monitoring data collected at every outreach session, the average child to facilitator ratio is 8 to 1, with slightly more than 4,100 children participating in outreach. About Uwezo Literacy Assessment The Uwezo literacy assessment is a tool developed by a regional (Kenya, Tanzania and Uganda) initiative called Uwezo, aimed to improve the competencies in numeracy and literacy among primary school-‐aged children Grades 2 through 7.3 As the NK Education Program’s goal is focused on increasing literacy for 1 The standard school year in Kenya consists of three terms. Approximate dates for each term are: January-‐March; April-‐July; August-‐November. 2 The World Bank cites Kuria West district as being >70% below the Kenyan rural poverty line ($1.25/day). 3 For more information on Uwezo, please see www.uwezo.net. 1/28/14 This information is confidential and is not intended for public distribution. Page 4 5 primary school-‐aged children in rural communities, Nuru implements the Uwezo assessment to assess literacy with the English and Kiswahili literacy4 portions of the assessment. The Uwezo tool can assess literacy simply and frequently; therefore, Nuru adapts the tool to be used at the school level for the purpose of both monitoring and evaluating Nuru’s Outreach Program. The Uwezo assessment measures a child’s English and Kiswahili literacy at the letter, word, paragraph and story levels. Testing starts at the paragraph level to save on time. For example, when testing, if the child struggles to read the paragraph, he/she is directed to the word level and instructed to read any five words from a word list. If he/she struggles at the word level, the child will be directed to read letters. If the child is successful, he/she will advance back up each level. If the child reads at the paragraph level successfully, he/she will be directed to the story level and then asked two questions to assess comprehension (Appendix 1). In addition to the literacy level of the child, information is recorded on each child’s grade, name, age and gender. Nuru conducts the Uwezo assessment at the beginning and end of each school year. Nuru also uses data officially collected and analyzed by Uwezo, the organization based in Nairobi, to compare results. Uwezo administers the Uwezo literacy assessment to children in Grades 1-‐8 across Kenya. According to the 2012 Uwezo report: “In 2012, 156 of the 158 (census) districts were covered, assessing a total of 153,900 children from 90,820 households.”5 Uwezo engages in a stratified random sampling process in which they survey children at 50 to 100 households in 30 enumeration areas (each about the size of a village) within each district in Kenya. Their large sample combined with strategic sampling process gives a good representation of how children are faring within Kuria West District, Migori County, Nyanza Province, and Kenya as a whole. Methodology Nuru implements a multi-‐tiered approach to assess impact among the following key evaluation indicators (Table 1). Table 1: Education Evaluation Indicators Education Indicator Evaluation Question % of children achieving English literacy What is the impact of Nuru Kenya’s Education’s Program on English Literacy? % of children who can at least read at the What is the impact of Nuru Kenya’s Education paragraph level Program on children’s ability to read at least at the paragraph level? Children at the intervention schools receive both the regular school curriculum and the Nuru Outreach Program throughout the year. Therefore, the impact of the Nuru Education Program cannot be studied with a pre-‐post study design alone (meaning strictly an assessment prior to the intervention and after the intervention); rather, a more rigorous study design with a comparison group would be required to determine the isolated impact of the Education Program from the regular school curriculum. Given limited 4 Kenya’s official languages are English and Kiswahili. 5 “Are Our Children Learning?: Annual Learning Assessment Report” Uwezo Kenya. 2012. 1/28/14 This information is confidential and is not intended for public distribution. Page 5 6 resources, the following evaluation methodology attempts to demonstrate the literacy impact of the regular school curriculum from the impact of the Nuru Education Program. First, Nuru implements evaluation methodologies with the following two approaches: • Tier 1: Longitudinally, over time, compare children in a subset of Nuru Kenya schools (aggregated literacy scores across four schools from Isebania Division) to official Uwezo data. This data is collected on a district level to determine if the increase in literacy trends over several years among Nuru schools is greater than the literacy trends reported at the Kuria West District. • Tier 2: Compare the change in literacy across one Nuru Kenya school – Iraha school from Mabera Division – the year before (2013) and after (2014) the Nuru Education intervention. This approach is aimed to determine if the annual change in literacy from beginning to end of year, after the Nuru Education intervention, is greater than without the Nuru intervention. This comparison also matches children from beginning to end of year to ensure the study population remains the same from year to year. Nuru also implements a focused literacy monitoring approach, whereby NK schools receiving the Education Program intervention (four schools from Isebania Division, three schools from Mabera Division) are measured at the beginning and end of year to compare literacy progression differences (Tier 3) across schools within the same school year. With this analysis, NK can track individual children, grades and schools for increases at the letter, word, paragraph, story and comprehension levels of literacy to ensure the program is effectively rolled-‐out. This multi-‐tiered methodology was developed over the course of a few years to determine the best approach to measurement for Nuru Kenya, Nuru International and other key stakeholders. The first two tiers are used primarily for external purposes while also helping NK to know its impact; while the third tier assists the NK Education Program in planning and preparation for programming within the various schools. Sampling Frame Regardless of the M&E strategy presented above, the NK M&E team surveys a census (100 percent) of children at each surveyed school in Grades 2-‐7 with the Uwezo literacy assessment. This approach ensures NK is able to decrease its margin of error in order to provide more precise estimates of data at each Grade level in addition to the aggregate school level. In 2012, all nine schools in Isebania Division were surveyed for a pre-‐post (beginning to end of year) analysis. In 2013, a purposeful selection strategy was used to identify four intervention schools to be surveyed in Isebania Division. Six schools in Mabera Division were also surveyed in 2013, which included three new intervention schools and three non-‐intervention schools further explained in the next section.6 See Figure 1 for information on the location of Isebania and Mabera within Kuria West. 6 The three non-‐intervention schools that were surveyed would become intervention schools in 2014. One of these schools, Iraha, was selected to be the point of comparison for the Tier 2 evaluation study specifically because it was the lowest performing school in 2013. 1/28/14 This information is confidential and is not intended for public distribution. Page 6 7 Figure 1 Data Collection Nuru M&E enumerators collected data at each school by coordinating the best dates and times with the Kenya Ministry of Education. Below is the timeline carried out by the NK M&E team to implement the survey: • • • Enumerator training: October 3, 2013 Data collection: October 7 – October 18, 2013 Data entry: October 8 – October 21, 2013 The survey collected each child’s name, age and gender as well as the literacy score for letter, word, paragraph, story, and comprehension levels.7 Thirteen enumerators, including two M&E Field Officers, were trained to conduct the Uwezo assessment, supervised by two NK M&E Field Managers. Beginning of year literacy data were collected in January and February 2013; end of year literacy data were collected in mid-‐October 2013. All children present in Grades 2 through 7 were assessed individually at each surveyed school. Even though outreach is conducted in kindergarten through to Grade 7, assessment begins at the Grade 2 level, consistent with Uwezo’s assessment methodology. Given Nuru’s M&E strategy, it was determined that it would be important to survey children in a purposeful selection of intervention and non-‐intervention schools. Four intervention schools in Isebania and one non-‐ intervention school in Mabera were surveyed to contribute to the Tier 1 and 2 strategies, respectively. For the Tier 3 strategy, seven schools were surveyed in Isebania (which included three schools also from Tier 7 Literacy refers to Standard 2 literacy, or the level at which a child should be able to read in the second grade. 1/28/14 This information is confidential and is not intended for public distribution. Page 7 8 1) and Mabera divisions. A combined total of 1,810 children were assessed at the beginning of year and 1,826 children at the end of the year. Some changes in the individual children assessed occurred between beginning and end of year due to absenteeism, children changing schools and children not returning to school. When comparing data over multiple years, also note that Grade 7 children matriculate and a new group of Grade 2 children are assessed. Analysis Data were analyzed for each of the three tiers described previously in this paper. Primarily, differences in literacy scores across the children in the four Isebania schools are analyzed from year to year, and from beginning of year to end of year. A difference-‐in-‐difference analysis is also used for the evaluation of impact (impact assessment) tiers of the methodology. For example, the difference in literacy change is assessed from beginning of year to end of year across two years. Furthermore, data are analyzed by school, grade and gender. Finally, the various levels of progression toward achieving literacy (i.e. letter, word, paragraph and story levels) are assessed for the Education Program to better understand their target population and customize outreach programming. Results Tier 1 Results When comparing a subset of NK schools (four total from Isebania Division) to the official Uwezo data collected at the district level, schools with Nuru outreach experience an increase in English literacy from 19.4 percent at beginning of year 2012 to 28.0 percent at beginning of year 2013. For 2012, official Uwezo data for public school children in Kuria West are at 37.7 percent with limited changed projected for 2013. Children who can at least read at the paragraph shifts from 49.6 percent to 59.7 percent from the beginning of year 2012 to 2013 in surveyed Isebania intervention schools. In 2012, official Uwezo data for public school children in Kuria West are at 64.0 percent and is expected to remain the same for 2013. It is assumed that if trends remain the same for the official Uwezo data, to be officially released in 2014, schools with Nuru outreach in the Tier 1 evaluation will demonstrate a greater increase in literacy relative to all of Kuria West District public school children in Grades 2-‐7 (Table 2). 8 8 According to Uwezo staff, the 2013 official Uwezo data will be available in April-‐May 2014 1/28/14 This information is confidential and is not intended for public distribution. Page 8 9 Table 2: Results of Evaluation of Indicators for Nuru Outreach Children in Isebania Division Compared to All Kuria West Public School Children, 2012-‐20133 Impact Indicators % of children achieving English literacy Difference % of children who can at least read at the paragraph level Difference Nuru Isebania Outreach Children (Beginning of Year measurement only) All KW Public School Children (Grades 2-‐7) 2012 2013 2012 20139 19.4% 28.0% 31.7% 31.7% 8.6% 49.6% 0.0% 59.7% 64.0% 10.1% 64.0% 0.0% Tier 2 Results Results for Tier 2 will be available at the end of 2014 for the comparison study of Iraha school in Mabera Division that compares literacy changes across one full school year before the Nuru Education intervention (2013) with the change in literacy in a school year after the intervention (2014). Figure 3 presents the difference between pre-‐intervention beginning of year to end of year scores. Data collected in 2013 (pre-‐ intervention) indicate that Standard 2 literacy among children in Grades 2 to 7 increased by 3.5 percent. A midpoint assessment of Iraha school to determine post-‐intervention scores will be carried out in 2014 in addition to an end of year assessment. 9 2013 data for all Kuria West Public School Children (Grades 2-‐7) are assumed by NI M&E staff to be equivalent to 2012 due to consistency in the regular school curriculum from one year to the next. 1/28/14 This information is confidential and is not intended for public distribution. Page 9 10 Figure 2 Tier 3 Results Tier 3 analyzes the beginning to end of year literacy changes among seven schools with Nuru outreach surveyed for 2013 (four in Isebania and three in Mabera Division). Due to variations among schools—such as irregular attendance, teacher credentials, population density surrounding each school, a drought that hit the region in 2013,10 among other factors, assessment of variations in literacy are generally complex. It is important to capture multiple dimensions of literacy progression and therefore, while, Tiers 1 and 2 will serve as the analysis needed to better understand Nuru’s impact in literacy at the school level, Tier 3 serves as monitoring for real-‐time program design and improvement. Overall, the result of Nuru’s Tier 3 analysis shows the average increase in literacy from beginning to end of year is 4.8 percentage points among the seven schools surveyed in Isebania and Mabera divisions, ranging from -‐5.8 percent in Sirori Simba to 10.4 percent in Keborui school (Figure 3). Further investigation by the NK Kenya Education team is being conducted to assess the change in Sirori Simba. 10 Famine Early Warning System Network (FEWS). East Africa Seasonal Monitor. 19 July 2013. Retrieved from http://www.fews.net/docs/Publications/East_Seasonal_Monitor_07_2013.pdf. 1/28/14 This information is confidential and is not intended for public distribution. Page 10 11 Figure 3 English(Literacy(Changes(for(Nuru(Schools(from( Beginning(to(End(of(Year(2013( 60.0%( 50.0%( 40.0%( 30.0%( 20.0%( 10.0%( 0.0%( Kebouri( Sirori( Simba( Beginning(of(Year(English(Literacy( 37.7%( 35.8%( 12.3%( 21.1%( 28.8%( 15.0%( 20.3%( End(of(Year(English(Literacy( 48.1%( 30.0%( 19.8%( 28.9%( 33.6%( 17.4%( 26.7%( Nyasese( Sorore( Burimago Gosese( ngo( Isebania( Moheto( Mabera( For the next key evaluation indicator, 70.0 percent of children can at least read at the paragraph level at the end of 2013, compared to 58.0 percent at the beginning of the year among the seven surveyed intervention schools. For Grades 2, 3 and 4, we see percentage point changes of 16.0, 21.5 and 14.5 percent, respectively. Interestingly, after fourth grade, this percent change difference begins to taper off, indicating that most children make large literacy advances from Grades 2 through 5 (Figure 4). Figure 4 Figures 5 and 6 (below) present the results of reading progression at the school level. The results range from not being able to read at all to reading at the letter, word, paragraph, story levels and then at full 1/28/14 This information is confidential and is not intended for public distribution. Page 11 12 literacy levels. The data in Figure 5 indicate that at the beginning of the year, Gosese has the greatest percentage of children with no literacy comprehension (33.6 percent) while Keborui has the great percentage of students with Standard 2 literacy (37.7 percent) compared to other schools. Figure 5 Highest(Literacy(Achieved(Beginning(of(Year(2013( 100%% 70%% 60%% 50%% 40%% 15.9%% 12.4%% 37.7%% 15.9%% 17.9%% 13.6%% 14.5%% 16.3%% 17.9%% 16.7%% 17.6%% 30%% 33.6%% 9.9%% Burimagongo% 5.6%% 15.0%% 21.1%% 35.8%% 14.1%% 21.5%% 12.8%% 11.9%% 14.7%% 11.0%% 19.8%% Nyasese% 5.9%% 0%% 18.5%% Moheto% 10.5%% Keborui% 10%% 15.7%% 13.7%% Gosese% 20%% 12.3%% 16.7%% 7.5%% 12.1%% 28.8%% 20.3%% 13.8%% Std%2%English%Literate% 14.8%% 19.6%% 14.4%% 8.6%% Story% Paragraph% Word% LeKer% None% Sorore% 80%% 15.0%% 28.8%% Sirori%Simba% 90%% Beginning%of%Year% In comparison to Figure 5, Figure 6 shows that from beginning to end of year in 2013, the number of children who can read nothing, only letters, or only words decreases from 42.0 percent to 31.6 percent, signaling a shift from lack of phonemic awareness to fluency and comprehension. For children able to read at the paragraph level and above, Moheto increases by 16.0 percent from beginning to end of year, while Sirori Simba demonstrated the smallest increase of 2.2 percent among the seven surveyed schools. Figure 6 Highest(Literacy(Achieved(End(of(Year(2013( 100%% 17.4%% 90%% 11.6%% 70%% 15.7%% 34.3%% 11.7%% 20.4%% 13.8%% 13.6%% 23.4%% 9.7%% 0.7%% 6.0%% 18.6%% 6.3%% 7.2%% Burimagongo% Gosese% Keborui% 12.9%% 14.2%% 5.8%% 16.4%% 18.2%% 18.1%% 17.9%% 9.1%% 10.2%% 10.2%% 10.0%% Sirori%Simba% 16.3%% 20%% 0%% 17.4%% 7.5%% 5.0%% Std%2%English%Literate% Story% Paragraph% Word% LeKer% None% Sorore% 14.0%% 40%% 10%% 19.1%% 15.3%% 50%% 30%% 28.9%% 20.3%% 22.7%% 22.1%% 30.0%% Nyasese% 60%% 48.1%% 19.8%% Moheto% 80%% 26.7%% 33.6%% End%of%Year% 1/28/14 This information is confidential and is not intended for public distribution. Page 12 13 Additional analysis shows that for Grade 2, specifically, the number of children who can read at the paragraph level or higher triples from 7.2 percent at the beginning of the year to 23.2 percent at the end of the year. In Grade 5, 78.3 percent of children can read at the paragraph level or higher at the beginning of the year; and by the end of the year, that number increases to 86.9 percent. Furthermore, English literacy for children in Grade 5 increases 15 percent from 28.5 percent to 43.8 percent, indicating a positive progression from the beginning to end of year 2013. This monitoring data show general trends toward literacy achievement to be able to help the Education Program iterate; these results are not indicative of the impact Nuru is making compared to children who are not receiving the Nuru Education intervention. Limitations The 2013 official Uwezo data for Kuria West will not be available for comparison until mid-‐2014; therefore, the analysis included in this report is limited to making informed projections about impact and non-‐ attributable review of literacy changes from beginning to end of year 2013 for children’s literacy levels. Once the official Uwezo data are available, approximately in May 2014, Nuru will more concretely understand the impact of the NK Education Program efforts. Due to children receiving both the regular school curriculum and the Nuru Education Outreach Program during the same time period, impact cannot be determined with a pre-‐post study design without a comparison (Tier 3). Additionally, there are variations among schools, including varying populations of children, teacher credentials and population density surrounding schools, which may impact literacy. NK M&E is building collection of these variables into 2014 and future data collection and analyses. Having stated these limitations, Tiers 1 and 2 will serve as the analysis needed to better understand Nuru’s impact in literacy at the school level while Tier 3 is strictly a monitoring tool. Throughout this report, the data reported include all children from the selected intervention and comparison schools that were in attendance on the day of assessment. As school populations may shift throughout the year, primarily due to absenteeism, the children assessed in a particular grade may be drastically different by the end of the year. Furthermore, the children assessed change from year to year due to matriculation of children in Grade 7 and addition of new children in Grade 2. Prior to 2013 data collection, the children were not matched from beginning to end of year thus, this could have caused further variations in the data. In 2014 and in future data collection efforts, NK M&E will ensure that children are able to be matched from beginning to end of year to prevent any error due to this variation from occurring. Conclusions and Recommendations Results of the Tier 1 longitudinal study show that from beginning of the year in 2012 to the beginning of the year in 2013, surveyed schools with Nuru outreach in Isebania experienced an 8.6 percentage point increase in English literacy. A similar percentage point increase (10.1 percent) was seen in children who can at least read at the paragraph level. These figures will be compared to the official Uwezo data for Kuria West later in 2014. Projected official Uwezo data show minimal change (close to 0 percent) among Nuru’s two key indicators from 2012 to 2013 across all Kuria West public school children in Grades 2-‐7. This indicates that children in Nuru outreach schools may demonstrate a greater increase in literacy relative to all comparable public school children in Kuria West District. 1/28/14 This information is confidential and is not intended for public distribution. Page 13 14 Results for Tier 2 will be available at the end of 2014 for the comparison study of Iraha school in Mabera division, to more directly compare literacy changes before the intervention (2013) to after the intervention (2014). For 2013, data show there was a shift in literacy of 3.5 percentage points from 16.2 percent to 19.7 percent in Iraha school. This change in literacy, before the intervention, will be compared to the beginning and end of year literacy scores in 2014 after a year of the intervention. Results of the Tier 3 monitoring efforts show a general increase in literacy by 8.0 percentage points from beginning to end of year among the seven intervention schools in Isebania and Mabera divisions. For the percent of children who can at least read at the paragraph level, this also increased throughout the year from 58.0 percent at the beginning of the year to 70.0 percent at the end of the year. Many of the increases outlined in Tier 3 of the analysis show that most literacy increases occur from Grades 2 through 5. Therefore, focusing on this level for outreach, monitoring and evaluation efforts could be more cost-‐ effective and impactful in the future. Other recommendations for Nuru’s M&E and Education teams to consider in future planning in Kenya and other Nuru projects are as follows: 1. Better link the components of the Education curriculum to the stages of literacy for improved measurement and impact over time. 2. Link names from the educational outreach attendance list to the beginning and end of year evaluations from the NK M&E team to ensure students can be matched and literacy progression can be better tracked. 3. Continue monitoring a subset of schools to determine beginning to end of year changes in literacy as Nuru continues to scale, ensuring a balance between the benefits of additional data collection and costs of collecting that data. 1/28/14 This information is confidential and is not intended for public distribution. Page 14 15 Appendix 1: Example English Uwezo Survey 1/28/14 This information is confidential and is not intended for public distribution. Page 15 16 1/28/14 This information is confidential and is not intended for public distribution. Page 16
© Copyright 2026 Paperzz