Improving Quality
Education and
Children’s Learning
Outcomes
and Effective Practices in the Eastern and Southern Africa Region
Report for UNICEF ESARO
a
Tim Friedman, Ursula Schwantner, Jeaniene Spink, Naoko Tabata and Charlotte Waters
Australian Council for Educational Research (ACER)
2016
Commissioned by UNICEF Eastern and Southern Africa Regional Office (ESARO), Basic Education and
Gender Equality (BEGE) Section.
©2016 United Nations Children’s Fund (UNICEF)
Cover photo: Francõise d’Elbee ©UNICEF Kenya/2009/d’Elbee
Permission is required to reproduce any part of this publication. Permission will be freely granted to
educational or non-profit organizations. Others may be requested to pay a small fee. Requests should be
addressed to: Basic Education and Gender Equality Section, Eastern and Southern Africa Regional Office,
UNICEF; Tel: +254 207-622-307 email: [email protected]. b
Contents
Lists of tables and figures...............................................................................................ii
Acronyms and abbreviations...........................................................................................iv
Acknowledgements...................................................................................................... vii
Foreword from the Eastern and Southern Africa Regional Director..................................... viii
Executive summary....................................................................................................... x
Introduction.................................................................................................................. 1
Context for primary education in the ESA region........................................................ 1
Conceptual framework........................................................................................... 2
1. Stock-taking and comparative analysis of existing assessments in the ESA region............ 5
1.1Overview of assessments in the stock-taking...................................................... 5
1.2Comparative analysis of assessments................................................................ 11
2. Literacy and numeracy in primary education in the ESA region:
Students experiencing limited learning outcomes and trends over time.......................... 27
2.1.Characteristics of students experiencing limited learning outcomes
in literacy and numeracy in primary education in the ESA region.......................... 28
2.2.Trends in learning outcomes of children in primary education
in literacy and numeracy in the ESA region....................................................... 38
3. Improving learning outcomes in the ESA region: Effective country-level practices........... 45
3.1Country-level programmes analysed................................................................. 46
3.2Key strategies for success.............................................................................. 47
4. A macro theory of change....................................................................................... 51
References................................................................................................................. 57
Appendix I: Methodology............................................................................................. 67
Appendix II: Detailed tables for Chapter 1...................................................................... 81
Appendix III: Country case studies................................................................................. 90
Appendix IV: Detailed tables and figures for Chapter 2................................................... 103
Appendix V: Detailed table for Chapter 3..................................................................... 115
Appendix VI: Main stock-taking table........................................................................... 117
i
Lists of tables and figures
Tables
Table 1: Participation in international assessments in ESAR countries................................. 6
Table 2: Participation in regional assessments in ESAR countries........................................ 7
Table 3: Implementation of national assessments by government/parastatal bodies
in ESAR countries............................................................................................ 8
Table 4: Implementation of EGRA/EGMA in ESAR countries..............................................10
Table 5: Purpose of the assessments from stock-taking................................................... 11
Table 6: Target population in the assessments from stock-taking...................................... 12
Table 7: Cognitive domains in assessments from the stock-taking.................................... 15
Table 8: Test administration methods of assessments from the stock-taking...................... 18
Table 9: National assessments and use of IRT in data analysis......................................... 20
Table 10: Descriptions and attainment of competency levels in literacy:
NASMLA in Kenya 2010................................................................................. 21
Table 11: Proportions of students defined as experiencing limited learning outcomes
by assessment, country, grade and domain....................................................... 30
Table 12: Direction of trends in SACMEQ reading and mathematics scale scores
from 2000 to 2007, based on (Hungi et al., 2011)............................................. 43
Table 13: Stock-taking framework.................................................................................. 68
Table 14: Assessment programmes for which data were available for analysis..................... 71
Table 15: Uwezo final sample for analysis in this report.................................................... 74
Table 16: Assessment implementation by type of assessments.......................................... 81
Table 17: ESAR countries with limited assessment activity in recent years.......................... 82
Table 18: Analytical techniques used in the assessments from the stock-taking
of assessments in ESAR................................................................................. 83
Table 19: Proportion of students experiencing limited learning outcomes
for each contextual variable of interest for Uwezo countries...............................104
Table 20: Proportion of students experiencing limited learning outcomes
for each contextual variable of interest for PASEC countries...............................107
Table 21: Proportion of students experiencing limited learning outcomes
for each contextual variable of interest for TIMSS.............................................109
Table 22: Proportion of students experiencing limited learning outcomes
for each contextual variable of interest for prePIRLS.......................................... 110
Table 23: Example programmes in ESAR with focus on improving learning outcomes
in literacy and numeracy of disadvantaged children in primary education.............. 115
Table 24: Main stock-taking table................................................................................. 117
ii
Figures
Figure 1: A macro theory of change. An evidence-based monitoring and
intervention cycle as premise for change: assessment, analysis, action................. xiv
Figure 2: Assessments from the stock-taking by type of assessment................................... 6
Figure 3: Mean percentage marks in Grade 3 mathematics by gender and province
– Annual National Assessment in Kenya........................................................... 22
Figure 4: Trends in English performance across time for students in Kenya (Uwezo)............. 39
Figure 5: Trends in mathematics performance across time for students in Tanzania (Uwezo).. 40
Figure 6: Trends in proportions of students experiencing limited learning outcomes
across Uwezo countries.................................................................................. 41
Figure 7: A macro theory of change. An evidence-based monitoring and
intervention cycle as premise for change: assessment, analysis, action................. 52
Figure 8: Trends in Swahili performance across time for students in Kenya (Uwezo)............ 111
Figure 9: Trends in mathematics performance across time for students in Kenya (Uwezo).... 112
Figure 10: Trends in English performance across time for students in Tanzania (Uwezo)........ 112
Figure 11: Trends in Swahili performance across time for students in Tanzania (Uwezo)........ 113
Figure 12: Trends in English performance across time for students in Uganda (Uwezo)......... 113
Figure 13: Trends in mathematics performance across time for students in Uganda (Uwezo).114
iii
Acronyms and abbreviations
3ie
International Initiative for Impact Evaluation
ACER
Australian Council for Educational Research
ANA
Annual National Assessment
ASALs
Arid and semi-arid lands
ASLI
Africa Student Learning Index
BEGE
Basic Education and Gender Equality
CBA
Competency-based approach
CO
Country Office
CONFEMEN
Conference of Ministers of Education of French-speaking Countries
DAC
Development Assistance Committee
DfID
Department for International Development (UK)
ECD
Early childhood development
EDF
Education Development Fund
EDI
Education for All Development Index
EFA
Education for All
EFA GMR
Education for All Global Monitoring Report
EGMA
Early Grade Mathematics Assessment
EGRA
Early Grade Reading Assessment
ELMI
Early Literacy and Maths Initiative
EMIS
Education Management Information System
ESA
Eastern and Southern Africa
ESAR
Eastern and Southern Africa region
ESSP
Education Sector Strategic Plan
ETF
Education Transition Fund
GPE
Global Partnership for Education
GPI
Gender Parity Index
IEA
International Association for the Evaluation of Educational Achievement
IEP
Integrated Education Programme
IfE
Innovation for Education
IIEP
International Institute for Educational Planning
IRT
Item response theory
LARS
Learning Achievement in Rwandan Schools
LLO
Limited learning outcomes
LNAEP
Lesotho National Assessment of Educational Progress
MICS
Multiple indicator cluster surveys
MLA
Monitoring learning achievement
iv
MoESAC Ministry of Education, Sport, Arts and Culture of Zimbabwe
MTPDS
Malawi Teacher Professional Development Support
NALA
National Assessment of Learner Achievement
NAPE
National Assessment of Progress in Education
NASMLA
National Assessment System for Monitoring Learning Achievement
NER
Net enrolment rate
NLA
National Learning Assessment
NSAT
National Standardized Achievement Test
OECD
Organisation for Economic Co-operation and Development
OVC
Orphans and vulnerable children
PASEC
Programme for the Analysis of the Education Systems of CONFEMEN Countries
PIRLS
Progress in International Reading Literacy Study
PISA
Programme for International Student Assessment
PRIMR
Primary Math and Reading Initiative
RCT
Randomised control trial
READ
Russia Education Aid for Development
REB
Rwanda Education Board
RTI
Research Triangle Institute
SACMEQ
Southern and Eastern Africa Consortium for Monitoring Educational Quality
SBA
School-based assessment
SSME
Snapshot of School Management Effectiveness
TAC
Teacher Advisory Centre
TIMSS
Trends in International Mathematics and Science Study
UNESCO
United Nations Educational, Scientific and Cultural Organization
UNICEF
United Nations Children’s Fund
UNICEF ESARO UNICEF Eastern and Southern Africa Regional Office
USAID
United States Agency for International Development
ZELA
Zimbabwe Early Learning Assessment
ZIMSEC
Zimbabwe School Examinations Council
v
vi
Acknowledgements
The Australian Council for Educational Research (ACER) was contracted by the United Nations
Children’s Fund (UNICEF) to deliver a consultancy service for improving the quality of education
and children’s learning outcomes and effective practices in the Eastern and Southern Africa region.
This study was co-financed by UNICEF and ACER’s Centre for Global Education Monitoring.1 The
authors of this report would like to thank Manuel Cardoso, Education Specialist, Programme
Division, UNICEF; Camille R. Baudot, Regional Adviser, Basic Education and Gender Equality
(BEGE), UNICEF Eastern and Southern Africa Regional Office (ESARO); Mitsue Uemura, Shiraz
Chakera, Inge Vervloesem, Benoit d’Ansembourg and Pablo Stansbery, Education Specialists
from BEGE, UNICEF ESARO, as well as the focal persons at the 21 UNICEF Country Offices in
the Eastern and Southern Africa Region (ESAR) for their support during the conduct of the study
and preparation of the report.2
We would also like to thank the key informants who took part in the interviews conducted as
part of this study: John Mugo (Regional Director of Uwezo at Twaweza); Ben Piper (Lead in the
Primary Maths and Reading Initiative, PRIMR, in Kenya) and the Kenya PRIMR team; Kenneth
Russell (Education Specialist, UNICEF Zimbabwe Country Office); Professor Robert McCormick
(Innovation for Education Evaluation and Monitoring advisor); and Joyce Kinyanjui (Programme
Manager, Women Educational Researchers of Kenya, Opportunity Schools).3
Thanks also to the many other individuals who provided information for this study, including
Aaron Benavot (Director, GMR team, UNESCO) and Nihan Blanchy-Koseleci (UNESCO GMR);
Scott Murray (DataAngel); Marc van der Stouwe (Mott MacDonald/Cambridge Education);
Lucy Maina (Regional Director of Africa Educational Trust, Somalia); and Vyjayanthi Sankar
(International Consultant, Quality Education, UNICEF ESARO). We would also like to acknowledge
the contributions of several education authorities in the ESA region, including the Botswana
Ministry of Education and Skills Development; the Botswana Examinations Council; the Ethiopia
National Educational Assessment and Examinations Agency; the Examinations Council of
Lesotho; the Madagascar Ministry of Education; the Malawi Ministry of Education, Science and
Technology; the Mozambique Ministry of Education; the Directorate of National Examinations
and Assessment Namibia; the Director-General Department of Basic Education of South Africa;
the Uganda National Examinations Board; the Examination Council of Zambia; and the Zimbabwe
School Examinations Council.
The authors of this report are Tim Friedman, Ursula Schwantner, Jeaniene Spink, Naoko Tabata
and Charlotte Waters, with valuable contributions from Elizabeth Cassity, Mary Kimani, Alejandra
Osses, and Adeola Capel.
1
For further information about ACER’s Centre for Global Education Monitoring (GEM) visit https://www.acer.edu.au/gem.
2
The Eastern and Southern Africa (ESA) region, as under UNICEF programming, encompasses 21 countries: Angola, Botswana,
Burundi, the Comoros, Eritrea, Ethiopia, Kenya, Lesotho, Madagascar, Malawi, Mozambique, Namibia, Rwanda, Somalia, South
Africa, South Sudan, Swaziland, the United Republic of Tanzania (later in this report referred to as ‘Tanzania’), Uganda, Zambia and
Zimbabwe.
3
Not all the interviews conducted for this study were included in the report. However, the contributions were highly valuable and
the authors thank all interview partners for their time and information.
vii
Foreword from the Eastern
and Southern Africa Regional
Director
Over the past few decades the world’s attention has
been focussed on attaining Millennium Development Goal
2 – universal access to primary education. During this
period, governments and the international community
have been investing on school infrastructure, training
teachers, and learning materials. For UNICEF globally and
across Eastern and Southern Africa, the challenge of our
time is now how to sustain the momentum in access and
to reinforce quality learning outcomes. There is, today, an
urgent global realisation that beyond getting children in
the classroom, it is imperative that they learn.
The new Sustainable Development Goal (SDG) 4, presents
huge opportunities to meet this challenge through a
strategic shift towards equitable quality education for
all. This shift is essential. Emerging evidence shows
that large numbers of children are in school, but are not
learning. In 2012, the Africa Barometer report by the
Centre for Universal Education at Brookings estimated
that of the 97 million who enter school on time in SubSaharan Africa, 37 million will not learn basic skills. Thirty seven million: that is one-third of all
children who go to school will reach their adolescent years unable to read, write, and/or perform
basic numeracy tasks.
The analysis from Improving quality education and children’s learning outcomes and effective
practices in the Eastern and Southern Africa region has similar conclusions. It reveals that as
many as 40 per cent of children in school do not reach the expected basic learning benchmarks
in numeracy and literacy. The new report also confirms that children from families with lower
socio-economic status and whose home language is different from the language of instruction
are less likely to learn.
The findings – that many children are in school, but not learning – represent a huge waste of
human and financial resources. Fundamentally, the promise of education and the transformative
opportunity of schooling for children, families and communities is not being fulfilled.
UNICEF believes that confronting this learning crisis, through high impact solutions, is the priority
for the education community. Encouragingly, the report highlights that many countries in East
and Southern Africa are promoting quality through improving learning monitoring by national,
regional and international learning assessments, and through developing targeted programmes
that improve teaching and learning.
viii
Given this learning crisis, acceleration of these trends is essential. UNICEF will push for an
increased system-wide emphasis on outcomes as opposed to inputs; improved assessments to
gauge children’s learning progress; and building knowledge of the pedagogic practices that can
improve learning.
There is so much potential. While highlighting challenges, this report also shows elements of
national progress. UNICEF encourages countries to accelerate these developments. We are
cognisant of the many challenges facing education in the region – 1 in 5 children not attending
school; a demographic boom in the region that will see 70 million additional children by 2030; and
continued overstretched public finances. In such critical circumstances, the winning combination
of access, quality learning, and affordability is ever more crucial.
It is in this context that the report provides us with a critical baseline on quality education in
every country in the region. The report assesses the learning outcomes being reached in the
region, the learning assessment tools countries are deploying to generate evidence on learning,
and the interventions countries are implementing to improve teaching and learning.
With this report, UNICEF and our many partners will be better equipped to support improvements
in quality education for children.
Leila Gharagozloo-Pakkala
Regional Director
Eastern and Southern Africa
United Nations Children’s Fund ix
Executive summary
Introduction
The Eastern and Southern Africa (ESA) region is progressing well towards achieving important
Education for All (EFA) goals, particularly with regard to increasing student enrolment in the
primary years (UNESCO, 2014). Despite this achievement, there is still considerable work to be
done to improve the quality of education. Primary school students in low-income sub-Saharan
African countries have, on average, learned less than half of what is expected of them (Majgaard
and Mingat, 2012, p.6). The gap between the learning achievements in developed economies
and the learning achievements in Eastern and Southern Africa is estimated to be at least four
grades (GPE, 2012, p.116).
In order to understand the major impediments to student learning in the region, the United
Nations Children’s Fund (UNICEF) contracted the Australian Council for Educational Research
(ACER) to take stock of and compare existing student assessments in the region, focusing on
students in primary education. The terms of the contract called for ACER to study the existing
assessment systems and methodologies in the region, and document how the assessment data
are derived and used to inform education policy in the region. We were also asked to identify
factors and practices that could help improve learning outcomes in literacy and numeracy in
primary education, specifically for disadvantaged children with limited learning outcomes (LLOs).
Our study consists of three research components. The first provides an overview and comparative
analysis of the existing assessments of student learning outcomes in literacy and numeracy in
primary education in the region. The second considers the characteristics of children experiencing
LLOs in the domains of literacy and numeracy, including trends in achievement over time. The
third looks at effective country-level practices in the ESA region that could improve learning
outcomes in the literacy and numeracy of disadvantaged children in primary education. Our
report concludes with a macro theory of change drawing on the evidence we gathered for this
report.
x
Comparative analysis of existing assessments in the
ESA region
Our study covered 23 countries and identified 58 existing assessment systems that evaluate
student learning outcomes in literacy and numeracy in primary education. Of these, EGRA and
EGMA are the most prevalent programmes (36 per cent) followed by regional (29 per cent),
national (28 per cent) and international (7 per cent) assessments. Most of these assessments
target lower-primary students (grade 2 or 3) and most commonly focus on literacy and numeracy
assessments. While these commonly use mean scores for the cognitive results or frequency
analysis for the contextual data, item response theory (IRT) methods, which can scale data and
meaningfully compare results across grades, contexts and time, is less prevalent. Contextual
data linked to the cognitive results is available for many, but not all of the assessments, making
it difficult to draw policy-related findings from the results. Not captured in our stock-taking was
whether, and to what extent, student assessment data is linked to Education Management
Information System (EMIS) data on a systems level. Access to the data is a challenge, and our
study found that while the results of 71 per cent of the assessments were published, we were
unable to obtain the original datasets for the remainder.
Students experiencing LLOs and trends over time
The objective of the study was to investigate the characteristics of children experiencing LLOs
and trends in their performance over time. In 32 out of the 58 assessments, competencylevel benchmarks are defined. However, each of the assessments (PASEC, UWEZO, TIMSS
and prePIRLS) used different metrics for literacy and numeracy. Therefore, there is no shared
benchmark among them that could be used to construct a common definition of ‘limited learning
outcomes.’ Instead, we employed the benchmarks each assessment used to gauge literacy and
numeracy. Given the differences in these metrics across data sets, countries and year level, the
percentage of students identified with LLOs is wide ranging, from 18 per cent to 40 per cent in
numeracy, and 18 per cent to 50 per cent for literacy.
In international and regional assessments for the ESA region, average test scores for literacy
and numeracy are generally low, with a considerable percentage of students failing to have
acquired basic skills in reading and mathematics. In Lesotho, for example, by Grade 6 only 48 per
cent of students have achieved basic reading skills. In Zambia and Malawi, only 27 per cent of
students achieved this level. In mathematics, the proportion of primary students with basic skills
is considerably lower, with fewer than 50 per cent of students in Grade 6 achieving the minimum
level in two-thirds of the countries (UNESCO, 2014, p.35).
xi
Consistent with other studies conducted in the region, individual and family characteristics of
students, such as gender, age, language spoken at home, socio-economic factors, preschool
attendance, activities prior to attending school, engagement and out-of-school tuition, were all
found to be associated with the likelihood that a student would experience LLOs in literacy or
numeracy. In addition, the type of school, the location of the school and the resourcing available
to the school that the student attends also contributes to the likelihood that the student would
be experiencing LLOs.
Our study showed that, in general, males are more likely to experience LLOs in literacy than
females. In Botswana, for example, males are almost three times more likely than females to
experience LLOs. While on the whole girls outperformed boys in reading literacy, rural boys
outperformed rural girls on almost all tasks, but in urban schools the opposite was the case (RTI,
2010, p.37). However, in mathematics, on the whole, boys outperformed girls.
The age of the student relative to the school entry grade is an equally important factor. While
the relations between age and performance are complex and may be determined by different
socio-economic and demographic factors, Hungi et al (2014) found that in developed countries
older students generally outperform their younger colleagues, while in developing countries,
especially in Africa, younger students perform better than older students. Our study supported
this insight. Across the region, we found that students who were relatively younger than the
median class age tended to be less likely to be experiencing LLOs. For instance, Grade 6 students
in Botswana who were 12 years or less were almost three times less likely to be experiencing
LLOs in mathematics than students 12 years of age or older.
The language spoken at home also has a strong impact on learning outcomes. In countries
where the official language is not the most common language spoken at home, there are strong
links between language and marginalisation in education. Evidence from PASEC and SACMEQ
show a strong link between home language and the language of instruction in determining test
scores (Fehrler and Michaelowa, 2009, UNESCO 2010, p. 154; Garrouste, 2011). While low
language skills are commonly viewed as a critical factor in literacy assessments, evidence from
Namibia using SACMEQ results suggests that they also make a significant contribution to low
performance levels in mathematics (Garrouste, 2011, p. 231).
Furthermore, the socio-economic status of students is a strong predictor of achievement. Our
study found that students from lower socio-economic backgrounds were more likely to experience
LLOs across all countries examined in both literacy and numeracy. We found this to be the
case across all countries, despite the different measures used to assess socio-economic status.
Among other factors, household possessions, including the availability of reading materials and
books in the home, and levels of parental education, were also found to be associated with LLOs.
Furthermore, the amount of time that students spent working was negatively associated with
achievement data (ACER and ZIMSEC, 2015).
xii
Students who had limited exposure to a learning environment in the home were disadvantaged
in performance at school. A profound impact on learning outcomes was evident in homes where
students were involved in reading and storytelling, were not required to work outside of the
home, started school early and were provided with adequate support in school by their teachers
to build foundational literacy skills, and attended schools that had relevant and engaging reading
and learning materials in buildings with clean water and sanitation.
Our study also considered student performance trends over time. While much of Eastern
and Southern Africa has experienced a marked improvement in student enrolment, student
performance has changed little over time. Indeed, on the whole, student performance has stagnated
or worsened. It must be noted, however, that given the limited comparable data available for
our study, drawing any general conclusions for the whole ESA region is problematic. Instead,
conclusions based on improvement or decline in student abilities should only be considered at
the national level.
Effective country-level practices
As part of our study, a number of strategies were identified that contributed to the success of
country-level practices in the ESA region. We found that while there is a considerable body of
literature for the ESA region on practices that increase quantitative aspects of education quality,
such as access, enrolment and retention rates, we found few reports on programmes to improve
student learning outcomes.
Altogether, we identified 10 programmes in 7 out of the 21 countries identified as having had
an impact on student learning in early grade literacy/numeracy. These comprised of teacher
training on reading/mathematics instruction; provision of teaching-learning materials; production
of reading materials in the local language; and community- and home-based reading activities
which were linked to effective ECD programmes. Additionally, programmes that aimed at a wholeschool improvement strategy were shown to have a significant impact on learning outcomes.
Broadly, these successful programmes use a three-pronged approach comprised of assessments,
teacher training and community support for children’s reading. They provide a combination of
well-targeted instructional interventions, regular professional development of teachers through
school-level training and coaching, with regular system-level follow up and support, matched with
sufficient relevant and quality classroom materials, and more literacy and numeracy instructional
time. Students having a reading buddy to support their learning to read had a positive effect
on learning outcomes in a number of locations. Overall, our study found that key strategies for
improving learning outcomes of disadvantaged children share two common features: a holistic
and coherent approach and consistent and continuous support over time.
xiii
A macro theory of change
Based on the evidence collected for our study, we developed a macro theory of change aiming
at monitoring and improving literacy and numeracy performance of children in primary education
in the region. The theory combines the main findings of each stage of the study and highlights
the ‘3As’ approach for long-term and sustainable change in student performance: assessment,
analysis and action. Critical to this framework is the dissemination of the assessment results
in order to initiate action by governments, communities, parents and development partners (see
Figure 1).
Figure 1. A macro theory of change. An evidence-based monitoring and intervention cycle as
premise for change: assessment, analysis, action
Evidence-based monitoring and intervention cycle
Output
School level
Classroom level
Student level
• Purpose: System level monitoring
• Target population: Early, multiple grades,
inclusion of out-of-school children
• Domains: Literacy and numeracy; Contexts
• Current state and progress: Performance and
contexts
• Dissemination strategy: Findings and
products, including datasets
Input
Process
Outcome
Assessment
xiv
Action
Analysis
• Target interventions and
strategies
• Integrated into a holistic
programme design, involving a
wide range of stakeholders
• Impact evaluation including
measuremt of performance
Policy analysis and interpretation
for strategic decision-making
and policy development towards
improving student performance
• Student performance levels
• Association with context
factors at the different levels
• Trends over time
Conclusions
Our study analysed existing student assessments, data resulting from some of these assessments
and effective country-level programmes in the ESA region.
We found that programmes targeted toward early learning in disadvantaged communities made
the biggest impact. The level of exposure students have to a learning environment, either through
home or school, in their early years and the presence of holistic, system-level educational
programmes that support quality early learning programmes in disadvantaged communities made
a significant contribution to improved student performance.
While many researchers have studied the factors that contribute to student school attendance,
fewer have explored what helps improve student learning. In financially constrained environments,
resources should be targeted at understanding the gaps in the system with regard to student
performance and supporting effective interventions.
In order to do this, policymakers must consider how learning assessment programmes that
provide quality comparable data across population subsets, between grades and over time, can
be integrated from the outset into education reform agendas.
xv
xvi
Introduction
Context for primary education in the ESA region
The Eastern and Southern Africa (ESA) region provides many challenges for primary education. The
main issues are poverty, health and social issues, and fragile political/economic circumstances—
particularly in Burundi, the Comoros, Eritrea, Madagascar, Somalia and South Sudan (UNESCO,
2014, p. 12).4,5
Overall, the region is making progress toward achieving Education for All (EFA) goals (UNESCO,
2014). The EFA Development Index shows a considerable improvement between 2000 and 2012,
with increases in the primary education completion rate, the literacy rate for those 15 years and
older, and the primary enrolment of girls and boys (UNESCO, 2014, p. 16; 2015a).
However, very little attention has been paid to the quality of education, or to progress in student
performance. While improvements in education can be assessed by quantitative aspects, such as
access to education, enrolment and completion rates, or gender parity, such metrics don’t assess
education quality or, even more importantly, the improvement of student performance.
In most sub-Saharan African countries, average test scores in international/regional assessments
of student learning are low. Primary school students in low-income, sub-Saharan African countries
have, on average, learned less than half of what is expected of them (Majgaard and Mingat,
2012, p. 6). A comparison of high- and low-income countries using data from the Southern
and Eastern Africa Consortium for Monitoring Educational Quality (SACMEQ) and the Progress
in International Reading Literacy Study (PIRLS) reveals large differences between the poorer
economies in SACMEQ (for example, Lesotho, Malawi and Zambia) and the mainly high-income
economies in PIRLS.6,7
The gap between the learning achievements in developed economies and the learning achievements
in East and Southern Africa is estimated to be at least four grades (GPE, 2012, p. 116).8
4
Most countries in the region (16 out of 21) are classified as least developed countries under the United Nations (UN) definition
<http://unohrlls.org/about-ldcs/>. Kenya is among low income countries; Swaziland forms part of the lower-middle income
country group; and Botswana, Namibia and South Africa are among upper-middle income countries (according to DAC-ODA–
recipient status (OECD Development Assistance Committee, Official Development Assistance). Source: DAC. List of ODA
Recipients effective as at 1 January 2015 for reporting on 2014, 2015 and 2016 flows available at <http://www.oecd.org/dac/
stats/daclist.htm>. However, there are also major disparities within these middle income countries; see <http://www.unicef.org/
esaro/theregion_old.html>.
5
<http://www.worldbank.org/en/topic/fragilityconflictviolence> viewed 5 May 2015; in 2014, Malawi was also categorised as a
fragile state.
6
ESAR countries participating in SACMEQ are Botswana, Kenya, Lesotho, Malawi, Mozambique, Namibia, South Africa, Swaziland,
Tanzania, Uganda, Zambia and Zimbabwe.
7
Ross (2009) scaled PIRLS and SACMEQ data using common items and a Rasch model to put the test results on the same scale,
based on anchor items and test equating, thus making the data comparable across economies (GPE, 2012, p. 116).
8
Considering that PIRLS measures learning outcomes of primary school students in Grade 4, and SACMEQ in Grade 6, this
potentially corresponds to two more grades of schooling (GPE, 2012, p. 116).
1
For this study we reviewed the assessment systems in the region that measure learning outcomes
in literacy and numeracy of children in primary education, and undertook a comparative analysis
of these assessments. We analysed existing data from the region to develop a portrait of students
who are experiencing LLOs in literacy and numeracy, and examined trends in performance over
time. We investigated education interventions that helped to improve students’ learning outcomes
in literacy and numeracy, especially for disadvantaged children, to identify strategies to further
improve student performance in the region. Using these elements, we developed a theory of
change that sets out processes for improved learning outcomes.
Conceptual framework
Policies aiming at monitoring and improving educational progress must be based on data on
learning outcomes and the factors related to the outcomes. ‘Learning outcomes’, as defined
in this study, refer to student performance in a cognitive domain, in particular literacy and
numeracy, as measured in the different assessments discussed in Chapter 1. The conceptual
framework we used sets out the contextual factors associated with student performance, which
provides a basis for the definition of factors to be considered in an assessment and support for
the decision on which level of the education system that innovations are most needed.
The conceptual framework combines the main characteristics of two models that have both been
highly influential in the field: the ‘Input-Process-Outcome model’ (Purves, 1987) and the ‘dynamic’
model of educational effectiveness (Creemers and Kyriakides, 2008; Kyriakides and Creemers,
2006).9,10 In these models, input, process and outcome factors operate at the different levels of
an education system, i.e., system-level (national, regional and community); as well as school-,
classroom- and student level. Input factors mainly refer to structural conditions, for example,
economic wealth or community infrastructure (system level); school type (public, private); school
location (rural, urban); and school resources (school level); class-size and teaching resources
(classroom level); as well as individual (e.g. gender, age, grade) and family factors, such as socioeconomic status and parental education (student level). Process factors mainly concern policies
and strategies, and range from national curriculum and teacher education (system level), through
management and leadership (school level), quality of instruction (classroom level), to the actual
learning process (student level). Outcome factors—i.e., performance in literacy and numeracy—
are measured at the student level and can be aggregated at the system level (national, regional
and community level), school level, or classroom level.11 Relations between the factors at the
different levels are complex and have not yet been fully investigated. In the ‘dynamic model,’
educational outcomes can become inputs for further development. For example, domain-related
attitudes and beliefs can be considered as outcomes of schooling or as inputs affecting student
behaviour (Creemers and Kyriakides, 2008; Kyriakides and Creemers, 2006; OECD, 2013).
9
The basic structure of the Input-Process-Outcome model was developed in the 1960s for the International Association for the
Evaluation of Educational Achievement (IEA) (Purves, 1987).
10 Other recent examples where the main characteristics of these two models are combined are the contextual framework for OECD
Programme for International Student Assessment (PISA) (OECD, 2013), or the input-process-output model for data in the student
learning environment (Biggs, 1999; Biggs and Moore, 1993).
11
2
Other outcome factors at the system, school, and classroom level relate to aggregated pass rates and graduation rates, enrolment
and retention rates; at the student level, domain-related and general school-related attitudes, beliefs and motivation are considered
important outcomes of schooling.
From an assessment-design perspective – regarding data collection of input, process and
outcome factors on the four levels – it may not always be possible, nor feasible, to represent the
full framework. The main source for information on input and process variables at the student
and school level are questionnaires, sometimes combined with qualitative methods, such as
classroom or school observation. Data on input and process factors on the system level are
collected at the system level, using tools such as the Education Management Information System
(EMIS). Outcome factors, i.e., performance in literacy and numeracy, are measured at student
level using cognitive tests.
The comprehensive conceptual framework allows the main effectiveness factors to be considered
in an assessment. The framework also has potential to inform the development of theories of
change: Input, process and outcome factors can be defined and analysed for their stability
and suitability for change. The framework can also inform decisions about the level at which
innovations need to be implemented to ensure their maximum effectiveness. For example,
determining which input, process and outcome factors need to be addressed by policy decisions
at the system and school level, and which can be best shaped by school development activities
at the school and classroom level.
3
4
1.Stock-taking and
comparative analysis of
existing assessments in the
ESA region
A wide range of assessment systems is in use in the ESA region.12 For a better understanding of
these assessments, the available data and how they support policies aiming at monitoring and
improving learning outcomes, we started with an overview of the systems. The focus was on
assessments that measure student performance in literacy and numeracy in primary education.
Following the overview, we analysed and compared the assessments, highlighting the strengths
and limitations of the different methodological approaches. Case studies for Zimbabwe and
Rwanda offer an in-depth understanding of specific measurement practices implemented by the
governments of these countries (see Appendix II). The detailed results of the stock-taking are
presented in the main stock-taking table (see Appendix VI).
1.1Overview of assessments in the stock-taking
The stock-taking considered assessments that were implemented from 2007 up to 2014/2015
in the ESA region.13 Overall, we identified 58 assessments that assess student performance in
literacy and numeracy in primary education in the ESA region.
The assessments can be grouped into four types:
• international assessments
• regional assessments
• national assessments
• Early Grade Reading Assessment (EGRA)/Early Grade Mathematics Assessment (EGMA).
Of the 58 assessments, 4 (7 per cent) are international and 17 (29 per cent) are regional.
Implementation of EGRA and EGMA accounts for 21 (36 per cent), which is the largest group
among the four types, and 16 (28 per cent) are national (see Figure 2).14
12 National examinations are not included in this study. Examinations don’t share the same purpose as learning assessments, which
leads to different choices about sampling, data analysis, reporting (e.g. on examination pass rates and/or achieved grades, i.e. pass
levels) etc.
13 The stock-taking of assessments took place between October 2014 and March 2015 and considers assessments that have
been implemented since 2007. Assessments conducted outside this period were not considered. In the case of reoccurring
assessments, the inception date of an assessment can be before 2007.
14 All percentages are rounded.
5
Figure 2. Assessments from the stock-taking by type of assessment
4(7%)
International
16(28%)
17(29%)
Regional
EGRA/EGMA
National
21(36%)
Overall, 20 countries have implemented one or more assessments (see Table 16 in Appendix II).
Six countries have used several types of assessments (i.e., Kenya, Malawi, Mozambique, South
Africa, Uganda and Zambia), while another six countries show limited assessment activities with
one or no assessment implementation in recent years (i.e., Angola, Comoros, Eritrea, Madagascar,
South Sudan and Swaziland) (see Table 17 in Appendix II).15
Participation in international assessments
Two countries in the region have participated in international assessments of the IEA (International
Association for the Evaluation of Educational Achievement): TIMSS (Trends in International
Mathematics and Science Study), PIRLS (Progress in International Reading Literacy Study) and
prePIRLS (see Table 1).16
Table 1. Participation in international assessments in ESAR countries
Country
Assessment
Botswana
TIMSS in 2007, 2011
PIRLS in 2011, prePIRLS in 2011
South Africa
TIMSS in 2011
PIRLS in 2006, 2011; prePIRLS in 2011
Since 1995, TIMSS has been measuring trends in mathematics and science achievement at
Grade 4 and Grade 8. Where it was expected that students in Grades 4 and 8 would find
TIMSS assessments too difficult, IEA encouraged countries to test children in higher grades.
15 Four of these countries are characterised as fragile states (Comoros, Eritrea, Madagascar and South Sudan), which might be one
reason for the limited assessment activities.
16 Zambia will participate in the pilot for PISA for Development in the coming years. This OECD/World Bank-led assessment aims to
enhance PISA’s survey instruments to make them more relevant for contexts found in developing countries, but still permit the
reporting of results on the standard PISA scales. For more information about PISA for Development, see <http://www.oecd.org/
pisa/aboutpisa/pisafordevelopment.htm>.
6
Thus, in TIMSS 2011 Botswana tested children in Grade 6 with the Grade 4 assessment, and
children in Grade 9 with the Grade 8 assessment.17 South Africa took part in TIMSS in 2011 for
the first time, testing Grade 9 children with the TIMSS Grade 8 assessment.18
First introduced in 2001, PIRLS measures trends in reading comprehension at Grade 4. In 2011,
PIRLS was expanded to include prePIRLS, which is a less difficult and shorter version of PIRLS.
PrePIRLS assesses the basic reading skills at the end of the primary school cycle that are a
prerequisite for success in PIRLS (Mullis and Martin, 2013, p. 4). Thus prePIRLS permits learners
from lower achieving countries to be measured more precisely than is the case using more
difficult and longer assessments, such as PIRLS (Howie, Staden, Tshele, Dowse, and Zimmerman,
2012, p. 22).19
Participation in regional assessments
Fourteen countries in the region have participated in one or more of the regional assessments of
SACMEQ (Southern Africa Consortium for Monitoring Educational Quality), PASEC (Programme
for the Analysis of the Educational Systems of CONFEMEN Countries) and Uwezo (see Table 2).
Table 2. Participation in regional assessments in ESAR countries
Country
Assessment
Botswana
SACMEQ II and III
Burundi
PASEC in 2008–2009
Comoros
PASEC in 2008–2009
Kenya
SACMEQ I, II and III Uwezo
Lesotho
SACMEQ II and III
Malawi
SACMEQ I, II and III
Mozambique
SACMEQ II and III
Namibia
SACMEQ I, II and III
South Africa
SACMEQ II and III
Swaziland
SACMEQ II and III
Tanzania
SACMEQ II and III20 Uwezo
Uganda
SACMEQ II and III Uwezo
Zambia
SACMEQ I, II and III
Zimbabwe
SACMEQ I and III
17 In TIMSS 2007, Botswana participated with Grade 8.
18 TIMSS 2015 introduces a new, less difficult mathematics assessment called TIMSS Numeracy for countries where most children
are still developing fundamental mathematics skills. TIMSS Numeracy assesses fundamental mathematical knowledge, procedures,
and problem-solving strategies at the end of the primary school cycle that are prerequisites for success on TIMSS (I. V.S. Mullis
and Martin, 2013, pp. 7–8).
19 PrePIRLS and TIMSS Numeracy intend to be responsive to the needs of the global education community and efforts to work
towards universal learning for all children. Depending on a country’s educational development, prePIRLS and TIMSS Numeracy can
be given at Grade 4, 5 or 6 (I. V.S. Mullis and Martin, 2013, pp. 7–8).
20 Only Zanzibar of Tanzania participated in SACMEQ I.
7
SACMEQ carries out large-scale, cross-national research studies in the Southern and Eastern
Africa region. It assesses the performance levels of Grade 6 students and teachers in literacy
and numeracy (ACER, 2015).
PASEC is an assessment programme for countries that have a link to the French-speaking
community. It was established in 1991 by the Conference of Ministers of Education of Frenchspeaking Countries (CONFEMEN), and it assesses Grade 2 and Grade 5 students in reading and
mathematics (CONFEMEN, n.d.).
Uwezo measures the literacy and numeracy competencies of school-aged children in Kenya,
Tanzania and Uganda. Its goal is to obtain data to inform improvements in educational policy and
practice (Twaweza, n.d.).
Implementation of national assessments
In 13 countries in the region, national assessments have been implemented by government or
parastatal bodies (see Table 3). In six countries (Eritrea, Ethiopia, Malawi, Rwanda, Somalia
and Zimbabwe), an international development partner was directly involved in supporting the
implementation.
Table 3. Implementation of national assessments by government/parastatal bodies in
ESAR countries
Country
Eritrea
Assessment
Monitoring Learning
Achievement (MLA)
Ethiopia
National Learning
Assessment (NLA)
Kenya
National Assessment
System for
Monitoring Learning
Outcomes (NASMLA)
Lesotho National
Assessment of
Educational Progress
(LNAEP)
Assessing Learner
Achievement
Monitoring Learning
Achievement (MLA)
Lesotho
Malawi*
Notes
Conducted irregularly by ministry and UNICEF
Most recent reported implementation in 2008
Conducted on a 3–4-year cycle by national
assessment/examination body and USAID
Most recent reported implementation in 2010–2011
Conducted at an unknown frequency by national
assessment/examination body
Most recent reported implementation in 2010
Conducted on a 1–2-year cycle by national
assessment/examination body
Most recent reported implementation in 2010
Conducted by ministry or national assessment/
examination body
Malawi
Conducted on an intended 3-year cycle by ministry
and UNICEF
First implementation in 2012
Mozambique* National Assessment Conducted by ministry or national assessment/
examination body
Namibia*
National Standardised Conducted biannually by national assessment/
Achievement Test
examination body
(NSAT)
8
Rwanda
Learning Achievement Conducted on a 3-year cycle by national assessment/
in Rwandan Schools examination body, UNICEF and UNESCO
(LARS)
First implementation in 2011, second in 2014
Somalia*
Monitoring Learning
Conducted irregularly by ministry and UNICEF
Achievement (MLA)
South Africa Annual National
Conducted annually by the ministry
Assessment
Most recent reported implementation in 2014
South Africa* National Assessment Conducted by ministry or national assessment/
of Learner
examination body
Achievement (NALA)
Uganda
National Assessment Conducted on a 1–3-year cycle by national
of Progress in
assessment/examination body
Education (NAPE)
Most recent reported implementation in 2010
Zambia
National Assessment Conducted on a 2-year cycle by the national
of Learning
assessment/examination body
Achievement (NALA) Most recently reported implementation in 2014
Zimbabwe
Zimbabwe Early
Conducted in 2012–2015 by the national
Learning Assessment assessment/examination body and UNICEF.
(ZELA)
*Note: These assessments are mentioned in the EFA Global Monitoring Report (UNESCO 2008, p. 2; 2015b), but only limited
information was available for the main stock-taking table (see Table 24).
Implementation of EGRA/EGMA
The Early Grade Reading Assessment (EGRA) and Early Grade Mathematics Assessment (EGMA)
measure the most basic foundation skills for literacy and numeracy acquisition in the early
grades. These assessments were developed by the Research Triangle Institute (RTI), with funding
provided by the United States Agency for International Development (USAID) and the World Bank
(Gove and Wetterberg, 2011). EGRA/EGMA was designed to serve as a sample-based national or
system-level diagnostic measure that would reveal gaps in reading competencies among students
and inform education ministries and development partners about system needs for improving the
professional development of teachers and pre-service programmes (Gove and Wetterberg, 2011).
However, EGRA/EGMA has been used to address a wider range of assessment needs, including
impact (programme) evaluations.
Although EGRA and EGMA have been used in many developing countries, including ESA and
other regions, they are not grouped as international assessments in this report. This is because
EGRA and EGMA have a common approach grounded on core foundation skills. At the same time,
they can be adapted for use in individual countries and languages. This approach differs from
international assessments where implementing countries are required to use an internationally
agreed model (ACER, 2014a). The adaptability of EGRA and EGMA also means that direct
comparison of the results is difficult due to differences in language structure and complexity. For
this reason, developers of these assessment tools generally advise against comparing subtask
results across countries and languages (Gove and Wetterberg, 2011).
9
Twelve countries in the region have completed at least one implementation of EGRA/EGMA for
either system-level diagnostic, monitoring or programme evaluation (see Table 4).21
The EGRA/EGMA tools are equally used for both purposes: in nine cases, EGRA and/or EGMA
were implemented for system-level diagnostics, and in 10 cases they were used for programme
evaluation.22 In two cases, EGRA and/or EGMA were implemented for the purpose of systemlevel monitoring.
Table 4. Implementations of EGRA/EGMA in ESAR countries23
Country
Assessment
Year
Purpose
Angola
EGRA
2010
System-level diagnostic
Burundi
EGRA
2011
System level diagnostic
Ethiopia
EGRA
2010
System-level diagnostic
2010–2012 Programme evaluation (Literacy Boost initiative)
2011
System-level diagnostic
EGRA
2007–
2008
Programme evaluation (EMACK initiative)
EGRA, EGMA
2012–2013 Programme evaluation (PRIMR initiative)
Madagascar
EGRA
2009
System-level diagnostic
Malawi
EGMA
2010
Programme evaluation (baseline for Malawi
Teacher Professional Development Support
initiative)
EGRA
2009–2010 Programme evaluation (Literacy Boost initiative)
Kenya
2010–2012 System-level monitoring
Mozambique
EGRA
2010–2011
Programme evaluation (Literacy Boost initiative)
2013
Programme evaluation (USAID/Aprender a Ler
(APAL) initiative)
System-level diagnostic
Rwanda
EGRA, EGMA
2011
Somalia
EGRA
2013–2014 Programme evaluation
Tanzania
EGRA, EGMA
2013
System-level monitoring
Uganda
EGRA
2009
System-level diagnostic
2010, 2012 Programme evaluation (Literacy Boost initiative)
Zambia
EGRA, EGMA
2011
Pilot for system-level diagnostic
EGRA
2012
Programme evaluation
2014
System-level diagnostic (as part of the National
Assessment Survey)
21 Further details of each EGRA/EGMA implementation can be found in the main stock-taking table in Appendix IV.
22 Assessments with a system-level diagnostic purpose are implemented to get a snapshot of learning levels at the system level
(usually national, and in a one-off administration), as compared to assessments with a system-level monitoring purpose which have
recurrent administrations to monitor learning levels at the system level.
23 We acknowledge that there are current EGRA/EGMA implementations which are not considered in our stock-taking and that took
place between October 2014 and March 2015. Assessments conducted outside this period were not considered. We wish to
thank UNICEF CO Rwanda and Tanzania for their input about a School Quality Assessment in three regions using EGRA/EGMA
methodology supported by UNICEF which was completed in 2015 focusing on providing baselines in the three UNICEF targeted
regions.
10
1.2Comparative analysis of assessments
Our comparative analysis of the assessments in the region highlights the strengths and limitations
of the different methodological approaches, and discusses when the different approaches
might be more or less appropriate. In some instances, best practices in terms of methodological
approaches are discussed, even if they do not feature in the majority of the assessments.
The assessments are analysed and compared based on the main eight elements of the stocktaking framework: purpose of the assessment; target population (grade-based, e.g., Grade 4, or
age-based, e.g., 10-year-old students); sampling design and methodology; cognitive domains
(assessment framework and major domains, i.e., literacy and numeracy); contextual instruments
(types, e.g., student questionnaire, and key factors, e.g., gender, grade level, parental education);
test administration (approaches for data collection); data analysis (key analytical approaches),
and reporting and dissemination products.24
1.2.1Purpose
There are three types of assessment purposes:
• System-level monitoring: Recurrent administrations to monitor learning levels at the system
level (usually national).
• System-level diagnostic: One-off administration for a snapshot of learning levels at the
system level (usually national).
• Programme evaluation: Smaller scale administration to evaluate the impact of a programme
to improve learning outcomes, with treatment and control groups, and usually involving
baseline, mid-line and end-line.
The majority of the assessments in the region (38, or 66 per cent) define their purpose at
system-level monitoring. Ten assessments (17 per cent) aim at evaluating programmes. Nine
assessments (16 per cent) measure learning outcomes for a system-level diagnostic purpose. One
assessment, NASMLA in Kenya, has a dual purpose of system-level diagnostic and monitoring
(see Table 5).
Table 5. Purpose of the assessments from stock-taking
Assessment purpose
Number of assessments
System-level monitoring
38
Programme evaluation
10
System-level diagnostic
9
System-level diagnostic and monitoring
1
Total
58
24 Details about the stock-taking framework and other methodological aspects of the stock-taking and comparative analysis are
provided in Appendix I.
11
1.2.2 Target population
The majority of the assessments in the region (52, or 90 per cent) have a grade-based target
population. An age-based population is targeted in three Uwezo assessments. There was no
information available about the target populations for another three assessments.
For the assessments with grade-based populations, the target populations range from early
primary grades to end-of-primary (see Table 6). Of the assessments, 44 target students at lowerprimary level (i.e., Grades 1, 2 or 3), 38 target Grades 4, 5 or 6, and 10 target Grades 7, 8 or 9.
Nearly half (47 per cent) target multiple grades.
Table 6. Target population in the assessments from stock-taking
Target grade
Number of assessments25
Grade 1
4
Grade 2
20
Grade 3
20
Grade 4
12
Grade 5
8
Grade 6
1826
Grade 7
4
Grade 8
2
Grade 9
4
Age-based
327
Unknown
3
The choice of the grade level for the target population depends on a number of factors. Usually a
particular grade is chosen because of recent or planned policy reform, or because it is considered
a pivotal point in children’s learning trajectories. For example, EGRA and EGMA typically target
early primary grades because, as the assessment names indicate (‘EG’ for Early Grade), they
measure the most basic foundation skills for literacy and numeracy acquisition in the early grades
(Gove and Wetterberg, 2011).
If the assessment’s purpose is programme evaluation rather than system-level diagnostic/
monitoring, the grade-based target populations are defined as only children in the treatment or
control groups.
For assessments that monitor system-level learning outcomes and report on trends over time, a
complete discussion of population-related matters is particularly important, as stakeholders must
be informed of any differences that may affect the comparability of results from one year to the
next.
25 The total number of assessments in this table is more than 58, as 27 of the 58 assessments target multiple grades.
26 12 assessments are SACMEQ studies
27 Uwezo in Kenya (6–16 years), Tanzania (6–16 years) and Uganda (7–16 years)
12
Of the assessments in our stock-taking, the documentation from SACMEQ (Paul M. Wasanga, A.
Ogle, and Wambua, 2012), and the IEA studies PIRLS and TIMSS (M.O. Martin and Mullis, 2012)
are good examples of a complete discussion of population-related issues, as are the National
Learning Assessment in Ethiopia (Ministry of Education of Ethiopia [FDRE], 2008), and NASMLA
in Kenya (P.M. Wasanga, Ogle, and Wambua, 2010).
1.2.3Sampling
In general, the assessments we reviewed employ scientific sampling procedures. The sampling
process for assessments with a system-level monitoring or diagnostic purpose is usually multistep, involving the sampling of schools, children and, sometimes, geographical units.
One exception is South Africa’s Annual National Assessment, in which all children in public and
state-subsidised independent schools are assessed. This approach may be more accurate, but it
is also more time-consuming and costly than testing a sample.
Uwezo, another exception to the general pattern, samples households instead of schools. This
may be necessary where a complete and up-to-date list of schools does not exist, but it could
make it more difficult to explore the relationships between learning outcomes and school-level
factors.28
In four assessments, children were sampled through a central body (e.g., national centre) prior
to testing. These assessments included TIMSS, PIRLS (M.O. Martin and Mullis, 2012), LARS in
Rwanda (Rwanda Education Board, 2012) and MLA in Malawi (Ministry of Education, Science
and Technology of Malawi, 2014). The success of this approach depends upon the availability of
complete and up-to-date lists of students before the test is being administered.
1.2.4 Cognitive domains
Assessment framework
An assessment framework is intended to guide test development and help interested stakeholders
understand the content and scope of the assessment. In general, a framework should support
and ensure consistency of test development and provide a common language for discussing the
assessment. An assessment framework should include:
• a definition of the constructs that are being measured;
• a discussion of skills/knowledge that are tested to measure the constructs, as well as
provision of a rationale for any omissions of skills/knowledge that one might expect to be
tested when the stated constructs are being measured;
• a discussion of any alignment of tests (e.g., to a particular grade level);
• specifications of task content (i.e., number or proportion of tasks per content area);
• specifications of task format (e.g., multiple choice, free-response);
• a discussion of scoring; and
• an outline of how the results are reported.
28 Even though Uwezo’s purpose is to monitor children’s learning at the system level – like ASER in India, on which its sampling
methodology is based – it eschews a more traditional school-based approach to sampling. This is partly because of the difficulty
of obtaining a complete and up-to-date list of schools, but also because this approach cannot yield data representative of the entire
population of school-aged children in a context where the percentages of school-aged children either not enrolled in school or not
attending school regularly are high enough to affect the representativeness of an in-school sample.
13
If the assessment’s purpose is system-level monitoring, and it reports on trends over time, then
having an assessment framework is particularly important because it ensures consistency in test
development from one assessment cycle to the next.
Comprehensive assessment frameworks were publicly available for three assessments. These
include the IEA studies PIRLS (I.V.S. Mullis, Martin, Kennedy, Trong, and Sainsbury, 2009) and
TIMSS (I.V.S. Mullis, Martin, Ruddock, O’Sullivan, and Preuschoff, 2009), and the Annual
National Assessment in South Africa (Department of Basic Education, Republic of South Africa,
2014).29 PASEC is planning a new methodological framework for the next assessment cycle.
Literacy and numeracy
One-third of the assessments in the region (19, or 33 per cent) assessed both literacy and
numeracy.30,31 Almost a third (17, or 29 per cent) focused on literacy as the only domain. Twelve
assessments (21 per cent, all of which were SACMEQ studies) measured student performance in
literacy, numeracy and health knowledge. These three combinations of domains (i.e., literacy and
numeracy; literacy only; and literacy, numeracy and health knowledge) constitute the majority of
the assessments in our stock-taking, totalling 48 (83 per cent). The rest of the assessments had
various combinations of domains (see Table 7).
29 Assessment frameworks for PIRLS, TIMSS and ANA in South Africa are publicly available. Information about the development of
an assessment framework for the National Assessment of Learning Achievement in Zambia (UNICEF Zambia Country Office, 2015)
was obtained from UNICEF CO in Zambia, but the assessment framework was not publicly available. This may also apply to other
assessments in the stock-taking.
30 Domains related to literacy are referred to differently in different assessments. The variations include ‘Reading’, ‘Mother tongue’,
‘Language’, ‘English’, or the names of particular local languages with slightly different scope of assessment. It is referred as
‘Literacy’ in this report to avoid confusion. Please see Table 24 for individual cases.
31 ‘Numeracy’ is sometimes referred as ‘mathematics’ as a domain with slightly different scope of assessment. It is referred to as
‘mathematics’ in this report to avoid confusion. Please see Table 24 for individual cases.
14
Table 7. Cognitive domains in assessments from the stock-taking
Domains
Literacy and numeracy
Literacy
Literacy, numeracy and health knowledge
Numeracy and science
Literacy, numeracy and environmental
science (Grade 4)
Literacy, numeracy, biology, chemistry and
physics (Grade 8)
Literacy, numeracy and life skills
Literacy, numeracy and life skills (Grade 5)
Literacy, numeracy and environmental
sciences (Grade 9)
Numeracy
Literacy and numeracy (Grade 5)
Literacy, numeracy and natural science
(Grade 7)
Literacy and numeracy (Grade 4)
Literacy, numeracy and science (Grade 7)
Literacy, numeracy and natural science
Literacy and one numeracy sub-task
Total
Number of
assessments
19
17
12
2
Notes
Including PIRLS studies
All are SACMEQ studies
Both are TIMSS studies
1
NLA in Ethiopia
1
Assessing Learner
Achievement in Malawi
1
1
National Assessment of
Learning Achievement in
Zambia
EGMA in Malawi
1
NSAT in Namibia
1
1
1
58
MLA in Somalia
NALA in South Africa
EGRA in Uganda
Out of the 58 assessments in the stock-taking, 23 (40 per cent) tested in multiple languages.
In this respect, South Africa is the most notable example, as its Annual National Assessment
covers English, Afrikaans and nine local languages (Department of Basic Education, Republic of
South Africa, 2014).
1.2.5 Contextual instruments
Type of contextual instruments
Almost all of the assessments collect contextual data of some kind through student, teacher and
school questionnaires. EGRA and EGMA are often administered using a contextual instrument
called the Snapshot of School Management Effectiveness (SSME). The SSME collects information
through student and teacher questionnaires, as well as through classroom observation and
classroom and school inventories. Uwezo also collects information through a similar technique
of observation and inventory. In addition, TIMSS and PIRLS include a curriculum questionnaire
completed by the national research centre in each participating country. Furthermore, PIRLS
(M.O. Martin and Mullis, 2012), EGRA in Angola (Ministry of Education of Angola, World
Bank, and Russia Education Aid for Development Programme [READ], 2011), MLA in Eritrea
(UNICEF Eritrea, n.d.) and LARS in Rwanda (Rwanda Education Board, 2012) include a parent
15
questionnaire. Data on a system level are not directly collected in the assessments, with the
exception of the curriculum questionnaire used in PIRLS and TIMSS. However, countries with an
Education Management Information System (EMIS) can derive data on a system level, and link it
to schools that were observed in the assessment.32
In assessments undertaken with limited funds, the benefit of obtaining information from a parent
questionnaire might be weighed against the risk that response rates will be low due to low
literacy skills. An option would be interviews, although these are likely to be more cost-intensive.
Experience shows that primary students are a reliable source of information about their parents
and households. However, this depends on the type of questions, the kind of data collection
instrument (questionnaire or interview), as well as the grade level and associated literacy skills
of the children.
Contextual instrumentation must be based on a sound and fully articulated theoretical framework.
Good examples for highly elaborated context frameworks are PIRLS and TIMSS. SAQMEC
and PASEC use analytical models to describe the context factors collected and the expected
relationships with achievement.
Key factors of contextual instruments
Complete contextual instruments (or documentation about its content) were obtained for the
following assessments:
• EGRA/EGMA (i.e., SSME instruments) (RTI, 2004)
• PIRLS, TIMSS (IEA, 2013b, 2013d)
• SACMEQ (Hungi, 2011; Hungi et al., 2011)
• Uwezo (Uwezo-Kenya, 2013)
• PASEC (CONFEMEN, 2010a, 2010b)
A review of these instruments and associated documentation suggests that the key factors in
contextual data collection on student, classroom and school level are:
• at the student level:
• individual factors, such as gender, age, grade level, grade repetition, health and well-being;
• family factors, such as socio-economic measures (possessions at home; books in the
home; parental literacy level, education and occupation); ethnic background and cultural
practices; language spoken at home; home resources; early learning opportunities (preschool
attendance); and family support;
• learning experiences at school (e.g., activities during instruction, teacher feedback provided
to students);
• learning experiences out of school (homework and out-of-school lessons; reading
independently in and out of school; working outside school/domestic work);
• learning time, attendance/absence;
• access to resources at school; being allowed to take books home;
• community support.
32 In order to link aggregated school data to system level data in EMIS, participating schools need to be able to be identified within
the EMIS.
16
• at the classroom level
• teacher background variables, including gender, teacher training, practice and experience
­ class size
• classroom equipment, teaching resources (e.g. availability of pedagogic materials for
students and teachers, learning material, text books, furniture)
• language of instruction, language of teacher
­ frequency of homework
• quality of instruction, teaching methods (e.g. teacher reads to learner, explaining things if not
understood, providing extra time to complete task; domain-related activities; instructional
time), classroom management, frequency and use of assessment for teaching
­ classroom climate, discipline;
• at the school level:
• input factors such as school type (public/private); school location (rural/urban); school size;
school funding, teacher-student ratio; socio-economic background; and ethnic/language
composition;
• teacher body, teacher absenteeism, teacher professional development;
• principal/head teacher background variables;
­ school management, school curriculum, assessment and evaluation;
­ language for instruction, provisions for students who do not speak the language of
instruction at home;
• school resources (e.g. library, computer rooms);
­ school facilities (e.g. condition of school building, electricity and water supply, toilets and
canteens);
• quality of instruction;
­ school climate.
The collection of contextual information is central to an assessment. Context helps define the
relationships between learning outcomes and background factors of research and policy interest.
The relationships between contextual factors and achievement – and not the learning outcomes
data alone – are essential to decision-making. The time and cost of collecting contextual
information can be minimised by ensuring that the instruments are well targeted to specific
areas of research and policy interest, and that particular questions yield response data that do
not require excessive processing.
1.2.6 Test administration
In just over half of the assessments (30, or 52 per cent), the cognitive assessment is administered
as a paper-based test in schools to groups of students, where each student completes the
assessment independently (i.e., by reading questions and recording responses on paper). Uwezo,
EGRA and EGMA are exceptions. Their cognitive assessments are one-on-one, with the test
administrator delivering the items orally, and students providing most of their answers orally.
These oral one-on-one assessments generally make use of paper-based instruments, though
EGRA and EGMA are starting to employ a method where the student refers to a paper-based
test, but the administrator records the data on a tablet-based application called TangerineTM (see
Table 8).33
33 See <http://www.tangerinecentral.org/home> for information about TangerineTM.
17
Table 8. Test administration methods of assessments from the stock-taking
Test administration methods
School-based
Group administration
Number of Assessments
assessments
30
PIRLS, prePIRLS
TIMSS
Paper-based administration
PASEC
School-based
SACMEQ
National assessments
EGMA
20
EGRA
One-on-one administration
Oral administration
Household-based
3
Uwezo
1
Pilot assessment of Grades 1, 2
and 3 in Lesotho
One-on-one administration
Oral administration
School-based
One-on-one or small group administration
Oral or tablet-based administration
Unknown
Total
4
58
A pilot assessment of Grades 1–3 in Lesotho provides an example of how tablets can be used
in oral assessments as more than mere data collection tools for the test administrators. In this
pilot, students formed a small group with one tablet per student, they interacted directly with
the tablets (i.e., the tablets ‘administered the test’ to the students), while the test administrator
monitored the groups.
These different methods for cognitive assessments suit different aims and purposes. Group
administration is most convenient if members of the target population have the skills to complete
an assessment independently, and are easily located in naturally occurring groups (e.g., in
schools). One-on-one oral administration is necessary if some students are expected to be unable
to complete an assessment independently. If one-on-one oral administration is used, a tabletbased data collection application such as TangerineTM is an effective way of reducing human
error. It can control the way the assessment is administered and restrict data input to only
valid-response values. A small group tablet-based oral administration, such as the one used in
the Lesotho pilot, offers further efficiency gains because children who would otherwise require
one-on-one administration can be tested simultaneously.
1.2.7 Data analysis
The assessments in our stock-taking use a range of different analytical techniques for data
analysis. Seven major techniques were identified:
• using item response theory (IRT) to scale cognitive data;
• establishing competency levels or benchmarks;
• conducting frequency analyses or calculating mean scores for cognitive results,
disaggregated by contextual variables of interest;
18
• conducting frequency analyses on contextual data;
• exploring relationships between cognitive performance and contextual factors via analytical
techniques;
• computing trends in cognitive performance;
• reporting international comparisons of cognitive data.
Grouping assessments by type reveals patterns of use of particular techniques (see Table 18 in
Appendix II). The last row of the table tallies the frequency of use for each technique. Those
used most frequently are: establishing competency levels or benchmarks; conducting frequency
analyses or calculating mean scores for cognitive results, disaggregated by contextual variables
of interest; and exploring relationships between cognitive performance and contextual factors
via analytical techniques. These techniques are described in subsequent sections, as is the use
of item response theory (IRT).
Use of item response theory
IRT is widely used in the design and analysis of educational assessments. Unlike classical test
theory, which is based on the assumption that all items in a test contribute equally to a student’s
performance, IRT takes into account different characteristics of test items (i.e., varying difficulty
levels, discrimination), and hence the probability of getting particular items right or wrong, given
the ability of the student taking the test (i.e., the probability that those who do well on the test
have a high level of performance, and those who do poorly have low levels of performance) (Kaplan
and Saccuzo, 1997). The different approaches account for different levels of test validity.34,35,36
A primary advantage of IRT is that scores obtained from a linked test design – where there are
multiple test forms with a certain number of common items shared across the forms – can be
placed on a common scale. In IRT, raw scores are converted to scale scores. Placing scores on a
common scale permits valid comparisons of results across different test forms, across different
grades, and over time.37 Hence, these functions offered by the use of IRT are of particular
importance for assessments with system-monitoring purpose.
Another important advantage of using IRT is that it offers a greater depth in reporting. Because
item difficulty and ability are on the same scale, it is possible to develop substantive descriptions
of the skills and knowledge required to correctly answer items of varying difficulty, and to
34 Classical test theory uses observed test scores of individuals that are composed of a ‘true score’ (raw score) an individual
would get if there were no measurement error. The measurement error, i.e. standard deviation of errors, is assumed to be a
random variable with a normal distribution. The larger the standard error of measurement, the less certain is the accuracy of the
measurement. Conversely, a small standard error of measurement indicates that an individual score is probably close to the true
score (Kaplan and Saccuzo, 1997).
35 Discrimination refers to the capacity of an item to distinguish between different levels of ability (i.e. good quality test questions
distinguish between students with the ability to answer the question correctly and those without).
36 For example, one-dimensional models consider item difficulty, two-dimensional models consider item difficulty and item
discrimination, and three-dimensional models also account for ‘guessing’ (i.e. test takers with very low levels of ability getting a
correct response).
37 If an assessment does not use a linked design and analyse data using IRT, comparisons of results are only true comparisons if the
same assessment items are administered in the same order to all children whose results are being compared. In many cases this
is not feasible, as it is often difficult to keep tests secure from one administration to the next. It is also impractical in instances
where the construct being assessed is broad, because all items that cover the construct cannot possibly be administered to each
child.
19
make statements about the skills and knowledge possessed by children with different levels of
ability. That way, consistent competency levels can be defined and used as a basis for setting
benchmarks.38
Moreover, the IRT analysis of the different components provides information on the psychometric
quality of the items, e.g., if items are well targeted in terms of difficulty, and can distinguish
between students with different levels of ability. With the use of IRT analyses, items of poor
psychometric quality can be identified and discarded or adjusted as required.
Of the assessments we examined, IRT is used in the international/regional assessments PIRLS,
TIMSS, PASEC and SACMEQ, as well as in five of the national assessments. These assessments
have a common purpose: system-level monitoring. However, in a large number of national
assessments (seven; the information was not available in four cases) that also aim at systemlevel monitoring, IRT is not used (see Table 9).39
Table 9. National assessments and use of IRT in data analysis
National assessment
MLA
National Learning Assessment (NLA)
Country
Eritrea
Ethiopia
NASMLA
Kenya
LNAEP
Assessment of Grades 1, 2 and 3 in Lesotho
Lesotho
Lesotho
Assessing Learner Achievement
MLA
National Assessment
NSAT
LARS
Malawi
Malawi
Mozambique
Namibia
Rwanda
MLA
Annual National Assessment
NALA
NAPE
NALA
ZELA
Somalia
South Africa
South Africa
Uganda
Zambia
Zimbabwe
IRT used
Unknown
Unknown
Unknown
Unknown
In Uwezo and in EGRA/EGMA implementations that aim at system-level monitoring, IRT is not
used (see Table 4). This finding shows that despite the advantages of effectively monitoring
system performance across contexts and over time, IRT is not extensively applied. One reason
for this may be the limited capacity for psychometric analysis at the country level.40
38 For some discussion of how results of IRT analysis can be reported and an example of a metric that uses both quantitative and
substantive information about children’s performance, see <http://www.acer.edu.au/files/Described_Proficiency_Scales_and_
Learning_Metrics.pdf>.
39 A more detailed table showing the analytical techniques used in all assessments from the stock-taking is presented in Appendix II.
40 This was one of the findings of the country case studies conducted in this study (see Appendix III).
20
Competency levels/benchmarks
In more than half of the assessments (32, or 55 per cent), results are presented with reference to
competency levels or benchmarks. In general, competency levels or benchmarks are established
by describing the specific skills required to provide correct responses to each test item. Then,
test items are placed into groups of items so that the items in each group have similar difficulties
and share a common ‘theme’ relating to the underpinning competencies required to provide the
correct response. Naming and defining the ‘themes’ identifies competency levels or benchmarks
(Hungi et al., 2010).
For example, the National Assessments System for Monitoring Learning Outcomes (NASMLA)
undertaken in 2010 in Kenya used this analytical technique to assess Grade 3 students in literacy
and numeracy. Four competency levels were identified in literacy (The National Assessment
Centre, 2010) (see Table 10).
Table 10. Descriptions and attainment of competency levels in literacy: NASMLA in
Kenya 2010
Competency Description
level
Percentage of
students who attained
the level
Level 1
Pre-reading: Matches words and pictures involving
concrete concepts and everyday objects.
Level 2
Emergent reading: Spells correctly simple everyday
words and recognises missing letters in such words.
Uses familiar words to complete simple everyday
sentences.
46.1
Level 3
Basic reading: Uses correct punctuation in simple
sentences. Infers meaning from short passages, and
interprets meaning by matching words and phrases.
Identifies the main themes of a picture.
36.7
Level 4
Reading for meaning: Links and interprets information
located in various part of a short passage.
Understands and interprets meaning of a picture and
writes short sentences to describe the theme.
11.0
6.2
Source: Adapted from Monitoring of Learner Achievement for Class 3 in Literacy and Numeracy in Kenya: Summary of results and
recommendations (The National Assessment Centre, 2010, p. 23).
Students’ attainment was analysed as follows: Slightly less than half of the pupils (47.7 per
cent) attained the desirable Levels 3 and 4 of competency in literacy. However, most Grade 3
pupils (46.1 per cent) demonstrated emergent reading ability, which is congruent with the Grade
2 level (The National Assessment Centre, 2010, p. 23).
As this example from Kenya demonstrates, competency levels can provide a more concrete
understanding of what students are actually able to do than can the insights obtained from merely
presenting test scores. Competency levels can also suggest instructional strategies relevant to
students who are learning at each level of competence. Such descriptions would be of great
21
assistance for the preparation of textbooks, the design of teacher in-service training programmes,
and the development of general classroom teaching strategies. All of these activities require a
sound knowledge of the skills already acquired and the higher order skills that must be mastered
in order to move to the next stage of learning (Hungi et al., 2010).
Frequency analysis and mean scores
All of the assessments we studied – other than those without documentation on data analysis
– use frequency analysis. These total 50 assessments (86 per cent). Frequency analyses are
conducted for performance data and mean scores are calculated. These analyses are usually
undertaken on data disaggregated by key contextual variables, such as gender, grade and
administrative location.
For example, the Annual National Assessment in South Africa is a system-level monitoring
assessment targeting students in Grades 1–6 and 9. It undertakes mean score analysis to
investigate the difference in learning achievement between boys and girls (Department of Basic
Education, Republic of South Africa, 2014). The mean percentage marks of Grade 3 students in
mathematics, calculated by gender and province, show that girls performed better than boys in
all provinces (see Figure 3).
Conducting frequency analyses and calculating mean scores are simple procedures, yet they can
yield information that is highly relevant to policy development in the implementing countries.
Figure 3. Mean percentage marks in Grade 3 Mathematics by gender and province –
Annual National Assessment in Kenya
100%
90%
80%
70%
60%
Girls
50%
Boys
40%
30%
20%
10%
22
To
tal
pe
ter
n
We
s
rth
No
ern
Source: Department of Basic Education Republic of South Africa, 2014, p. 85
Ca
We
s
t
pe
Ca
ga
rth
No
Mp
um
a
lan
po
p
o
l
Um
ata
-N
n
ute
Kw
aZ
ulu
Ga
Sta
te
Fre
e
Ea
s
ter
nC
ap
e
0%
Relationship between cognitive performance and contextual factors
Relationships between cognitive performance and contextual factors are explored in 34
assessments (59 per cent). Analytical techniques used were correlational analysis, regression
analysis and multi-level modelling.
EGRA and EGMA in Zambia, implemented in 2011 with support from USAID, provide an example
of this approach to data analysis. These reading and mathematics assessments were administered
to Grade 2 and Grade 3 students in the Bemba-speaking regions as a pilot study for system-level
diagnostic assessment. The final report of the assessments discusses which factors best predict
student performance in reading and mathematics. Using multiple regression models, the report
shows that five main factors contributed to students’ performance in school: socio-economic
status; having attended preschool; starting school at the expected age; reading independently in
and out of school; and receiving corrective feedback from teachers (Collins et al., 2012).
Conducting analyses such as these helps to ensure that cognitive results are not misinterpreted,
as they can be when they are presented without any context. However, the results should not
be taken to mean that the relationships are necessarily causal.
1.2.8 Reporting and dissemination
In any assessment programme, availability of quality information and data that address a diverse
audience is the key to successful dissemination. Public availability of assessment results and
data are important. They allow a wide range of stakeholders to instigate change within the
system to improve student performance. The availability of a fully documented database can
also, in turn, inform the work of independent researchers.
In the assessments we reviewed, results are publicly available for 41 (71 per cent). Approximately
half (22, or 54 per cent) provide additional dissemination products, including results summaries,
press releases and policy briefs. Large-scale international and regional assessments, such as
TIMSS, PIRLS, SACMEQ and Uwezo, provide ample dissemination products to reach a diverse
audience.
Results are not publicly available for 15 (26 per cent) of the assessments we reviewed. For
two national assessments where results reports are not publicly available, other means of
disseminating them were identified. MLA in Eritrea held workshops at national and sub-national
levels (UNICEF Eritrea, n.d.). In Zambia, results of the Grade 5 national assessment in 2008
were disseminated at provincial level. Also, remedial materials were developed for the areas that
were found to be challenging for teachers and learners based on the test item analysis (UNICEF
Zambia Country Office, 2015).
23
In addition to reports of the results, considerable effort was made to obtain full datasets from
the assessments during our review. However, data from only four regional and international
assessments that were implemented in seven ESA countries were available for this study within
our time constraints: Uwezo (Kenya, Tanzania, Uganda), PASEC (Burundi, Comoros), TIMSS
(Botswana) and prePIRLS (Botswana, South Africa).41 Data management and data cleaning
procedures must be undertaken before assessment data can be analysed, reported and eventually
released to the public. The more countries that are involved in an assessment – for example, in
international and regional assessments such as PIRLS, TIMSS, PASEC and SACMEQ – the longer
it takes for the data to be cleaned at the national and regional/international level and scaled and
analysed. Both PIRLS 2011 and TIMSS 2011 studies took approximately two years and three
months between main data collection and the release of international datasets (IEA, 2013a,
2013c). Hence, the most recent available datasets for our analysis are for Uwezo 2012, PASEC
2008–2009 and PIRLS/TIMSS 2011 (see Chapter 2). 41 The authors thank UNICEF Headquarters, UNICEF ESARO and COs for their support in requesting access to data from SACMEQ
and national assessments. It is acknowledged that in instances where national data were requested from national education
ministries or national examination bodies, country-level processes for approving the use of national assessment data for secondary
analysis may have required more time than we had available for data analysis. This was the case for national data for South Africa
where access was granted outside the time available, and these data were not included.
24
25
26
2.Literacy and numeracy
in primary education in
the ESA region: Students
experiencing LLOs and
trends over time
Average test scores for literacy and numeracy in international and regional assessments undertaken
in the ESA region were generally low, with a considerable proportion of students not achieving
basic skills in reading and mathematics. Results from SACMEQ III (2007) show wide disparities
in basic reading and mathematics skills by the end of primary education (Grade 6). In 3 of the 12
participating countries in the ESA region (Kenya, Tanzania and Swaziland) between 80 per cent
and 93 per cent of students achieved the minimum reading level in SACMEQ. In six countries
(Botswana, Zimbabwe, Namibia, Mozambique, Uganda and South Africa), between 50 per cent
and 80 per cent of students achieved the minimum level. In Lesotho, 48 per cent of students
in Grade 6 achieved basic reading skills; in Zambia and Malawi, only 27 per cent of students
reached this level. In mathematics, the proportion of primary students reaching basic skills was
considerably lower, with less than 50 per cent of students in Grade 6 reaching the minimum
level in three-quarters of the countries. In the remaining quarter of participating countries (again
Kenya, Tanzania and Swaziland) between 56 per cent and 62 per cent of students learned basic
mathematics skills (UNESCO, 2014, p. 35).42
Characteristics of low-performing students and trends in student performance over time in
literacy and numeracy in primary education are the focus of this chapter. Specific analyses
draw upon data from four different assessments in the region (as discussed in Chapter 1):
Uwezo (Kenya, Tanzania and Uganda); PASEC (Burundi and Comoros); prePIRLS (South Africa
and Botswana); and TIMSS (Botswana).43 The four assessments cover 7 of the 21 ESA countries.
Trends in literacy and numeracy performance were analysed for three countries – Kenya, Tanzania
and Uganda – where the same assessment, Uwezo, with the same key features (assessment
framework, design, target population and conditions of administration) was implemented more
than once.44 Data from the regional assessment SACMEQ, in which 12 ESA countries participated
at least twice, and data from national assessments, were not available for this study.45 The
limited available data makes it difficult to draw conclusions about the characteristics of low-
42 Source: IIEP/Pôle de Dakar Indicator Database (UNESCO, 2014, p. 35); (Hungi et al., 2010)
43 For details about the datasets available, see Appendix I.
44 Any comparison of results between different assessments—or between different cycles of the same assessment that don’t have
the same key features across the different cycles—requires sophisticated linking procedures and analyses that are beyond the
scope of this report.
45 The authors thank UNICEF Headquarters, UNICEF ESARO, and UNICEF COs for their support in requesting access to data from
SACMEQ and national assessments during the time of this study. Findings from SACMEQ and national assessments are referred to
in the discussion where reports including analysis of contextual data were obtained.
27
performing students and trends in performance over time for the region. Given these constraints,
we drew upon findings from a broad variety of reports from the ESA region, including SACMEQ
and national assessments, to supplement our analyses. The findings that were reported provide
valuable insights into the main characteristics of students experiencing LLOs in literacy and
numeracy, the nature of associations between context factors and student performance, and
changes in student performance over time.
2.1Characteristics of students experiencing LLOs
in literacy and numeracy in primary education in
the ESA region
Equipping all students with basic literacy and numeracy skills, and minimizing the number of
low-performing students in these domains, are fundamental goals of education systems. In order
to quantify the number of students at different levels of performance and to monitor progress
over time, competency levels must be defined and benchmarks set. Understanding the factors
associated with low-performing students is critical for the development of targeted education
policies.
As outlined in the previous chapter, 32 out of the 58 assessments that we reviewed define
competency levels or benchmarks. However, there are no common metrics in literacy and
numeracy across the different assessments, and different benchmarks are used to define ‘limited
learning outcomes’. Hence, for our study limited learning outcomes are based on the benchmarks
used for literacy and numeracy in the different assessments analysed in this report: PASEC,
Uwezo, TIMSS and prePIRLS. The criteria used to set these benchmarks are conceptually quite
different.46
For PASEC (Burundi and Comoros), student achievement scores in reading (French, Kirundi)
and mathematics are categorised in three levels.47 Students at Level 3, the highest level, have
acquired a basic level of knowledge (CONFEMEN, 2010b, p. 93). At the other end, at Level 1,
students are considered to be close to failing. This group is categorised as experiencing LLOs.48
Following Uwezo’s approach, in which tests are aligned with the national Grade 2 curriculum in
the three participating countries (Kenya, Tanzania and Uganda), all children attending Grade 3 are
expected to have achieved the highest level in each domain (‘story’ for literacy and ‘multiplication/
division’ for numeracy). Hence, children enrolled in Grade 3 and above who have not achieved the
highest level of performance in English, Swahili and numeracy are considered to be experiencing
LLOs.
46 Details about the definition and identification of students with limited learning outcomes in the different assessments are
presented in Appendix I.
47 Grade 2 students from Burundi were also assessed in Kirundi.
48 Swahili was only assessed in Kenya and Tanzania.
28
For TIMSS (Botswana) and PIRLS (or prePIRLS) (Botswana and South Africa) four international
benchmarks are defined, ranging from ‘advanced’ to ‘low’ (I. Mullis, Martin, Foy, and Arora,
2012; I. Mullis, Martin, Foy, and Drucker, 2012).49,50 Students reaching the ‘Low International
Benchmark’ for Grade 4 show some basic mathematical knowledge and reading skills. For the
purposes of our analysis, students who have not achieved the Low International Benchmark are
considered to have experienced LLOs.
We listed in a table the percentage of students experiencing LLOs, organised by assessment,
country, grade and domain, as well as the criteria used for defining the LLOs for the different
assessments (see Table 11).
Due to the conceptually different criteria or benchmarks used, the percentage of students
experiencing LLOs varies considerably across the assessments. In Burundi and Comoros (PASEC),
approximately one in five students experienced LLOs, with a similar percentage for literacy and
numeracy domains across Grade 2 and Grade 5. In Kenya, Tanzania and Uganda (Uwezo), the
percentage of primary school age students experiencing LLOs differed between the domains.
For mathematics, approximately one-third of the students experienced LLOs. For literacy, the
percentages varied across the Uwezo countries. In Kenya, around one in five students performed
low in English or Swahili. In Tanzania, every second student of primary school age experienced
LLOs in English, and approximately every third student in Swahili. In Uganda, 39 per cent of
primary school age students experienced LLOs in English. In Botswana (TIMSS), 40 per cent of
Grade 6 students experienced LLOs in mathematics. Around one in four students in Botswana
and South Africa showed LLOs in reading (prePIRLS).51
49 Botswana participated in TIMSS 2011 with Grade 6 students, using Grade 4 TIMSS assessment. If it was expected that a
country’s Grade 4 students would find TIMSS assessment too difficult, IEA encouraged the country to test children in a higher
grade.
50 PrePIRLS was chosen over the traditional PIRLS dataset as it is better targeted towards the achievement of students from
participating countries for the region. Botswana and South Africa participated in prePIRLS with Grade 4 students in 2011.
51 It is worth noting that Grade 6 students in Botswana were assessed using the TIMSS test material targeted to Grade 4; the
proportion of low performing students in mathematics would presumably be higher if Grade 4 students were tested with the Grade
4 assessment material. For reading, Grade 4 students in Botswana and South Africa were assessed with the ‘easier’ or better
targeted prePIRLS test materials; presumably the proportion of low-performing students would be higher if measured with the
standard PIRLS tests.
29
Table 11. Proportions of students defined as experiencing limited learning outcomes by
assessment, country, grade and domain
Assessment
(Year in
brackets)
Criteria for
Country
limited learning
outcomes
Grade
Mathematics
literacy
Reading
literacy
Burundi
Grade 2
18%
19%
(French),
18% (Kirundi)
Burundi
Grade 5
23%
22%
Comoros
Grade 2
21%
21%
Comoros
Grade 5
24%
20%
Students
Kenya
enrolled in
Grade 3 and
above who
Tanzania
could not
achieve the
highest level of
Uganda
performance
Primary school
age
39%
23%
(English),
21% (Swahili)
Primary school
age
31%
50%
(English),
32% (Swahili)
Primary school
age
32%
39%
TIMSS
(2011)
Botswana
Students
scoring
below ‘low
achievement
threshold’
proficiency
standard (score
of less than
400)
Grade 6
40%
Pre-PIRLS
(2011)
Botswana
Students
scoring
South
below ‘low
Africa
achievement
threshold’
proficiency
standard (score
of less than
400)
Grade 4
23%
Grade 4
29%
PASEC
Students with
(2008/2009) a test score of
less than 25
out of 100
UWEZO
(2012)
30
2.1.1 Individual and family characteristics of students with LLOs in
literacy and numeracy
Individual and family characteristics of students found to be important in association with LLOs
are: gender; age; language spoken at home; a range of socio-economic factors; learning activities
prior to attending school; engagement in reading lessons; and whether the student attends
lessons out of school. Overview tables with detailed results for each of the factors observed in
the analysis per country, grade and domain are presented in Appendix IV.
Gender
In general, the assessments showed that males were more likely to experience LLOs in literacy
than females. This pattern was found in Burundi (PASEC, both French and Kirundi), Kenya (Uwezo,
both English and Swahili), South Africa and Botswana (prePIRLS, English). The most extreme
example was in Botswana, where males were almost three times more likely than females to be
experiencing LLOs. The exception was Comoros, where females from Grade 5 were slightly more
likely to be experiencing LLOs in French than males (1.1 times more likely). It is not clear why
the direction of the gender difference was reversed in Comoros, but we assume other factors
contributed to it. For example, females in Comoros with illiterate parents were noticeably more
likely to experience LLOs than males.
The findings for gender and its relationship with literacy mirror previous findings from the region.
Girls were reported to outperform boys in reading literacy in Botswana (Monyaku, 2012); South
Africa (Moloi and Chetty, 2010); Zimbabwe (ACER and ZIMSEC, 2015); and Eritrea (UNICEF
Eritrea, n.d.). In contrast, an EGRA study in Ethiopia showed that Ethiopian boys had higher
levels of early reading scores than girls. However, there was an interactional regional effect. Rural
boys outperformed rural girls on almost all tasks, but in urban schools, the opposite effect was
the case, with girls outperforming boys (RTI, 2010, p. 37).
For mathematics, girls were found to have greater representation in the LLO group in Burundi
and Comoros, but less in Kenya and Botswana. Again, the differences across countries are
likely attributable to other factors. For example, in Botswana, a greater percentage of students
considered relatively old for their grade (aged above 14 years) were male compared to other age
groups.
The different directions of the gender differences found in the assessments are consistent
with the findings in prior studies from the region. Boys were reported to have outperformed
girls in mathematics in Kenya, Uganda and in provinces from Somalia (Report on Monitoring
Learning Achievements (MLA) in Grade 4 in Puntland and Somaliland, 2012; Uganda National
Examinations Board, 2010; Paul M. Wasanga et al., 2012). However, the opposite pattern was
found in Zimbabwe and South Africa (ACER and ZIMSEC, 2015; Moloi and Chetty, 2010).
Age
Age – relative to school entry and grade – is an important factor associated with student
performance in the ESA region. However, the relationship between age and performance is
complex, and may be influenced by other factors, such whether school entry occurs at the
official starting age, the student’s cognitive development at the time of school entry, prior learning
opportunities that affect progression through the grades, as well as instructional practices in
31
response to student diversity.52 A literature review conducted for a study by Hungi et al. (2014)
found different effects of age – relative to the grade where the students are at – for developed
countries, where ‘older’ students in class generally outperformed their ‘younger’ colleagues, and
developing countries, where most studies, especially from Africa, show that ‘younger’ students
perform better than ‘older’ students (Hungi et al., 2014, p. 249).53
The assessments we reviewed showed that across the region younger students consistently
performed better.54 For example, Grade 6 students in Botswana who were 12 years or under
(approximately one-fifth of the population) were almost three times less likely to be experiencing
LLOs in mathematics than students aged over 12 years of age. This is well supported by the
literature in the region. A study by Kunje (cited in Hungi et al., 2014) showed that younger
students in Grade 7 in Malawi outperformed their older colleagues in English literacy, Chichewa
(a local language) and mathematics. SACMEQ III in 2007 showed that in 12 out of 15 countries,
younger students performed better than older students; in nine of those countries, younger
students performed better in reading as well as in mathematics (Hungi et al., 2014). Additionally,
data from 740,000 students from the 2010 Kenya Certificate of Primary Education examination
showed that younger students performed better than older students, according to Keith et al.
(cited in Hungi et al., 2014).
Students relatively older than the average class age were over-represented in the LLOs groups
for Botswana (prePIRLS, TIMSS), South Africa (prePIRLS) and Grade 5 students in Comoros and
Burundi (PASEC). In contrast, Grade 2 students in Burundi who were relatively older, and older
children from Kenya, were less likely to be experiencing limited learning. These mixed findings
may be attributable to the different year-levels examined. Students relatively older in the latter
years of primary school would have been more likely to have repeated a grade than students in
the earlier years. At earlier years, age differences are more likely due to different ages of school
commencement. For example, of the students from Grade 5 in Comoros who were considered
relatively older, two-thirds had repeated a grade at some stage. These students were considerably
more likely to be experiencing LLOs than those who did not repeat (and therefore started school
at a later age).
The 2012 Monitoring Learning Achievement project (MLA) for Malawi found that those learners
who had repeated at least one grade scored significantly lower than those who had never
repeated a grade. The achievement levels of those who repeated were generally low and were
more evident in higher than lower grades (Ministry of Education, Science and Technology of
Malawi, 2014). These studies suggest that while grade repetition is not necessarily the cause of
poor performance it does indicate that repeating a grade may not help low-performing students,
regardless of age (Njora Hungi et al., 2014).
52 For example, as reported in Hungi, Ngware, and Abuya (2014), reasons given for early school entry by some parents in Kenya is
‘... a hope, that, if the children do not do well, they can always repeat because they have a year or two to spare compared to their
classmates’ (Hungi et al., 2014, p. 256).
53 Hungi et al. (2014) investigated the optimal age with the greatest positive impact on literacy achievement for Grade 6 students
from low-income families across six major slums in Kenya. For this study, a sample of 7041 Grade 6 students from 226 schools
across six major urban slums in Kenya was drawn (Hungi et al., 2014, p. 247).
54 The criteria and relative proportions in each age group varied across countries and datasets. In PASEC for Burundi and Comoros
this was defined as 5 years old or below for Grade 2 students and 8 years or below for Grade 5 students. For prePIRLS for South
Africa and Botswana (also for TIMSS) this was defined as 12 years or less. For Uwezo for Kenya, Tanzania and Uganda this was
defined as children who were aged between 6 and 9 years. From the data we are unable to determine whether younger students
began school at an earlier age or whether a high proportion of other students were late entrants to schooling or had repeated a
grade.
32
Language spoken at home
The language spoken at home, or rather the degree of alignment between the language spoken at
home and the language of instruction, has a strong impact on learning outcomes. In multilingual
environments, such as the ESA region, options for multiple language instruction are evident,
which can pose complex challenges to education policy and management (Heugh, Bogale,
Benson, and Yohannes, 2006). Research on first language instruction shows that children benefit
from mother-tongue instruction for their cognitive development in general, and early literacy
acquisition in particular (Ball, 2010; Bialystok, 2001; Cummins, 2000; Heugh et al., 2006;
RTI, 2010; UNESCO, 2010).55 However, the alignment between language spoken at home and
language of instruction is just one factor affecting student achievement. Other interacting factors
are socio-economic status; the overall quality of language/reading instruction and instruction in
general; provision and use of language instruction materials/books; linguistic complexity; the
teacher’s proficiency in the language of instruction; teacher training; and the school environment.
Given these complex interactions and particular country contexts, further research is needed on
language of instruction policies and practices (RTI, 2010).
The language spoken at home was captured for Burundi, Comoros (both PASEC), South Africa
(prePIRLS) and Botswana (TIMSS and prePIRLS). In most instances across the assessments,
speaking the test language at home was an advantage for children. In Comoros, where students
were assessed in French, 97 per cent of all Grade 2 students spoke Shikomori at home (96 per
cent of Grade 5 students), while only 3 per cent spoke French at home (4 per cent of Grade
5 students). Those students who spoke French at home were less likely to experience LLOs
in both French and also in mathematics. In Burundi, 95 per cent of students at Grade 2 level
speak Kirundi at home (which is the teaching language until Grade 4), but were no more likely
to experience LLOs for this language; however, they were over-represented in the LLOs group
for French language assessment. For South Africa in prePIRLS, the vast majority (91 per cent)
of students spoke the language of the assessment at home at least sometimes.56 For Botswana,
in comparison, approximately three in four students spoke the language of assessment (English)
at home (74 per cent in prePIRLS, 78 per cent in TIMSS). Those who spoke the test language
at home were less likely to be experiencing LLOs in South Africa, with mixed findings found
for Botswana (a difference was found in TIMSS but not for prePIRLS). It is not apparent why
the effect of test language was not consistent across studies. However, the data does suggest
that any relationship between speaking the language of test at home and performance is likely
moderated by the influence of socio-economic factors. Students from both South Africa and
Botswana who always (or almost always) spoke the language of test at home were more likely
to have greater home resources.
55 A child’s first language is also often referred to as mother tongue and as the language spoken at home in assessments.
56 In South Africa, the prePIRLS assessment was administered in the 11 official languages, and more than 90 per cent of the
population were administered the test in a language they speak at least sometimes at home. Information from the South African
PIRLS 2011 national report indicates that less than 10 per cent of the population mainly speak English at home, despite English
being the language of instruction for almost 80 per cent of the population at Grade 4 (Howie et al., 2012).
33
Other studies within the ESA region show similar results. Evidence from PASEC and SACMEQ
show a strong link between home language and the language of instruction in influencing test
scores (Fehrler and Michaelowa, 2009, cited in UNESCO, 2010, p. 154; Garrouste, 2011). While
this is more commonly seen as a factor in literacy assessments, evidence from Namibia using
SACMEQ results suggests that low language skills also are a large contributor to low performance
levels in mathematics (Garrouste, 2011, p. 231). Students from Zimbabwe’s ZELA project who
spoke English at home had greater performance on the English and mathematics tests than
students who spoke other languages at home (the mathematics test was also administered
in English).57 Very few students (3 per cent) spoke English at home, as opposed to the main
language spoken at home, which was Shona (70 per cent) (ACER and ZIMSEC, 2015, p. 29).
Socio-economic factors
The socio-economic status of students is a strong predictor of achievement. The relationship
between socio-economic factors and achievement exists consistently across the different
assessments. Our analyses support this finding: students from lower socio-economic backgrounds
were more likely to experience LLOs across all countries examined in both literacy and numeracy.
This relationship was found across all assessments examined, even when the assessments (and
the countries within the assessments) used different measures of socio-economic status.58
Parental education and literacy was also consistently found to be associated with LLOs. The
mothers of students from Kenya, Tanzania, Uganda, South Africa and Botswana experiencing
LLOs were more likely to have no formal education or lower levels of education (the same pattern
was found for fathers for the latter two countries).59 Similarly in Burundi and Comoros, the
most common pattern was that students identified with LLOs were more likely to have illiterate
mothers and fathers.
Other socio-economic measures we examined showed that the socio-economic background of
students had an effect on their school performance. A number of indicators for home possessions
were identified, such as housing materials. In Kenya and Tanzania, students were more likely to
experience LLOs (literacy and mathematics; Uwezo) if they indicated that the walls of their
houses were made of the least expensive material of the possible options (mud) and less likely to
experience LLOs if the walls were made of the most expensive materials of the options (stone/
brick). This is not to suggest that there is a direct link between housing material and student
learning but rather using housing material as a proxy for socio-economic status supports the
finding that student performance is affected by home background.
57 The ZELA English test was administered in the context of the national policy that children in Zimbabwe learn English as a subject
from Grade 1 (ACER and ZIMSEC, 2015, p. 36).
58 The assessments analysed as part of this study did not have data related to nutrition, child health or other related child
development factors, which may constitute an area of further investigation with regard to student performance.
59 Uwezo only collected data on maternal education for Kenya, Tanzania and Uganda, whereas TIMSS and prePIRLS collected data for
both parents.
34
The assessments we analysed all included measures of household possessions. Home possession
indices for Comoros and Burundi (PASEC) and for Botswana (TIMSS and prePIRLS) and South
Africa (prePIRLS) in general showed that students experiencing LLOs, regardless of the domain,
were more likely to have fewer household resources.60,61 In general, across all assessments,
students experiencing LLOs were less likely, compared to their peers, to have a range of household
possessions (e.g., fridge, TV, phone, books) basic facilities (access to electricity or clean water)
and individual possessions such as children’s books.
Students experiencing LLOs in Comoros were also found to have more outside work activities
than other students, including farm, household and retail work. These students were also more
likely to indicate that their work hindered their ability to study at home, attend school and
concentrate at school. The same association between non-school-related work and students
was not found for students from Burundi. A similar association was found in ZELA, where the
amount of time that students spent working was negatively associated with achievement data
(ACER and ZIMSEC, 2015).
Learning activities prior to attending school
Analysing data from Botswana and South Africa (prePIRLS and TIMSS) we found that students
who attended preschool (46 per cent of Grade 4 students in Botswana and 83 per cent in South
Africa) were less likely to experience LLOs than students who did not.62 This is a well-supported
finding. Other regional studies from Zimbabwe, Zambia and Malawi found a relationship between
preschool attendance and achievement levels (ACER and ZIMSEC, 2015; Collins et al., 2012;
Ministry of Education, Science and Technology of Malawi, 2014). However, it is important to note
that the relationship is not necessarily causal. For example, students who attended preschool in
Botswana and South Africa were also more likely to have greater home resources for learning and
higher levels of parental education.
Parents of students in Botswana and South Africa were also asked about their children’s exposure
to activities prior to attending school related to reading (prePIRLS) and mathematics (TIMSS),
as well as their level of competency in these domains once they started school. Early activities
included reading books, telling stories, singing songs, playing word games, writing letters or
words, reading aloud signals and labels. Examples of early numeracy activities are counting
different objects, playing games involving shapes, playing with building blocks or construction
toys, or playing board games or card games. Students experiencing LLOs were less likely to have
had exposure to such activities before attending school and were rated as having lower levels of
competency at commencement. Again, the socio-economic status of the family was found to be
associated with these two measures. This suggests that families with greater home resources
are more likely to engage in learning activities with their children, which, in turn, is likely to
increase the capabilities of students when commencing school.
60 This index was based on the following home possessions and basic facilities: electricity, a television, a telephone, a fridge, gas
heating, a video recorder, a computer and a car.
61 This index includes number of books in the home, number of children’s books in the home, number of home study supports,
highest parental education level, highest parental occupation level.
62 45 per cent of Grade 6 students in Botswana assessed in TIMSS attended preschool.
35
Engagement in reading lessons and out-of-school lessons
Students from South Africa and Botswana (prePIRLS) were presented with a series of statements
that probed their engagement with reading lessons.63 Their responses were categorised as
‘Engaged’, ‘Somewhat engaged’ or ‘Not engaged’. Students who were ‘Engaged’ were far less
likely be experiencing LLOs. Those who were ‘Somewhat engaged’ or ‘Not engaged’ were much
more likely to be experiencing LLOs. In South Africa, students who fell into the latter category
were almost three times more likely to be experiencing LLOs. It is important to note that the
relationship between engagement and performance is likely to be reciprocal: more engaged
students are more likely to perform better and higher performing students are more likely to be
more engaged.
The indicator of home resources for learning was lower for those students who experienced LLOs
and were categorised as ‘Not Engaged’, suggesting that socio-economic status may influence
engagement. Howie and colleagues emphasize the need for schools to provide students with
a ‘variety of stimulating, developmentally appropriate reading materials aligned to teaching
practices that encourage active learning on the part of learners’, in order to help them engage
with their reading (Howie et al., 2012, p. 109).
Kenyan students who were found to be experiencing LLOs (English, Swahili or mathematics)
were less likely to have received extra lessons or tuition. Much of this may be explained by other
demographic factors. For instance, students who received extra lessons or tuition were more
likely to have attended private schools, have a higher number of household possessions and have
greater access to basic facilities.64 Parents in Kenya, as well as Mauritius, have been reported to
be among the highest users of extra tuition outside school (Paviot et al., 2008). However, it may
be that the relative costs of such services mean they aren’t often used by families with fewer
resources, regardless of academic need (Buchman, 2000).
2.1.2 School-level characteristics of students with LLOs in literacy
and numeracy in ESAR
School-level characteristics that were observed to be factors related to LLOs are school resources,
school type and school location. Overview tables with the detailed results for each of the factors
observed in the analysis per country, grade and domain are presented in Appendix IV.
School resources
School resources captured in PASEC, prePIRLS and TIMSS included in this analysis cover
electricity, drinking water facilities, toilets, a school library, and a computer room or computer
for instruction. The proportion of students attending schools where such facilities are available
varies broadly. For example, a maximum of 27 per cent of Grade 5 students in Comoros attended
a school with electricity, as opposed to 7 per cent in Burundi. Students in Comoros also had
more access to drinking water in school (71 per cent versus 40 per cent of Grade 5 students in
63 Items that form the scale include ‘I like what I read about in school’; ‘My teacher gives me interesting things to read’; ‘I know
what my teacher expects me to do’; ‘I think of things not related to the lesson’; ‘My teacher is easy to understand’; ‘I am
interested in what my teacher says’; ‘My teacher gives me interesting things to do’.
64 The available data for this study do not provide information about the quality of the extra lessons and their direct impact on
learning outcomes.
36
Burundi). Toilets were available for at least three-quarters of students in Comoros, and for nearly
all students (92 per cent) in Burundi. A school library was available for a maximum of 12 per cent
of students in Grade 5 in Comoros and 3 per cent of Grade 2 students in Burundi. Approximately
half of the students participating in prePIRLS and TIMSS in South Africa and Botswana were in
schools with a library. The biggest difference was found for computer resources: while 1 per cent
of students in Comoros and Burundi were in schools with a computer room, between 48 per cent
of Grade 4 students in South Africa and 70 per cent of Grade 6 students in Botswana attended
a school with computers for instruction.
Students who attended schools with access to electricity and drinking water facilities in Comoros
and Burundi (PASEC) were less likely to experience LLOs than students attending schools without
such facilities. The ZELA study in Zimbabwe also found adequate water and electricity resources
to be strongly associated with student achievement even once other factors such as home
resources of the student are taken into account (ACER and ZIMSEC, 2015).65
In Botswana (TIMSS, prePIRLS) and South Africa (prePIRLS), smaller percentages of students
in schools with library facilities than those without experienced LLOs. This pattern was not
observed in Comoros and Burundi (PASEC), where far fewer schools had access to this resource.
Similar results were observed with access to school computers. Schools in Botswana and South
Africa with computers for instruction tended to have fewer students experiencing LLOs, whereas
the proportion of schools in Comoros and Burundi with computer rooms was negligible.
Majgaard and Mingat (2012) provide a comprehensive overview of school inputs that contribute
to learning achievement in primary schools in low-income sub-Saharan African countries. In
their report, the authors combined test scores from three international learning assessment
programmes – SACMEQ, PASEC and MLA surveys – to create a comparable Africa Student
Learning Index (ASLI) for the region. Their major findings at the school level on how learning
outcomes can be improved include observable characteristics such as the quality of school
buildings and the availability of libraries, although the authors note that it is likely that resources
by themselves do not necessarily improve learning. Pedagogy plays a large role. Studies that we
analysed show that school resources such as clean water, adequate sanitation and access to
suitable learning and reading materials, such as the provision of libraries, are associated with
positive student learning outcomes.
School type and school location
Data from Kenya, Tanzania and Uganda (Uwezo) indicate an association between school type
(public versus private) and student outcomes. Students attending public schools were more likely
to be experiencing LLOs than those attending private schools.
In Botswana (TIMSS, prePIRLS) and South Africa (prePIRLS), students attending rural schools
were more likely to be experiencing LLOs than their peers attending urban schools.
School type has consistently been shown as a good predictor of achievement (ACER and
ZIMSEC, 2015; Ministry of Education, Science, and Technology of Malawi, 2011; Wasanga et
al., 2012). Students attending non-government schools traditionally outperform those attending
65 The study reported the presence of electricity and water facilities at the school to be significantly associated with mathematics
achievement for both rural and urban areas, but only rural (and not urban) areas for English achievement.
37
government schools (Uganda National Examinations Board, 2010). Non-government schools also
did better than government schools in the three learning areas (UNICEF Eritrea, n.d.). The same
pattern has been reported in numerous studies regarding school location. Students attending
schools in urban environments tend to have greater achievement levels than those attending
schools in rural environments, a finding almost universally consistent across studies (ACER and
ZIMSEC, 2015; Makuwa, 2005; Ministry of Education et al., 2011; Moloi and Chetty, 2010;
Wasanga et al., 2012).
This is not to say that schools perform better because they are located in urban areas. The
amount of resourcing that a school has available likely explains the relationship. As an example,
data from Botswana (TIMSS) show that urban schools were more likely to be private and better
resourced than schools in rural areas. Indeed, it is likely that school resourcing accounts for much
of the variation found between different school types, and between different school locations.
2.2Trends in learning outcomes of children in
primary education in literacy and numeracy in
the ESA region
Tracking progress in student performance over time is an important element of system-level
monitoring in education. As discussed in Chapter 1, approximately two-thirds of the assessments
we reviewed have a system-level monitoring purpose, and conduct recurring assessments to
monitor changes in students’ literacy and numeracy performance levels. Apart from SACMEQ,
PASEC, Uwezo and PIRLS/TIMSS, these assessments are conducted at a national level. As
a regional assessment in Southern and Eastern Africa, SACMEQ is best placed to provide
comparable data for a large number of countries (12 ESA countries participated at least twice).
Since SACMEQ data were unavailable for this study, trends in performance were analysed for the
three countries participating in Uwezo: Kenya, Tanzania and Uganda.66
2.2.1 Uwezo literacy trends
Performance data for Uwezo relate to the percentage of primary-school-aged students that are
able to successfully complete each of the tasks within a learning domain.67 Hence performance is
presented at task level for each participating country for each year to explore changes over time.
Students undertaking the literacy component of Uwezo are rated on their highest level of task
completion on this ascending scale of difficulty: ‘nothing’, ‘syllables’, ‘words’, ‘paragraphs’ and
‘story’. We show in the bar graph below an example of performance levels for Kenyan students
completing the English literacy component over the three years for which data are available (see
Figure 4). The data are presented in a hierarchical format that displays the highest task that each
student could complete. For 2009–2010, it shows that 49 per cent of students were able to
complete the story task. As this is the most difficult, this group of students would therefore also
have successfully completed the paragraph, word and letters tasks. Only 6 per cent of students
were not able to complete the most basic task (letters).
66 This study in particular looks at trends in performance since 2007. At the time of data analysis, PASEC was last implemented in
Burundi and Comoros in 2008/2009 (only once). The first administration of prePIRLS was in 2011. Botswana had participated in
TIMSS before, but with varying target populations (TIMSS 2007: Grade 8; TIMSS 2011: Grade 6 and Grade 9).
67 See Appendix I for further discussion of Uwezo data-related issues for trend analyses.
38
Figure 4 shows a similar proportion of students being able to complete each task across the
three years. The differences in proportions across time for each task – or changes in student
performance – are all relatively minor. Similar figures for trends in English are provided in Appendix
IV for Tanzania and Uganda, and for Swahili in Kenya and Tanzania.
In Tanzania, there was little improvement over time for English in terms of the proportion of
students able to complete the most complicated task (story). However, there was a significant
reduction in those not able to complete any task, dropping from 34 per cent in 2009–10 to 29
per cent in 2012. This difference was still significant – although reduced in magnitude – after
a model that incorporated age and gender was introduced. In Uganda, the opposite pattern was
found. There was little change in the proportions of students who could not complete any task,
but a significant increase in the proportion of students able to complete the most complicated
task, rising from 17 per cent in 2009–10 to 27 per cent in 2012. Because the populations of
students that completed the assessments were different, a separate analysis was conducted
to determine whether age and gender might account for the difference. Even after these two
demographic variables were controlled for, the increase remained significant.
For Swahili, there were few observable trends found across Kenya, but in Tanzania there was a
noticeable drop in the proportion of children able to complete the most difficult task, from 45 per
cent in 2009–10 to 36 per cent in 2012. The drop remained significant even after the age and
gender of the populations were taken into account.
Figure 4. Trends in English performance across time for students in Kenya (Uwezo)
English performance in Kenya (Uwezo)
100%
90%
6
5
6
15
15
15
17
15
15
15
15
13
80%
70%
60%
50%
Nothing
40%
Syllables
30%
20%
10%
0%
49
50
50
Words
Paragraph
Story
39
2.2.2 Uwezo numeracy trends
Similar to reading, seven common groups of tasks of increasing difficulty were also defined for
mathematics, including ‘counting’, ‘numbers’, ‘values’, ‘addition’, ‘subtraction’, ‘multiplication’
and ‘division’. An example for mathematics performance in Tanzania over time is shown below
(see Figure 5). Little variation is evident across the three years in the percentage of students not
able to complete anything (11 per cent). At the other end of the performance spectrum, there is
an increase from approximately one-third of students able to complete multiplication in 2009–10
to approximately one-half of all students in 2012, a significant improvement that remained after
age and gender of the populations were taken into account.
Figure 5. Trends in Mathematics performance across time for students in Tanzania
(Uwezo)
Mathematics performance in Tanzania (Uwezo)
100%
11
80%
26
11
11
6
6
5
5
7
Nothing
8
Counting
15
Numbers
11
7
60%
19
40%
13
Values
11
Addition
20%
33
45
50
Subtraction
Multiplication
0%
2009-10
2011
2012
For Kenya, we compared mathematics trend results between Uwezo 2011 and 2012 and found
little difference in in performance across the two years.68 In Uganda, there was a drop from 32
per cent to 26 per cent between 2011 and 2013 in the percentage of students who were able
to successfully complete the most difficult task, which appears to be related to the increased
percentage of students achieving their highest task performance at lower levels.
Tables with detailed results for the mathematics performance trends over time in Kenya and
Uganda are presented in Appendix IV.
68 The Uwezo division task was not administered in Kenya in 2009–10. The nature of the mathematics performance data from
2009–2010 is therefore not consistent with data from the other two assessments, and was not included in the trends analysis.
40
2.2.3 Uwezo: trends for children experiencing LLOs in literacy and
numeracy
In addition to monitoring trends in performance at each task level, the trends in the percentage
of children experiencing LLOs over time were analysed, using the same criteria reported in
Chapter 2.1 and defined in Appendix II. This analysis allows us to assess whether any potential
interventions or policy changes at national levels had the desired effects on children performing
at the lowest levels. The trend data for the proportions of students who are experiencing LLOs
for each domain, for each country participating in UWEZO appears below (see Figure 6).69
60%
2010/2011
2012
50%
2013
40%
30%
20%
10%
Kenya
Tanzania
Mathematics
English
Mathematics
Swahili
English
Mathematics
Swahili
0%
English
Proportion of students experiencing LLO(%)
Figure 6. Trends in proportions of students experiencing LLOs across Uwezo countries
Uganda
Uwezo
We found little change over time in the relative percentages of students experiencing LLOs in
Kenya for each of the three domains. A slight but significant increase can be seen from 2009 to
2010 in Swahili.
In Tanzania, however, a significant reduction in the percentage of students experiencing LLOs
in English and mathematics is observed from 2009/2010 to 2011 and also from 2011 to 2012.
Conversely, a greater percentage of children experienced LLOs in Swahili from 2009/2010 to
2011.
69 Mathematics data for Kenya in 2010/2011 was not comparable with data for 2012 and 2013, and is thus not reported in this
figure.
41
In Uganda, we observed a reduction in the proportions of children experiencing LLOs in English
across the three time periods. For mathematics, however, we noted a significant increase from
2012 to 2013 in the proportions of children experiencing these problems.
Overall, when looking at the trend patterns from Uwezo across domains, there are instances
where a higher percentage of students were able to complete the even more difficult tasks. On
the other hand, there are instances where the proportion of students unable to complete the most
basic of task changed. The reasons for these changes are not clear, nor whether they are positive
or negative. To comprehend them, a better understanding is needed of the policy, financing and
socio-political environments for education in each country – and how these developed over the
years during which Uwezo was conducted.
2.2.4 Other literacy and numeracy trends for the ESA region
Additional information on trends in performance for the region is available from SACMEQ and
PIRLS. Table 12 summarises the direction of major trend increases or decreases for the 15
entities that participated in SACMEQ reading and mathematics assessments for 2000 and 2007.
A major increase or decrease is defined as greater than 10 scale points (Hungi et al., 2011).
Lesotho, Mauritius, Namibia, Swaziland and Tanzania (Mainland) all had increases in both reading
and mathematics of more than 10 points. In Botswana and Tanzania (Zanzibar), only reading
scores improved by this margin over the time period, while in Malawi only mathematics scores
improved to this extent. A decrease of more than 10 points was found in Mozambique for both
reading and mathematics, and only for mathematics in Uganda (Hungi et al., 2011).
Grade 5 students from South Africa participated in the PIRLS study for two cycles (2006 and
2011). Although reading performance for students in 2011 was higher than for students in 2006,
this difference was not significant for either English or Afrikaans (Howie et al., 2012). Girls
outperformed boys in both cycles, but the gender difference fell from 37 scale points in 2006 to
26 scale points in 2011.
The limited data available for this aspect of learning outcomes make any generalisations difficult
for the ESA region. Conclusions based on improvement or decline in student abilities should
only be considered at the national level, with careful consideration given to national contextual
factors. The contextual background differences of students across years should also be taken
into account when interpreting the results.
42
Table 12. Direction of trends in SACMEQ reading and mathematics scale scores from
2000 to 2007
Country
Reading
Mathematics
Botswana
Kenya
Lesotho
Malawi
Mauritius
Mozambique
Namibia
Seychelles
South Africa
Swaziland
Tanzania (Mariland)
Tanzania (Zanzibar)
Uganda
Zambia
Zimbabwe
•
Increase of more than 10 points from 2000 to 2007
Decrease of more than 10 points from 200 to 2007
Less than 10 point increases or decreases from 200 to 2007
•
Assessment not administered in 2007
Source: Hungi et al., 2011
43
44
3.Improving learning outcomes
in the ESA region: Effective
country level practices
Low learning outcomes in literacy and numeracy – along with a considerable proportion of
disadvantaged students not reaching basic skills in these domains – are among the many
challenges for primary education in the ESA region. The focus of this chapter is on countrylevel practices that have proved to be effective in improving the literacy and numeracy learning
outcomes of disadvantaged primary students in the ESA region.
When speaking about the ‘effectiveness’ of a programme to improve student performance, two
main aspects need to be considered:
• The characteristics and strategies of the programme that were developed and implemented
to improve student performance, such as teacher training in core reading skills, producing
reading materials in the local language, introducing a reading buddy;
• The characteristics of the programme evaluation design that allows for the measurement
of the impact of the programme on student performance, such as using randomised control
trials, and establishing a baseline, mid-line and end-line of student performance using
adequate performance measures.
Both aspects are important. While the former make a programme effective, the latter provides
evidence of a programme’s effectiveness. The more closely that the evaluation design is aligned
with the programme’s intended goals – and the higher the quality of the tools to measure
performance – the more valid and meaningful the results, which can then feed into strengthening
educational interventions to improve student learning outcomes.
There is a substantial body of literature for the ESA region on the impact of practices to increase
the quantitative aspects of education quality – such as access, enrolment and retention rates.
In contrast, there are few reports on programmes to improve student learning outcomes and its
impact. This observation is not particular to our study. A review of 115 impact evaluation studies
in 33 low- and middle-income countries (Murnane and Ganimian, 2014) found a variety of policies
that were effective in increasing enrolment of students from low-income families (Murnane and
Ganimian, 2014, p. 43). Strategies to improve the quality of education – and hence student
performance – were found to be more complex and thus more challenging to undertake. The
review revealed that findings about the impact of strategies on student performance are often
inconsistent, and that ‘blanket statements about the effectiveness of particular reform strategies
are neither accurate nor helpful’ (Murnane and Ganimian, 2014, p. 44). Also, interventions may
have different effects on different groups targeted. A study in Kenya, for example, found that lowand high-achieving students derived very different benefits from free English textbooks (Glewwe,
Kremer and Moulin, 2009). It is therefore important to consider the effects of an intervention for
specific groups or sub-groups, in order to understand whether the same intervention would have
a similar outcome with a different population (Murnane and Ganimian, 2014, p. 44).
We take these considerations into account in the following analysis of effective country-level
practices that focus on the improvement of learning outcomes in the literacy and numeracy of
45
disadvantaged children in primary education in ESAR. We selected the programmes based on
three main principles:
• The programme aims at improving learning outcomes in literacy and numeracy.
• The programme targets disadvantaged children in primary education.
• The impact of the programme on children’s literacy and numeracy learning outcomes has
been evaluated.
3.1Country-level programmes analysed
Altogether 10 programmes were identified, in seven out of the 21 ESA countries that effectively
improved learning outcomes in the literacy and numeracy of disadvantaged children in primary
education. An overview of these programmes is presented in Appendix V (see Table 23, Appendix
V). It is important to note that the programmes presented are examples and do not claim to be
exhaustive.70
We categorised the programmes into four groups according to their main objectives. We did this
to assist the analysis and help readers identify programmes that match their interests:
• Early grade literacy or numeracy programmes (7):
­ - Literacy Boost in Ethiopia, Malawi and Mozambique
­ - Reading to Learn in Kenya and Uganda
­ - Primary Maths and Reading Initiative (PMRI) in Kenya
­ - Malawi Teacher Professional Development Support (MTPDS)
• School improvement programme (1):
­ - JET’s School Improvement Programme in South Africa
• Early childhood development programmes (2):
­ - ECD component of the Early Literacy Project in Mozambique
­- Early Literacy and Maths Initiative (ELMI) as part of the Innovation for Education Programme
in - Rwanda.
The majority of the identified programmes aim at improving early grade literacy or numeracy (or
both). Disadvantaged children targeted in these programmes are low performers from a low socioeconomic background or from economically disadvantaged or remote areas. Literacy Boost in
Mozambique addresses children affected by HIV/Aids. Young children in communities affected by
HIV/Aids are also the target group of the ECD programme in Mozambique. ELMI targeted preschool
children, including children from remote areas without access to ECD programmes. JET’s school
improvement programme was designed for schools in economically disadvantaged rural areas.
All programmes were evaluated using either Randomised Control Trial (RCT) or quasi-experimental
designs. In these settings, students being studied are allocated (or ‘randomly allocated’ in the
case of RCT) to the intervention under study and compared with control groups receiving no
intervention. In all programme evaluations a baseline and end-line was conducted to measure the
progress of performance. Employing such an experimental evaluation design, with pre- and postmeasurement of performance, ensures that these features are built in from the very beginning of
the programme design. This is commonly observed in the example programmes.
70 A variety of programmes are implemented in the ESA region, as well as in countries where no examples have been identified
for this analysis. In general, we observed a lack of evaluation reports or other documentation about the impact of a particular
programme on learning outcomes; another 10 programmes were identified during this study where evaluation, including
measurement of learning outcomes, was still in progress. They were not included.
46
3.2Key strategies for success
One common feature among the programmes we investigated is the holistic approach they take,
in which a number of interventions to improve learning outcomes of disadvantaged children are
implemented at different levels. Viewing a programme holistically means seamlessly weaving
together activities for teachers’ professional development, provision of teaching/learning
materials, community mobilisation, and capacity-building at the system level. The interventions
flowing from such an approach would work in harmony at the system-, community- and
school-levels. The one shortcoming is that holistic approaches make it difficult to assess which
strategies are successful and which, in turn, pose challenges for scaling up an intervention.
Another shortcoming is the difficulty of ensuring all stakeholders involved are equally committed
to the intervention.
Common intervention strategies identified for the four programme groups are discussed in more
detail in the following sections.
3.2.1 Early grade literacy and numeracy programmes
The main interventions we observed in the early grade literacy and numeracy programmes are
teacher training on reading/mathematics instruction; provision of teaching/learning materials;
production of reading materials in the local language; and community- and home-based reading
activities that increase access to reading materials for children in and out of school.
All seven programmes reported significant improvement in the reading and mathematics skills of
students who were targeted for these interventions.
A number of interventions contributed to the success of Save the Children’s Literacy Boost in
Ethiopia, Malawi and Mozambique. In Ethiopia, having a reading buddy turned Literacy Boost
non-reading students into readers. In Malawi, Literacy Boost employed a three-pronged approach
that included: ‘(1) use of assessments to identify gaps and measure improvements in the five
core reading skills; (2) training of teachers in the national curriculum with an emphasis on core
reading skills; and (3) community action through community mobilisation to support children’s
reading’ (Save the Children, n.d., p. 2). The success of the programme is credited to the use of
these three measures.
Reading to Learn was implemented in the official languages of reading instruction in the early
primary grades, i.e., Lango in Uganda and Swahili in Kenya. At the heart of the Reading to Learn
programme is a ‘five-step scaffolding approach to literacy instruction, building from a conceptual
understanding of stories, to decoding letter-sound relationships and eventually writing new
sentences and stories’ (Lucas, McEwan, Ngware and Oketch, 2014, p. 951). The coherence of
this instructional model, which was aligned with materials, teacher training and well-targeted
instructional interventions, helped ensure its success.
The Primary Maths and Reading Initiative (PRIMR) in Kenya contributed to the development
and use of pedagogical materials and practices to improve children’s foundational reading and
mathematics skills. Materials developed were congruent with Kenyan curriculum documents and
included detailed lesson plans based on best teaching practices identified through research,
47
student books, training manuals for Teacher Advisory Centre (TAC) tutors and teacher training
videos, teacher-support mechanisms, student assessment tools and teacher observation tools.71
The end-line impact evaluation showed remarkable improvements in pupils’ literacy and numeracy
abilities, especially for pupils starting at the lowest levels. Learning outcomes in English and
Kiswahili improved significantly; smaller improvements were seen in mathematics. Overall, girls
performed at the same level as boys – if not better – especially in literacy.
Strategies found to be highly beneficial for the implementation of PRIMR were:
• consistent TAC tutors’ visits to schools to support teachers (including its facilitation through
reimbursement);
• regular professional development through brief trainings, with follow-up and refresher
meetings;
• changing mindsets from traditional teaching to more active, student-focused approaches;
• accurate distribution of classroom materials to schools based on school enrolment data;
• planning and a sophisticated distribution network;
• limiting the number of schools that a TAC tutor is responsible for;
• provision of books at a 1:1 ratio;
• accommodating more literacy and numeracy instructional time during the week;
• keeping the transfer of teachers trained in PRIMR to a minimum (RTI, 2014).
The main purpose of the Malawi Teacher Professional Development Support (MTPDS) project
was to enhance the instructional practices of teachers, especially for early-grade reading. The
project focused mainly on supporting the lower primary sub-sector, with an emphasis on teacher
skill development, classroom support, and materials development. The evaluation report showed
that students who participated in the intervention achieved noticeable gains in performance,
surpassing students in the control schools, with significant differences seen on all sub-tests. Two
key strategies were identified as having the most significant impact:
• intensive coaching for teachers provided by the project’s primary education advisors;
• continuity in support over time (Randolph, Nkhoma and Backman, 2013).
3.2.2 School improvement programmes
JET’s School Improvement Programme aims at improving the efficiency of the educational
system through a systemic approach to enhancing the functioning and educational performance
of schools. ‘The key assumption underlying the model is that educational outcomes will improve
if teachers are effective and the teaching and learning environments are supported by effective
school organisation, community involvement, and district support and monitoring’ (JET Education
Services, n.d., p. 9).
The school improvement model entails seven elements:
1)Stakeholder mobilisation in the community to support the improvement programme;
2)Planning and organisation to improve school management and the functioning of schools
as organisations, including curriculum management, strategic planning and financial
management;
3)Teacher performance, including awareness of teaching goals, focus on learning outcomes,
access to efficient curriculum delivery systems and resources and provision of curriculum
planning and delivery materials, school support visits and cluster-level activities;
71 Ben Piper and the PRIMR team, personal communication, 31 March 2015.
48
4)Parent involvement through a parent mobilisation programme, which includes setting up
home study groups monitored by parents, and developing a practical guide on how parents
should support their children’s learning;
5)District support provided at two levels, the district office and the circuit involved in the
project, to provide additional capacity for planning and programming of school support and
monitoring activities, and for coordination with district-level activities;
6)Teacher competence (subject knowledge and teaching skills): monitoring, planning and
facilitating of teachers’ professional development;
7)Research, monitoring and evaluation, including ongoing monitoring conducted by the
project schools and district officials.
The key assumption underlying the model is that educational outcomes will improve if teachers
are effective, and if the teaching and learning environments are supported by effective school
organisation, community involvement, and district support and monitoring (JET Education
Services, n.d.). The impact evaluation shows significant improvement in learning outcomes. The
mathematics and literacy performance of learners at project schools improved by five percentage
points compared with those at non-project schools (JET Education Services, n.d.).
3.2.3 Early Childhood Development programmes
The main interventions observed in early childhood development programmes are teacher (and
parent) training on the Early Childhood Development (ECD) approach and provision of teaching
and learning materials. A distinctive feature of the Early Literacy and Maths Initiative (ELMI) is
that it equips parents and caregivers with the tools to support their children in developing ELMpromoted skills that encourage playful activities. This ‘ELM at Home’ approach aims at extending
opportunities to develop ELM skills at home, especially for those children with no access to ECD
centres (Save the Children, 2014).
Children who benefited from both of these ECD programmes show significant improvement in their
cognitive development. The ECD programme in Mozambique reports that preschool interventions
in rural communities improved a number of important dimensions of child development. These
included cognitive, fine motor and socio-emotional development, which contribute to higher
levels of school readiness and significantly increased primary school enrolment at the appropriate
age. The report concludes that low-cost, community-based preschool interventions, such as
those studied in the Early Literacy Project, show potential for positively affecting early childhood
development in rural African contexts (Martinez, Naudeau and Pereira, 2012).
For ELMI, the preliminary findings from the mid-line evaluation demonstrate the effectiveness
of the ELMI programme on children’s learning gains, regardless of whether the programme is
implemented at home by parents or by teachers at an ECD centre. In investigations of the drivers
of children’s learning gains, typical background characteristics, such as maternal education level
and socio-economic status, were found to play a role. More importantly, the time spent in play
activities at home at mid-line showed the most consistent relationship with skill growth. The
mid-line report concludes that the strong relationship between playful activities between parents
and children and learning gains highlights the benefits of children engaging in developmentally
appropriate play at early ages (Save the Children, 2014).
49
50
4.A macro theory of change
Based on the evidence collected for this study, we developed a macro theory of change, which is
aimed at monitoring and improving the literacy and numeracy performance of primary students in
the region. The theory combines the main findings of the stock-taking and comparative analysis
of the assessments we reviewed, the main messages derived from the data analysis and related
literature on characteristics of children with learning outcomes and trends in performance over
time, and the experiences of effective country-level practices in the ESA region.
The theory identifies an evidence-based monitoring and intervention cycle, in which the
interdependent sequence of assessment, analysis and action – known as the ‘three A’s – forms
the basis of long-term and sustainable change in student performance (see Figure 7):
• Assessments, i.e., the systematic, strategic and regular collection of data on educational
outcomes and factors related to these outcomes. Assessments form the basis of the
evidence-based monitoring and intervention cycle.
• Analysis and interpretation of the findings on student performance, contexts, their relations
and trends over time at the education policy level. This is to identify the factors to be
addressed at the different levels of the education system in order to improve student
performance.
• Action, the development of targeted educational interventions to improve progress for all
learners, based on the factors and levels identified in the education policy analysis.
The three As are embedded in the conceptual framework of input, process and outcome factors at
the different levels of an education system, which forms an integral component of the evidencebased monitoring and intervention cycle (see Figure 7).72
For assessments, the conceptual framework includes the classification and definition of the main
factors to be considered, and offers guidance on the levels at which data needs to be collected.
For analysis, the framework allows for the identification of factors to be addressed at the different
levels of the education system in order to propel change, as well as their stability and suitability
for change.
For action, the framework helps identify the programmes required and the level at which they
should be implemented. For example, it helps policymakers decide which input, process and
outcome factors need to be addressed in policy decisions at the system and school level, and
which ones can best be shaped by school development activities at the school and classroom
level.
Through the systematic, strategic and regular collection of data through assessments, progress
in student performance is monitored, providing findings to be analysed and interpreted on policy
level, which can then be transformed into action to further improve student performance.
72 The two-dimensional taxonomy is outlined in the conceptual framework in the introduction to this report.
51
Figure 7. A macro theory of change. An evidence-based monitoring and intervention
cycle as premise for change: assessment, analysis, action
Evidence-based monitoring and intervention cycle
Output
School level
Classroom level
Student level
Outcome
Assessment
Input
Process
• Purpose: System level monitoring
• Target population: Early, multiple grades,
inclusion of out-of-school children
• Domains: Literacy and numeracy; Contexts
• Current state and progress: Performance and
contexts
• Dissemination strategy: Findings and
products, including datasets
Action
Analysis
• Target interventions and
strategies
• Integrated into a holistic
programme design, involving a
wide range of stakeholders
• Impact evaluation including
measuremt of performance
Policy analysis and interpretation
for strategic decision-making
and policy development towards
improving student performance
• Student performance levels
• Association with context
factors at the different levels
• Trends over time
Key outcomes of the study informing
the three A’s
In our study, we found the following elements of the three A’s important for the effectiveness of
the monitoring and intervention cycle.
Assessment
Purpose
With the majority of the assessments conducted in the ESA region aimed at system-level
monitoring (66 per cent), it is important that regular assessments be undertaken to monitor
student performance levels and related context factors over time.
52
Target population
One of the initial policy decisions must be to define the target population(s) to be assessed.
Grades 2 and 3 are the most frequently targeted in the ESA region. Nearly half of the assessments
test multiple grades. Apart from Uwezo, which is a household-based assessment, all other
assessments are school-based. The school setting excludes a significant proportion of children
of primary school age who are out of school. Considerations on how to include out-of-school
children in the assessment are necessary in order to gain a full picture of children’s performance
levels in a country or region.
Domains and contexts
Among basic competencies, literacy and numeracy are the most widely assessed. Twothirds of the assessments in the ESA region assess both. Frameworks that clearly define and
operationalize what educational themes are to be assessed are invaluable. They can help guide
test development, data analysis and interpretation, and support a common understanding of
the content and scope of the assessment among all stakeholders. In addition to performance
data, it is essential to collect context data in order to explore the relationships between learning
outcomes and background factors, and to identify factors relevant to change. The conceptual
framework, which sets out relevant input, output and process factors on the different levels of
the education system, can help ensure that the intervention instruments are well targeted in
terms of scope and information source, e.g., students, teachers, principals, parents, EMIS.
Current state and progress: Performance and contexts
In order to draw conclusions about performance in literacy and numeracy and to identify which
factors to address at which levels to initiate change, it is essential that competency levels and
benchmarks be defined. In addition, to monitor progress over time, the relationships between
context factors and performance must be understood.
In more than half of the assessments (55 per cent), results are presented with reference to
competency levels or benchmarks. The description of the skills and knowledge required for the
different competency levels provide a concrete understanding of what students can do. Based on
this information, measures can be established to address specific learner needs at each level, for
example, developing instructional strategies or identifying areas for professional training.
In order to accurately monitor progress over time, learning outcomes can only be compared
between different implementations of the same assessment that have the same key features,
such as assessment framework, design, target population and sampling design, and conditions
of administration across multiple cycles. The contextual background differences of students
across years in a study should also be taken into account when interpreting the results.
The same applies for comparisons across countries or regions. At present, the majority of
the assessments in the ESA region are conducted at the national level, 29 per cent at the
regional and 4 per cent at the international level.73 An innovative approach to compare student
performance across different assessments and contexts requires the development of common
learning metrics for literacy and numeracy based on international and regional benchmarks for
national achievement results and curricular expectations.
73 We view the purpose of the EGRA/EGMA implementations from the stock-taking as either system diagnostics or programme
evaluation.
53
Item response theory (IRT) is an essential technique for scaling cognitive data in order to establish
competency levels and effectively monitor performance on system level across contexts and over
time. At present, IRT is not often used in the region. However two country case studies included
in our study where IRT was used (ZELA in Zimbabwe and LARS in Rwanda) are good examples
of innovative capacity-building programmes with a focus on data analysis techniques.
Dissemination
To engage stakeholders and instigate change, a clearly articulated dissemination strategy for the
assessment findings is essential. Dissemination strategies were applied in the assessments that
we reviewed. The majority of the assessments (71 per cent) made their results reports publicly
available. Approximately half also released to the public results summaries, press communiqués
and policy briefs. Some also made public fully documented datasets, which enable further
independent analyses.
Analysis
The analysis and interpretation of assessment findings on student performance, context factors
and trends over time can inform strategic decision-making and policy development. For example,
the analysis we undertook of learning outcomes in the ESA region showed the proportion of
students who experienced LLOs ranged from 18 per cent to 40 per cent for numeracy and
18 per cent to 50 per cent for reading literacy across datasets, countries and year levels. The
message we take from these results is that individual student characteristics, home background
and resourcing (at both the student and school level) are important factors that affect the ability
of students to achieve basic levels of competency in literacy and numeracy. The results suggest
that students would benefit from increased resourcing at the school level. At home, activities
aimed at increasing student engagement with their studies would also have a profound impact on
learning outcomes. The findings also suggest that encouraging parents to involve their children
in literacy and numeracy activities before they start school, giving them more school-related
responsibilities, and providing them with external tuition (if this was somehow made more readily
available for those in need) would all positively impact their children’s learning development.
Data that show improved learning outcomes in literacy and numeracy over time are indicative of
systems that have helped improve overall student capabilities. Examining the education policies,
financing and socio-political environments that may have contributed to these improvements
would be helpful, as would studying the learning environments in countries where negative
trends in outcomes have been observed.
Action
For our study, we examined 10 country-level programmes that have proven effective at improving
literacy and numeracy learning outcomes of disadvantaged primary students.
Overall, common features included:
• Interventions and strategies targeted to a particular group of disadvantaged children,
including children with low literacy achievement, children from disadvantaged socioeconomic backgrounds, marginalised children in slums and non-formal settlements, and
children in economically disadvantaged rural areas.
54
• A holistic programme design that addresses as many levels of the education system
and stakeholders as possible. Such an approach typically involves teachers’ professional
development, provision of teaching/learning materials, community mobilisation and capacitybuilding at system level. This is a multi-level approach to programme implementation, which
provides interventions at system-, school- and community-levels.
• Impact evaluation, including the measurement of progress in learning outcomes to
provide evidence of success. Programmes with randomised control trials (RCT) or quasiexperimental designs require baseline, mid-line and end-line studies to ensure a built-in
monitoring process from the beginning of the study.
Within programmes that share a particular purpose, we identified a number of key strategies that
contributed to the success of country-level practices in the ESA region.
Effective early-grade literacy/numeracy programmes included teacher training on reading/
mathematics instruction, aligned with the provision of teaching and learning materials and the
production of reading materials in the local language, more active, student-focused approaches,
the use of assessments, well targeted instructional interventions (e.g., students having a reading
buddy to support their learning to read), increased instructional time, and community support for
children’s reading.
Additionally, programmes that aimed at a whole-school improvement were shown to have a
significant impact on learning outcomes. Effective strategies included strong school organisation,
community involvement, district support and monitoring and supportive learning environments.
Effective Early Childhood Development programmes comprised of teacher (and parent) training
based on a particular model (e.g. encouraging playful activities involving early literacy and
numeracy skills) and provision of teaching materials and tools also helped support early literacy
and maths skill development.
These findings underline the importance of exposing children to a rich learning environment in
their early years. Combined with an effective organisation that supports teaching and learning
at all levels and equips teachers (and parents) with instructional approaches and educational
material, these factors establish a sound basis for literacy and numeracy skills development.
Conclusions
Synthesising the main findings from this study, we developed a macro theory of change, anchored
in the ‘three A’s approach (assessment, analysis and action) and aimed at initiating a long-term
and sustainable change in student performance. While substantial research has been undertaken
to identify the factors that contribute to student school attendance, more research is required
to understand how student learning can be improved, particularly in developing countries. We
need to deepen our knowledge of where students are at in their learning and how performance
progresses over time so that effective targeted interventions can be developed. In order to do
this, assessment programmes must be undertaken to provide quality comparable data across
populations, between grades and over time. Finally, the results of the assessments must be
integrated into education reform agendas, so that human and financial resources to address
children’s needs are efficiently deployed.
55
56
References
ACER. (2014a). The Early Grade Reading Assessment: Assessing children’s acquisition of
basic literacy skills in developing countries. Assessment GEMS Series No. 2. ACER Press,
Melbourne. <http://www.acer.edu.au/files/AssessGEMs_EGRA.pdf>
ACER. (2014b). Inception report on specific methodology: Consultancy service for improving
quality education and children’s learning outcomes and effective practices in the Eastern
and Southern Africa region. ACER Press, Melbourne.
ACER. (2015). The Southern and Eastern Africa Consortium for Monitoring Educational
Quality. Assessment GEMS Series No. 8. ACER Press, Melbourne.
ACER. (n.d.). Learning assessments at a glance. ACER Press, Melbourne. Viewed 30 March
2015, <http://www.acer.edu.au/gem-la>.
ACER, and ZIMSEC. (2013). Zimbabwe Early Learning Assessment (ETF programme) Base-line
Study.
ACER, and ZIMSEC. (2015). Evaluation of the Education Development Fund Programme:
Zimbabwe Early Learning Assessment (ZELA). 2014 Monitoring Report.
American Institutes for Research (AIR). (2012). Ethiopia Early Grade Reading Assessment Data Analytic Report: Language and Early Learning: American Institutes for Research (AIR).
American Institutes for Research (AIR). (n.d.). Namibian National Standardized Achievement
Test (NSAT).
Retrieved 17 March, 2015, from http://www.air.org/project/namibiannational-standardized-achievement-test-nsat
Beattie, K. (2014). Early Grade Reading Assessment results Somalia Programme: Concern
Worldwide.
Beattie, K., & Grogan, R. (2013). Results of the Early Grade Reading Assessment conducted
in five schools in Mogadishuschools: Concern Worldwide.
Biggs, J. (1999). Teaching for Quality Learning at University. Philadelphia, PA: Society for
Research into Higher Education and Open University Press.
Biggs, J., and Moore, P. (1993). The Process of Learning (3rd edn). Australia: Prentice Hall.
Brombacher, A., Nordstrum, L., Davidson, M., Batchelder, K., Cummiskey, C., & King, S.
(2014). National Baseline Assessment for the 3Rs (Reading, Writing, and Arithmetic) Using
EGRA, EGMA, and SSME in Tanzania: Study Report.
Cao, Y., Dowd, A. J., Mohammed, O., Hassen, S., Hordofa, T., Diyana, F., & Ochoa, C. (2011).
Literacy Boost, Dendi Ethiopia: Three month report: Save the Children.
Collins, P., Galbert, P. D., Hartwell, A., Kochetkova, E., Mulcahy-Dunn, A., Nimbalkar, A., and
Ralaingita, W. (2012). Pupil Performance, Pedagogic Practice, and School Management: An
SSME Pilot in Zambia: RTI, North Carolina.
CONFEMEN. (2010a). Rapport PASEC–Burundi 2010: Enseignement primaire: Quels défis
pour une éducation de qualité en 2015? CONFEMEN, Dakar, Senegal.
57
CONFEMEN. (2010b). Rapport PASEC–Union des Comores: Diagnostique et préconisations
pour une scolarisation universelle de qualité. CONFEMEN, Dakar, Senegal.
CONFEMEN. (2010c). RAPPORT PASEC Union des Comores 2010. Diagnostic et préconisations
pour une scolarisation universelle de qualité. PASEC/CONFEMEN, Comoros.
CONFEMEN. (n.d.). Programme d’Analyse des Systèmes d’Éducatif de la CONFEMEN (PASEC).
Viewed 16 March 2015, <http://www.confemen.org/le-pasec/>
Constitution of Zimbabwe Amendment (No. 20), 2013. (2013).
Creemers, B., and Kyriakides, L. (2008). The Dynamics of Educational Effectiveness: A
Contribution to Policy, Practice, and Theory in Contemporary Schools. Routledge, London.
Crouch, L., Korda, M., & Mumo, D. (2009). Improvements in Reading Skills in Kenya: An
Experiment in the Malindi District: RTI.
Department of Basic Education Republic of South Africa. (2011). Report on the Annual
National Assessments of 2011.
Department of Basic Education Republic of South Africa. (2012). Report on the Annual
National Assessment of 2012: Grades 1-6 & 9.
Department of Basic Education Republic of South Africa. (2013). Report on the Annual
National Assessment of 2013: Grades 1-6 & 9.
Department of Basic Education Republic of South Africa. (2014). Report on the Annual
National Assessment of 2014: Grades 1–6 and 9. Department of Basic Education, Pretoria.
DeStefano, J., Ralaingita, W., Costello, M., Sax, A., & Frank, A. (2012). Task Order 7: Early
Grade Reading and Mathematics in Rwanda: RTI.
Dowd, A. J., & Fonseca, J. (2012). Literacy Boost Results Mozambique: Save the Children.
Dowd, A. J., and Mabeti, F. (2011). Literacy Boost Results Malawi: Year 2 report. Save the
Children, Washington DC.
Dowd, A. J., Wiener, K., & Mabeti, F. (2010). Literacy Boost Results Malawi: Year 1 report:
Save the Children.
Examinations Council of Lesotho (ECoL). (n.d.). Examinations Counil of Lesotho: Research and
Publications. Retrieved 17 March, 2015, from http://www.examscouncil.org.ls/research.
aspx
Examinations Council of Zambia. (2015). Zambia - National Assessment Survey.
fhi 360 Education Policy and Data Centre. (2015). EPDC Policy Brief: Mapping national
assessments. Education Policy and Data Centre, Washington DC.
fhi 360 Education Policy and Data Centre. (n.d.-a). Data. Retrieved 2 April, 2015, from http://
www.epdc.org/data
fhi 360 Education Policy and Data Centre. (n.d.-b). National Education Profiles. Viewed 2 April
2015, <http://www.epdc.org/tags/national-education-profiles>.
Friedlander, E., Candiru, S., & Dowd, A. J. (2010). Literacy Boost Uganda: Baseline Report
2010: Save the Children.
58
Friedlander, E., Hordofa, T., Diyana, F., Hassen, S., Mohammed, O., and Dowd, A. J. (2012).
Literacy Boost, Dendi Ethiopia: Endline II. Save the Children, Washington DC.
Fullan, M. (2006). Change theory: A force for school improvement. Seminar Series Paper no.
157, November. CSE, Melbourne.
Garrouste, C. (2011). ‘Explaining learning gaps in Namibia: The role of language proficiency.’
International Journal of Educational Development, 31(3), 223–233. doi: 10.1016/j.
ijedudev.2010.06.016
Glewwe, P., Kremer, M., and Moulin, S. (2009). ‘Many Children Left Behind? Textbooks and
Test Scores in Kenya.’ American Economic Journal: Applied Economics, 1(1), 112–135.
Global Partnership for Education. (2012). Results for Learning Report 2012: Fostering EvidenceBased Dialogue to Monitor Access and Quality in Education. GPE, Washington DC.
Gove, A., and Wetterberg, A. (2011). The Early Grade Reading Assessment: Applications and
Interventions to Improve Basic Literacy. RTI International, North Carolina.
Government of Zimbabwe. (2009). Short Term Emergency Recovery Programme (STERP):
Getting Zimbabwe moving again. Government of Zimbabwe, Harare.
Guajardo, J., Night, C. S., Heijnen, E., Komakech, C., P’kech, E. A., Odwong, O. D., . . .
Dowd, A. J. (2010). Literacy Boost Uganda: Midline Report 2012: Save the Children.
Hassen, S., & Friedlander, E. (2012). Literacy Boost Results Ethiopia: Save the Children.
Hivos/Twaweza. (2014). Are Our Children Learning? Literacy and Numeracy across East Africa
2013. Uwezo East Africa at Twaweza, Kenya.
Howie, S., Staden, S. v., Tshele, M., Dowse, C., and Zimmerman, L. (2012). PIRLS 2011.
South African Children’s Reading Literacy Achievement Report. Centre for Evaluation and
Assessment, University of Pretoria, Pretoria.
Human Sciences Research Council South Africa. (n.d.). Testing, testing First national
assessment of Grade 9 pupils shows much work lies ahead. Retrieved 20 March, 2015,
from http://www.hsrc.ac.za/en/review/september-2011/testing-testing#sthash.q42e9GXL.
dpuf
Hungi, N. (2011). Characteristics of school heads and their schools. UNESCO, Paris.
Hungi, N. (2011a). Accounting for variations in the quality of primary school education.
Retrieved from: http://www.sacmeq.org/sites/default/files/sacmeq/reports/sacmeq-iii/
working-papers/07_multivariate_final.pdf
Hungi, N. (2011m). Characteristics of grade 6 pupils, their homes and learning environments
SACMEQ Working Paper. Paris: SACMEQ.
Hungi, N., Makuwa, D., Ross, K., Saito, M., Dolata, S., van Cappelle, F., . . . Vellien, J.
(2010). SACMEQ III project results: Pupil achievement levels in reading and mathematics.
SACMEQ, Paris.
Hungi, N., Makuwa, D., Ross, K., Saito, M., Dolata, S., van Cappelle, F., . . . Vellien, J. (2011).
SACMEQ III project results: Levels and trends in school resources among SACMEQ school
systems. SACMEQ Working Document. Paris: SACMEQ.
59
Hungi, N., Ngware, M., and Abuya, B. (2014). ‘Examining the impact of age on literacy
achievement among grade 6 primary school pupils in Kenya.’ International Journal of
Educational Development, 39, 247–259.
IEA. (2013a). PIRLS 2011. IEA, Amsterdam
IEA. (2013b). PIRLS 2011 Contextual Questionnaires. Viewed 17 April 2015, <http://
timssandpirls.bc.edu/pirls2011/international-contextual-q.html>.
IEA. (2013c). TIMSS 2011. Viewed 7 August 2015, <http://timssandpirls.bc.edu/timss2011/
schedule.html>
IEA. (2013d). TIMSS 2011 Contextual Questionnaires. Viewed 17 April 2015, <http://
timssandpirls.bc.edu/timss2011/international-contextual-q.html>
IEA. (n.d.). IEA Study Data Repository. Viewed 17 March 2015, <http://rms.iea-dpc.org/>.
JET Education Services. (n.d.). Sustainable school improvement. JET, Zambia.
Joncas, M. (2011). Methods and procedures: PIRLS 2011 Target populations: IMSS & PIRLS
International Study Centre, Lynch School of Education, Boston College.
Kaplan, R. M., and Saccuzo, D. P. (1997). Psychological testing: Principles, applications and
issues. Brooks Cole Publishing Company, Pacific Grove, California.
Kyriakides, L., and Creemers, B. (2006). ‘Using the dynamic model of educational effectiveness
to introduce a policy promoting the provision of equal opportunities to students of different
social groups.’ In D. M. McInerney, M. Dowson, and S. V. Etten (eds), Effective Schools,
vol. 6, pp. 17–41. Information Age Publishing, North Carolina.
Lesotho National Assessment of Educational Progress (LNAEP) Survey Report, 2010. (n.d.).
Lucas, A. M., McEwan, P. J., Ngware, M., and Oketch, M. (2014). Improving early-grade
literacy in East Africa: Experimental evidence from Kenya and Uganda. University of
Delaware, Delaware.
Majgaard, K., and Mingat, A. (2012). Education in sub-Saharan Africa: A comparative analysis.
World Bank, Washington DC.
Makuwa, D. (2005). The SACMEQ II Project in Namibia: A Study of the Conditions of Schooling
and the Quality of Education. Ministry of Basic Education, Sport and Culture, Namibia.
Makuwa, D. (2011). Characteristics of grade 6 teachers SACMEQ Working Paper. Paris:
SACMEQ.
Martin, M. O., and Mullis, I. V. S., (eds). (2012). Methods and procedures in TIMSS and PIRLS
2011. TIMSS and PIRLS International Study Center, Boston College, MA.
Martin, M. O., Mullis, I. V. S., Foy, P., and Arora, A. (2012). TIMSS 2011 International Results
in Mathematics. TIMSS and PIRLS International Study Center, Boston College, MA.
Martin, M. O., Mullis, I. V. S., Foy, P., & Stanco, G. M. (2012). TIMSS 2011 International
Results in Science. Chestnut Hill, MA: TIMSS & PIRLS International Study Center, Boston
College.
Martinez, S., Naudeau, S., and Pereira, V. (2012). The Promise of Preschool in Africa: A
Randomized Impact Evaluation of Early Childhood Development in Rural Mozambique. 31e,
Washington DC.
60
Mason, M. ( 2014). Complexity Theory in Education Governance: Initiating and sustaining
systemic change. UNESCO Bureau of Education, Paris.
Miksic, E., & Harvey, S. (2012). Malawi National Early Grade Reading Assessment Survey:
Midterm Assessment: RTI.
Ministère de l’Éducation Nationale et de la Recherche, & PASEC-CONFEMEN. (2010). Rapport
PASEC - Union des Comores: Diagnostique et préconisations pour une scolarisation
universelle de qualité. Dakar, Senegal: CONFEMEN.
Ministère de l’Enseignement de Base et Secondaire, de l’Enseignment de Métiers, de la
Formation Professionnelle et de l’Alphabétisation, & PASEC-CONFEMEN. (2010). Rapport
PASEC - Burundi 2010: Enseignement primaire: Quels défis pour une éducation de qualité
en 2015? Dakar, Senegal: CONFEMEN.
Ministry of Education, Science, and Technology of Malawi. (2011). The SACMEQ III project in
Malawi. Malawi Ministry of Education, Science and Technology, Malawi.
Ministry of Education, Science and Technology of Malawi. (2014). Monitoring Learning
Achievement in Primary Education Malawi Report. Malawi Ministry of Education, Science
and Technology, Malawi.
Ministry of Education of Angola, World Bank, A. R., Human Development Department, and
Russia Education Aid for Development Programme. (2011). Early Grade Literacy in Angola.
Ministry of Education of Ethiopia (FDRE). (2008). Ethiopian Third National Learning Assessment
of Eighth Grade Students. Ministry of Education of Ethiopia, Addis Ababa.
Ministry of Education of Ethiopia (FDRE). (2013). Ethiopian 4th National Learning Assessment
of Grades 4 and 8 Pupils: Data Analytic Report. Addis Ababa, Ethiopia.
MOESAC. (2006). Education Amendment Act, 2005 (Chapter 25:04), H.B.6A, 2005.
Government of Zimbabwe, Harare.
MOESAC. (2009). Education at a Glance 2009. MOESAC, Harare.
MOESAC. (2011). Education Medium Term Plan 2011–2015. MOESAC, Harare.
Moloi, M. (n.d.). Mathematics achievement in South Africa: A comparison of the official
curriculum with pupil performance in the SACMEQ II Project.
Moloi, M. Q., and Chetty, M. (2010). The SACMEQ III project in South Africa: A study of
the conditions of schooling and the quality of education: South Africa Country Report.
Department of Basic Education, Pretoria.
Monyaku, B., & Mmereki, O. A. (2011). The SACMEQ III project in Botswana: A study of the
conditions of schooling and the quality of education: Botswana Ministry of Education and
Skills Development, Division of Planning Statistics and Research.
Monyaku, B. (2012). The SACMEQ III project in Botswana: A study of the conditions of
schooling and the quality of education. Botswana Ministry of Education and Skills
Development, Botswana.
61
Mullis, I., Martin, M., Foy, P., and Arora, A. (Eds.). (2012). TIMSS 2011 International Results in
Mathematics. TIMSS and PIRLS International Study Center, Boston College, and International
Association for the Evaluation of Educational Achievement (IEA). Chestnut Hill, MA.
Mullis, I., Martin, M., Foy, P., and Drucker, K. (2012). PIRLS 2011 International Results in
Reading. TIMSS and PIRLS International Study Center, Boston College, and International
Association for the Evaluation of Educational Achievement (IEA). Chestnut Hill, MA.
Mullis, I. V. S., and Martin, M. O. (Eds.). (2013). PIRLS 2016 Assessment Framework. TIMSS
and PIRLS International Study Center, Boston College, and International Association for the
Evaluation of Educational Achievement (IEA). Chestnut Hill, MA.
Mullis, I. V. S., Martin, M. O., Foy, P., & (with Olson, J. F., Preuschoff, C., Erberber, E., &
Galia, J.). (2008a). TIMSS 2007 International Science Report: Findings from IEA’s Trends
in International Mathematics and Science Study at the Fourth and Eighth Grades. Chestnut
Hill, MA: TIMSS & PIRLS International Study Center, Boston College.
Mullis, I. V. S., Martin, M. O., Foy, P., & (with Olson, J. F., Preuschoff, C., Erberber, E., Arora,
A., & Galia, J.). (2008b). TIMSS 2007 International Mathematics Report: Findings from
IEA’s Trends in International Mathematics and Science Study at the Fourth and Eighth
Grades. Chestnut Hill, MA: TIMSS & PIRLS International Study Center, Boston College.
Mullis, I. V. S., Martin, M. O., Foy, P., & Drucker, K. T. (2012). PIRLS 2011 International Results
in Reading. Chestnut Hill, MA: TIMSS & PIRLS International Study Center, Boston College.
Mullis, I. V. S., Martin, M. O., Kennedy, A. M., & Foy P. (2007). IEA’s Progress in International
Reading Literacy Study in Primary School in 40 Countries. Chestnut Hill, MA: TIMSS &
PIRLS International Study Center, Boston College.
Mullis, I. V. S., Martin, M. O., Kennedy, A., Trong, K., and Sainsbury, M. (2009). PIRLS 2011
Assessment Framework. TIMSS and PIRLS International Study Center and International
Association for the Evaluation of Educational Achievement. IEA, Amsterdam.
Mullis, I. V. S., Martin, M. O., Ruddock, G. J., O’Sullivan, C. Y., and Preuschoff, C. (2009).
TIMSS 2011 Assessment Frameworks. TIMSS and PIRLS International Study Center and
International Association for the Evaluation of Educational Achievement. IEA, Amsterdam.
Mungoi, D., Mandlante, N., Nhatuve, I., Mahangue, D., Fonseca, J., and Dowd, A. J. (2010).
Endline Report of Early Literacy among Pre-school and Primary School Children in Gaza
Province. Save the Children, Mozambique
Murnane, R. J., and Ganimian, A. (2014). Improving educational outcomes in developing
countries: Lessons from rigorous evaluations: NBER Working Paper 20284. National Bureau
of Economic Research, Cambridge, MA.
National Educational Assessments and Examinations Association of Ethiopia. (n.d.). National
Educational Assessments and Examinations Association. Retrieved 17 March, 2015, from
http://www.nae.gov.et/HOMEE.aspx
Nhongo, K. (2014, 19 March). Grade 5 NSAT results depressing. Windhoek Observer. Retrieved
from http://observer24.com.na/national/3131-grade-5-nsat-results-depressing
Nyanguru, A., and Peil, M. (1991). ‘Zimbabwe since Independence: A people’s assessment.’
African Affairs, 90(361), 607-620.
62
OECD. (2013). PISA 2012 Assessment and Analytical Framework: Mathematics, Reading,
Science, Problem Solving and Financial Literacy. OECD Publishing.
Piper, B. (2010). Uganda Early Grade Reading Assessment Findings Report: Literacy Acquisition
and Mother Tongue: RTI.
Piper, B., & Mugenda, A. (2013). The Primary Math and Reading (PRIMR) Initiative: Midterm
Impact Evaluation. North Carolina, USA: RTI.
Pouezevara, S., Costello, M., & Banda, O. (2013). Malawi National Early Grade Reading
Assessment Survey: Final Assessment - November 2012: RTI.
Purves, A. C. (1987). ‘The Evolution of the IEA: A Memoir’. Comparative Education Review,
31(1), 10–28.
Randolph, E., Nkhoma, M., and Backman, S. (2013). ABE/LINK Malawi Teacher Professional
Development Support: Project M&E report. USAID.
Raupp, M., Newman, B., & Revés, L. (2013). Impact evaluation for the USAID/Aprender la
ler project in Mozambique: Baseline report: International Business & Technical Consultants,
Inc. (IBTCI).
Report on Monitoring Learning Achievements (MLA) in Grade 4 in Puntland and Somaliland.
(2012).
Results for Development. (n.d.). Center for Education Innovations. Viewed 2 April 2015,
<http://www.educationinnovations.org/>.
Ross, K. (2009). The Southern and Eastern Africa Consortium for Monitoring Educational
Quality (SACMEQ). Paper presented at the Global Partnership for Education’s ‘Biennial
Partnership Meeting’ Copenhagen, April 20–21.
RTI. (2004). Eddata II: Education data for decision making: School management. Viewed 17
April 2015, <https://www.eddataglobal.org/management/index.cfm>.
RTI. (2010). Ethiopia Early Grade Reading Assessment–Data Analytic Report: Language and
Early Learning. RTI International, North Carolina.
RTI. (2011). Malawi National Early Grade Reading Assessment Survey: Baseline Assessment:
RTI.
RTI. (2012). The Primary Math and Reading (PRIMR) Initiative: Baseline Report. North Carolina,
USA.
RTI. (2014a). EGRA Tracker.
RTI. (2014). USAID/Kenya Primary Math and Reading (PRIMR) Initiative: Final Report. RTI,
North Carolina.
RTI. (2015a). EGRA Tracker. In EGRA_tracker_for_web_16June2015.xlsx (Ed.).
RTI (2015c, January 9). [Personal Comunication, Request for data access].
RTI. (n.d.). EdData: Education data for decisionmaking.
https://www.eddataglobal.org/index.cfm
Retrieved 17 March, 2015, from
Rwanda Education Board. (2012). Learning Achievement in Rwandan Schools (LARS). Rwanda
Education Board (REB), Kigali, Rwanda.
63
Rwanda Education Ministry. (2014). 2013 Education Statistical Yearbook. Ministry of
Education, Kigali.
SACMEQ. (2013). SACMEQ. Viewed 31 March 2015, <http://www.sacmeq.org/>.
Sakala, C. T., & Chilala, M. M. (2007). The Role of the Zambia National Assessment Programme
in Evaluating the Attainment of Educational Goals. Paper presented at the 33rd International
Association for Educational Assessment Annual Conference, Baku, Azerbaijan. http://www.
iaea.info/documents/paper_1162d22700.pdf
Sasman, C. (2011, 19 July). Grade 7 tests show disappointing results. The Namibian. Retrieved
from http://www.namibian.com.na/indexx.php?archive_id=82769&page_type=archive_
story_detail&page=1580
Satio, M. (2011). Trends in the magnitude and direction of gender differences in learning
outcomes, SACMEQ Working Paper No .4. SACMEQ.
Save the Children. (2014). Early Literacy and Maths Initiative (ELMI): Rwanda Midline Report.
Save the Children, Washington DC.
Save the Children. (n.d.). Literacy Boost. Viewed 17 March 2015, <http://resourcecentre.
savethechildren.se/>.
The Australian Council for Educational Research, & Zimbabwe School Examination Council.
(2013a). Evaluation of the Education Development Fund Program - Zimbabwe Early Learning
Assessment (ZELA): 2013 Monitoring Report.
The Australian Council for Educational Research, & Zimbabwe School Examination Council.
(2013c). Zimbabwe Early Learning Assessment (ETF program) Base-line Study.
The Australian Council for Educational Research, & Zimbabwe School Examination Council.
(2015). Evaluation of the Education Development Fund Program - Zimbabwe Early Learning
Assessment (ZELA): 2014 Monitoring Report.
The Kenya National Examinations Council. (n.d.). The Kenya National Examinations Council:
NASMLA.
Retrieved 17 March, 2015, from http://www.knec.ac.ke/main/index.
php?option=com_phocadownload&view=section&id=43:nasmla
The National Assessment Centre. (2010). Monitoring of learner achievement for class 3 in
literacy and numeracy in Kenya: Summary of results and recommendations. The Kenya
National Examinations Council, Nairobi, Kenya.
The National Assessment Centre. (2010a). Monitoring of learner achievement for class 3 in
literacy and numeracy in Kenya. Nairobi, Kenya: The Kenya National Examinations Council.
The National Assessment Centre. (2010c). Monitoring of learner achievement for class 3 in
literacy and numeracy in Kenya: Summary of results and recommendations. Nairobi, Kenya:
The Kenya National Examinations Council.
TIMSS and PIRLS International Study Center. (n.d.). TIMSS and PIRLS. Viewed 16 March
2015, <http://timssandpirls.bc.edu/>.
Twaweza. (n.d.). Uwezo. Viewed 17 March 2015, <http://www.uwezo.net/>.
Uganda National Examinations Board. (2010). The Achievement of Primary School Pupils in
Uganda in Numeracy, Literacy in English and Local Languages. Uganda National Examinations
Board, Kampala.
64
Uganda National Examinations Board. (n.d.). Uganda National Examinations Board - Home.
Retrieved 18 March, 2015, from http://www.uneb.ac.ug/index.php?link=Home
UNESCO. (2008). EFA global monitoring report 2008—Education for all by 2015: Will we
make it? UNESCO, Paris.
UNESCO. (2010). EFA Global Monitoring Report: Reaching the marginalized: Regional overview:
sub-Saharan Africa. UNESCO, Paris.
UNESCO. (2014). Education for All in sub-Saharan Africa Assessment Report 2014. UNESCO
Dakar Regional Office, Dakar.
UNESCO. (2015a). EFA Global Monitoring Report 2015. EFA 2000–2015: Achievments and
Challenges. UNESCO, Paris.
UNESCO. (2015b). National Assessments [Excel file for Annex of 2015 EFA GMR report—in
publication].
UNESCO. (n.d.-a). Education For All Global Monitoring Report. Viewed 2 April 2015, <http://
www.unesco.org/new/en/education/themes/leading-the-international-agenda/efareport/>.
UNESCO. (n.d.-b). International Bureau of Education. Viewed 2 April 2015, <http://www.ibe.
unesco.org/en.html>.
UNICEF Eritrea. (n.d.). Monitoring Learning Achievement in Eritrea.
UNICEF Mozambique Country Office (2015). [Comments on the Improving Quality Education
and Children’s Learning Outcomes and Effective Practices in the Eastern and Southern
Africa region: Final Report].
UNICEF. (2008a). News note: Zimbabwe education system in a state of emergency. Viewed 1
April 2015, <http://www.unicef.org/media/media_45950.html>.
UNICEF. (2008b). Progress report for UNICEF’s education in emergencies and post-crisis
transition programme. UNICEF.
UNICEF Rwanda Country Office (2015). [Comments on the Improving Quality Education and
Children’s Learning Outcomes and Effective Practices in the Eastern and Southern Africa
region: Final Report].
UNICEF Somalia. (n.d.). Report on Monitoring Learning Achievements (MLA) in Grade 4 in
Puntland and Somaliland.
UNICEF. (2011). The Education Transition Fund II, 2012–2015: Programme Document. UNICEF,
Harare.
UNICEF Zambia Country Office (2015). [Comments on the Improving Quality Education and
Children’s Learning Outcomes and Effective Practices in the Eastern and Southern Africa
region: Final Report].
UNICEF. (2012). Zimbabwe 2012: Millennium Development Goals Progress Report. UNICEF
Zimbabwe, Harare.
UNICEF. (2013a). Country Office Portal: Annual report 2013 for Rwanda. ESARO.
UNICEF. (2013b). Country Office Portal: Annual report 2013 for Zimbabwe.
UNICEF. (2014). The Education Development Fund: Stronger systems, better outcomes: Sixth
Progress Report. Harare: UNICEF Zimbabwe.
65
UNICEF Eritrea. (n.d.). Monitoring Learning Achievement in Eritrea.
UNICEF Zambia Country Office (2015). Comments on the Improving Quality Education and
Children’s Learning Outcomes and Effective Practices in the Eastern and Southern Africa
region: Final Report. UNICEF.
USAID. (2011). USAID Funded Malawi Teacher Professional Development Support (MTPDS)
Program - 2010 Early Grade Assessment (EGMA): National Baseline Report 2010.
Uwezo-Kenya. (2013). Are Our Children Learning? Annual Learning Assessment Report 2012.
Nairobi, Kenya: Uwezo, WERK.
Uwezo-Kenya. (2013). Survey Booklet 2013. Uwezo.
Uwezo-Tanzania. (2013). Are Our Children Learning? Annual Learning Assessment Report
2012. Dar es Salaam, Tanzania: Uwezo, TEN/MET.
Uwezo-Uganda. (2013). Are Our Children Learning? Annual Learning Assessment Report
2012. Kampala, Uganda: UWEZO Uganda.
Uwezo. (2014). Are Our Children Learning? Literacy and Numeracy across East Africa 2013.
Nairobi, Kenya: Uwezo, HIVOS/Twaweza.
Uwezo. (2014). Datasets viewed November 19 2014, <http://www.uwezo.net/>.
Vogel, I. (2012). Review of the use of ‘Theory of Change’ in international development:
Review report. DFID, London.
Wasanga, P. M., A. Ogle, M., and Wambua, R. M. (2012). The SACMEQ III project in Kenya:
A study of the conditions of schooling and the quality of education. The Kenya National
Examinations Council, Nairobi.
Wasanga, P. M., Ogle, M. A., and Wambua, R. M. (2010). The Report on Monitoring Learner
Achievement Study for Class 3 in Literacy and Numeracy. The Kenya National Examination
Council, Nairobi.
Wasanga, P. M., Ogle, M. A., & Wambua, R. M. (2012). The SACMEQ III project in Kenya:
A study of the conditions of schooling and the quality of education: The Kenya National
Examinations Council.
World Bank. (2015). Zimbabwe. <http://data.worldbank.org/country/zimbabwe>.
66
Appendix I: Methodology
Methodology for Chapter 1:
Stock-taking and comparative analysis of existing
assessments in the ESA region
For our stock-taking of existing assessments on primary students’ literacy and numeracy learning
outcomes in the ESA region, we focused on assessments that provide data for one of the
following three purposes:
• system-level diagnostic;
• system-level monitoring;
• programme evaluation.
National examinations were not included. They have a wholly different purpose, and use their
own methods of sampling, data analysis and reporting. Furthermore, examinations tend not be
accompanied by the publicly available documentation that learning assessments generate.
Stock-taking framework
The framework used for presenting and analysing the results of our stock-taking was developed
in relation to previous work ACER undertook to characterise assessments, as well as to the
2008 EFA Global Monitoring Report and other recent attempts to map the national assessment
landscape (see Table 13).74
The detailed results of the stock-taking, with information for all framework categories for each
assessment, are presented in the main stock-taking table in Appendix VI.
74 For an example of previous work ACER has done on characterising assessments, see Learning assessments at a glance (ACER,
n.d.). For the assessment mapping framework used in the 2008 EFA Global Monitoring Report, see the annex ‘National learning
assessments by region and country’ in UNESCO (2008, pp. 208–220). For another example of a recent attempt to map the
national assessment landscape, see fhi 360 Education Policy and Data Centre (2015).
67
Table 13. Stock-taking framework
Framework
element
How assessments were classified within the framework element
Country
Name of country of implementation
Assessment
name
Name of assessment
Organisations/
institutions
responsible
Name of implementing body
Purpose
Assessment purpose, distinguishing:
• System-level diagnostic: Administered once to get a snapshot of
student performance levels at the system-level (usually national)
• System-level monitoring: Administered repeatedly to monitor student
performance levels at the system-level (usually national)
• Programme evaluation: Administered on smaller scale to evaluate the
impact of a programme that aims to improve student performance,
with treatment and control groups, and usually involving baseline,
(mid-line), and end-line
Inception
Assessment start date
Frequency
Frequency of assessment administration:
• cycle length if the assessment is conducted regularly
• years of implementation if the assessment is repeated, but not
regularly
• ‘N/A’ if an assessment is administered only once (one-off)
Target
population
Grade-based (e.g., Grade 4) or age-based (e.g., 10 year old students)
Sample
Brief details of achieved sample, including whether it is nationally
representative
Cognitive
domains
Cognitive domains covered in the assessment to measure student
performance (e.g., literacy and numeracy)
Contextual
instruments
Contextual data collection instruments (e.g., student questionnaire, teacher
questionnaire, school head questionnaire)
Test
administration
Test administration methods, distinguishing between the following:
• School-based or household-based for administration location
• Group administration, small group administration or one-on-one
administration for administration method
• Paper-based, tablet-based or oral for administration mode
68
Data analysis
Reporting and
dissemination
Data
•
•
•
analysis approaches, particularly:
if IRT analysis is used to scale data on student performance;
if competency levels/benchmarks are established;
how student performance is analysed (e.g. frequency analyses, mean
scores);
• if relationships between student performance and contextual factors
are explored via analytical methods such as correlation, regression,
multilevel modelling;
• if trend analysis is conducted;
• if international comparisons are used (in multi-country assessments).
• if results reports are publicly available
• other reporting and dissemination methods
Stock-taking approach
The approach for our stock-taking of the assessments included:
• collating current knowledge within ACER about assessments in ESAR;
• identifying gaps in information about particular assessments, and ESAR countries where
little or nothing is known about assessments;
• attempting to fill information gaps by consulting the following data/documentation:
­- data/documentation from sources including UNESCO’s International Bureau of Education
(UNESCO (n.d.-b); UNESCO’s EFA Global Monitoring Reports (UNESCO (n.d.-a); the
Education and Policy Data Centre maintained by fhi360°, particularly their education
profiles, databases and the findings of their national assessment mapping activity (fhi
360 Education Policy and Data Centre (2015, n.d.-a, n.d.-b); the Centre for Education
Innovations maintained by Results for Development (Results for Development (n.d.); and
the EdData website75 (RTI International, 2004);
­- the annual country office reports and education statistics reports that have been provided
by the UNICEF ESARO and UNICEF Cos;
- data/documentation from activities undertaken as part of UNESCO’s Observatory of
Learning Outcomes (OLO);
- data/documentation from ministry websites of governments in ESAR;
• attempting to fill information gaps by consulting the following contacts as required:
-existing contacts established by ACER in the course of earlier work (e.g. contacts
­
at Results for Development, RTI, Uwezo, and national assessment bodies in specific
countries in ESAR);
­- new contacts established through this consultancy (e.g. contacts with people within
ministries or donor agencies through UNICEF ESARO and COs).
Consultation with existing and new contacts was conducted via email, using standardised
questionnaires that, first, sought an overview of the assessment activities in the country of
interest and, second, sought a more detailed information about particular assessments relevant
to the consultancy.
75https://www.eddataglobal.org/
69
Methodology for Chapter 2:
Literacy and numeracy in primary education in the
ESA region—Students experiencing LLOs and trends
over time
Available datasets
Several criteria were used to select the datasets for our analysis. First, we used datasets relating
to literacy and numeracy that were as current as possible. Second, we selected datasets that
were representative of the population they were assessing, and were based on an appropriate
census or household survey, or sample-design with sampling weights used to represent the target
population. Third, for the data to be useful for profiling students, the dataset had to include links
between achievement data and contextual information about the student.
For the analysis, data from four different assessments implemented in seven ESA countries were
available: Uwezo, PASEC, TIMSS and prePIRLS. PrePIRLS was chosen over the traditional PIRLS
dataset because it is better targeted to the achievement of students from participating countries
for the region. SACMEQ data was not available within our research timeline.76 The datasets used,
the countries involved, the year(s) of implementation, the target population and the domains
assessed appear below (see Table 14).
Specifics about the different datasets used for this analysis are described in the following sections.
Uwezo
Uwezo is a household and age-based survey that assesses literacy and numeracy outcomes for
children in Kenya, Tanzania and Uganda. Achievement data is collected from the children, and
contextual information is obtained via an interview with the head of the household, as well as
from observations made in the school or home environment (Hivos/Twaweza, 2014). Uwezo data
from 2009/2010, 2011 and 2012 were available for the purposes of our research.
The main focus of our data analysis was to examine literacy and numeracy learning outcomes
of children in primary education. A procedure for selecting the relevant children was established
using criteria for students who attend school, students who have dropped out of school, and
children who never attended school.
PASEC
PASEC is a large-scale survey of students’ abilities in mathematics and reading in French that is
administered across 13 countries across multiple grades (generally Grades 2 and 5). Students are
typically assessed at the beginning and end of each grade in order to measure growth over the
course of the year (CONFEMEN, 2010a). PASEC databases for Burundi and Comoros for Grades
2 and 5 were used for our analyses.
76 Efforts were made to gain access to SACMEQ data as well as data from national assessments. In total, 12 ESA countries
participated in SACMEQ. The SACMEQ data archive remained offline when we conducted this research and the process required to
obtain national data to be included in our analysis exceeded the timeline for this study. We thank UNICEF ESARO and COs for their
support in requesting access to data during our research. Findings from SACMEQ and national assessments are referred to in the
discussion where reports were available and contained analysis of contextual data.
70
TIMSS and prePIRLS
TIMSS and PIRLS are IEA studies run on a regular cycle (TIMSS every four years; PIRLS ever five
years) to monitor mathematics, science and reading literacy skills among children in participating
countries from different regions of the world. PrePIRLS was introduced for the PIRLS 2011 cycle
as an assessment of reading literacy that is easier for students than the traditional PIRLS format
(M. O. Martin, Mullis, Foy, and Arora, 2012; I. Mullis, Martin, Foy, and Drucker, 2012). In this
report, data was analysed from TIMSS 2011 Grade 6 (mathematics only) for Botswana. Data was
also analysed from prePIRLS 2011 Grade 4 for both Botswana and South Africa.
Table 14. Assessment programmes for which data were available for analysis
Assessment Countries
Name
Year of implementation Target
population
Domains assessed
Uwezo
Kenya,
Tanzania,
Uganda
2009/2010
2011
2012
Age-based
Literacy (French,
household (6-16) Swahili), Numeracy
PASEC
Burundi,
Comoros
2008/2009 (Burundi)
2008/2009 (Comoros)
Grade 2 students Literacy (French,
Grade 5 students Kirundi), Numeracy
TMSS
Botswana
2011
Grade 6 students Mathematics
Malawi
Botswana,
South Africa
2011
Grade 4 students Literacy (English)
Methodology for the characterisation of students experiencing LLOs
in literacy and numeracy in the datasets
In each of the Uwezo, PASEC, TIMSS and prePIRLS databases, the first step in characterising
students experiencing LLOs was to define who each of these children were. For each of the
defined assessment variables within each database, a dichotomous variable was created that
signified whether the student was defined as experiencing LLOs (value of 1) or whether they are
not considered to be experiencing LLOs (value of 0) for the assessment variable in question. In
each case, the decision of whether or not someone was experiencing LLOs was made based on
definitions in each study. Students without a score for the assessment in question were treated
as missing data.
The second step was to define a list of contextual variables in each dataset that would be
considered of potential interest in characterising the students experiencing LLOs. This list was
devised after considering the types of variables previously identified as being associated with
student achievement. The variables cover a range of information from the student, including
home background, household possessions, parental education and literacy, and student academic
background experiences at school.
The third step was to explore descriptive statistics of two groups with overlapping students for
each of the relevant contextual variables. The first group was students in the target population,
for example, the percentage of the Botswana population for prePIRLS that are female. The second
group was students identified in the first methodological step who were in a LLOs group. The
example for this group would be the female proportion of the Botswana population of prePIRLS
students who were identified in the first stage as experiencing LLOs in reading.
71
The next step was to use significance testing to determine whether students who are categorised
as experiencing LLOs for the assessment in question (value of 1) differ from the remaining
students who were considered not to be experiencing LLOs for the assessment in question
(value of 0). For this we used a logistic regression, a statistical technique that allows for a
dichotomous dependent variable. A significant logistic regression test would imply that students
identified as experiencing LLOs have significantly different characteristics to students who were
not identified. All our analyses use weighted data that enable us to relate our findings back to
the relevant target populations of each of the studies.
Specific methodologies relating to each assessment are listed below.
Profiling methodology: Uwezo
The main focus of our data analysis was to examine literacy and numeracy learning outcomes of
children in primary education. Therefore, the total sample of children participating in Uwezo was
modified. This section provides details on the procedure we used to select the final sample of
children included in our analysis. It also lists the performance and contextual data available for
use in the analyses.
Uwezo is a household- and age-based assessment. Because of these characteristics, the
assessment reaches children with different school enrolment status. From children who are
within the formal education system, Uwezo assesses those attending preschool, primary or
secondary education.
We selected children for our study based on three criteria. First, of all children enrolled in school,
we selected only those in primary grades. This included children attending up to Grade 8 in
Kenya, and children attending up to Grade 7 in Tanzania and Uganda.
Second, among children who had dropped out of school, only those who left during primary
education were considered. Children whose last year at school was Grade 9 and above in Kenya,
and Grade 8 and above in Tanzania and Uganda, were excluded.
Third, among children who had never been enrolled in school, only those within the typical ages
for primary education were considered. One more year was added to the age in which children
are expected to finish primary education in each country. This cut-off meant we only included
out-of-school children 14 years and below in Kenya and Tanzania, and 13 years and below in
Uganda (see Table 15).
The first step in examining the characteristics of children experiencing LLOs is to define the
concept of LLOs. The nature of the Uwezo assessment and the variety of enrolment status
among children taking the tests complicated the definition.
Uwezo tests are administered to a wide population of students aged 6–16, regardless of
enrolment status and, if enrolled in school, the grade level they are attending. Different forms of
the same test are given to all children. The test is aligned with the national Grade 2 curriculum in
the three countries. Because of this alignment, it is assumed that all children attending Grade 3
and above should be able to reach the highest level of achievement: ‘story’ for the literacy tests
and ‘multiplication/division’ for the numeracy test.
72
Typically, the national reports present the results of the assessment in terms of the percentage of
in-school children reaching each of the performance levels by grade. In particular, countries focus
on the outcomes of students above Grade 3. The focus of the regional reports is a comparison
of the percentage of children above Grade 3 who can achieve the highest level of performance
in each test for the three countries. For the purpose of this study, and consistent with UWEZO’s
analytic approach, children who experience LLOs are characterised as those enrolled in Grade 3
and above unable to achieve the highest level of performance.
Uwezo does not report the learning outcomes of out-of-school children. Two types of children
are within this category: those who dropped out of school and those who never received formal
education. A similar approach to that used for identifying in-school children with LLOs was used
for the former group. Thus, children who dropped out of school in Grade 3 and above and who
could not perform at the top level in each domain were identified as experiencing LLOs. For the
group of children who never received formal education, the criterion to identify those with LLOs
was age-based. Those 10 years old and above who could not perform at the top level in each
domain were identified as experiencing LLOs.
Table 15 shows the final sample used for our analysis. The numbers are disaggregated by
enrolment status. The application of the above mentioned exclusion criteria result in the exclusion
of 7 to 14 per cent of the assessed children from our analysis.
Only data for 2012 were used for the purpose of profiling students experiencing LLOs.
Performance data we used in the analyses were for language literacy in English (all countries) and
Swahili (Kenya and Tanzania), as well as numeracy (all countries). Children were scored on their
ability to perform different tasks of increasing difficulty in each domain assessed. For language
literacy, students were graded according to their ability to successfully complete tasks that relate
to letters, words, paragraphs and a story (in increasing order of difficulty). It is assumed that
the ability to complete a task at a higher level means the child can complete the task at a lower
level. For numeracy, students were graded on whether they were able to successfully complete
tasks that relate to counting, numbers, values, addition, subtraction and multiplication. Children
in Kenya were also given a division task.
Contextual information available from UWEZO used in this report includes:
• gender;
• age of student (6–9 years; 10–13 years; 14–16 years);
• type of wall at home (Kenya: mud, polythene, iron sheet, timber, stones/bricks; Tanzania:
mud, burnt bricks, cement bricks, other);
• home resources (access to electricity, TV, radio, phone, clean water, car, fridge, motorbike);
• mother’s level of education (none, some primary, some secondary, post-secondary) (options
vary across countries);
• whether the child receives extra lessons or tuition (Kenya only);
• school type (public, private, other; completed);
• school location (Tanzania only).
73
Table 15. Uwezo final sample for analysis
Uwezo
country
Kenya
Total
2009-10
2011
2012
Tanzania
2009-10
2011
2012
Uganda
2009-10
2011
2012
74,781
125,661
145,564
35,540
110,435
105,352
32,768
100,550
92,188
Children not Children
retained
retained
Children retained
In school
Dropout
Never
enrolled
6,089
68,692
94,489
622
3,582
-8.10%
-91.90%
-93.90%
-0.90%
-5.20%
13,377
112,284
105,286
1,147
5,851
-10.60%
89.40%
-93.80%
-1.00%
-5.20%
15,180
130,384
121,617
1,355
7,412
-10.40%
-89.60%
93.30%
-1.00%
-5.70%
3,747
31,793
29,584
1,307
902
-10.50%
-89.50%
-93.10%
-4.10%
-2.80%
10,743
99,692
89,931
5,404
4,357
-9.70%
-90.30%
-90.20%
-5.40%
-4.40%
14,563
90,789
85,129
2,112
3,548
-13.80%
-86.20%
-93.80%
-2.30%
-5.40%
2,322
30,446
27,878
912
1,656
-7.10%
-92.90%
-91.60%
-3.00%
-5.40%
7,224
93,326
87,370
2,227
3.758
-7.20%
-92.80%
-93.60%
-2.40%
-4.00%
6,137
86,051
81,503
1,845
2,703
-6.70%
-93.30%
-94.70%
-2.10%
-3.10%
Profiling methodology: PASEC
All students included in the databases for PASEC, which is a school-based assessment, are in
the target population. Some difficulties were encountered in reconciling contextual information
across the databases of the two countries that we focus on in this chapter, as the variables and
the names used for them were not comparable. Information about the variables used in PASEC in
these two countries was mostly taken from the French language national reports (CONFEMEN,
2010a, 2010c).
74
For the Burundi Grade 2 database, a weighted sample of 2,694 students represents 405,429
students. For the Burundi Grade 5 database, a sample of 2,625 students represents 253,524
students.
For the Comoros Grade 2 database, a sample of 2,120 students represents 22,490 students. For
the Comoros Grade 5 database, a sample of 1,945 students represents 9,765 students.
Achievement data for PASEC students are collected near the beginning and towards the end of
each assessed grade. This enables measurement of the student’s improvement over the course
of the year. We used performance data obtained towards the end of the year, consistent with
PASEC reporting.
PASEC assesses students in French and mathematics for Burundi and Comoros. In addition,
Grade 2 students from Burundi were also assessed in Kirundi.
In PASEC, there are 3 levels for international comparison (CONFEMEN, 2010b, p. 93):
• Level 1: Students who have a score of less than 25 (out of 100). These students are either
responding randomly and would be considered to be failing at school, or close to it.
• Level 2: Students who have a score between 25 and 40 (out of 100).
• Level 3: Students who have a score between 40 and 100. Students at this level are
considered to have acquired a basic level of knowledge.
For the PASEC datasets, students who have a score within the Level 1 range are considered to
be experiencing LLOs.
Contextual information used in the analyses included the following:
• Gender (proportion of females);
• Whether the student is below the normal age for the grade (below 5 years old for Grade
2 or below 9 for Grade 5) and whether the student is above the normal age for the grade
(above 8 years old for Grade 2, or above 11 for Grade 5);
• An indicator of household possessions – a tally of eight home possessions is used:
electricity, a television, a telephone, a fridge, gas heating, a video recorder, a computer and
a car. The indicator is grouped into three categories:
- less than three possessions;
- between three and five possessions;
- more than five possessions.
• If the student participates in farm work;
• If the student participates in housework;
• If the student participates in retail work;
• Whether work hinders the student’s ability to study at home;
• Whether work hinders the student’s ability to attend school;
• Whether work hinders the student’s ability to concentrate at school;
• Literacy of parents (both father and mother);
• Language spoken at home (Shikomori, Arabic, French, English, Kirundi, Swahili, other);
• Presence of basic facilities in the school (library, computer room, toilets, electricity, drinking
water).
75
Profiling methodology: TIMSS and prePIRLS
For the purpose of our study, data for TIMSS 2011 and prePIRLS 2011 were sourced and analysed.
The TIMSS data relate to Grade 6 mathematics performance for Botswana. Grade 6 is outside
the typical primary level grade assessed by TIMSS (Grade 4).
The prePIRLS 2011 data are for Botswana and South Africa (Grade 4). PIRLS and TIMSS are run
at different cycles: TIMSS is run every four years, whereas PIRLS is run every five years; in 2011
they were conducted at the same time.
Performance in prePIRLS can be linked to the PIRLS reading achievement scale. The PIRLS 2011
item parameters were used to anchor the prePIRLS scale. The results are reported on its own
scale, using the same 0–1000 scale used in TIMSS and PIRLS. Given the widespread familiarity
with the scale used by PIRLS and TIMSS, this metric was also used for prePIRLS. The prePIRLS
scale was centred at 500 as the mean achievement of the three countries combined, and 100
points on the scale was set to the standard deviation of the combined achievement distribution.
As with PASEC, the relevant databases for TIMSS and prePIRLS only include sampled students
from the target grade. Thus all cases in the databases were included in our analysis. The survey
included contextual information collected from students (student questionnaire), from parents
(parent questionnaire), teachers (teacher questionnaire), school principals (school questionnaire)
and from national centres (national context survey). For our research, we used only data that can
be linked back to the students, so the National Context Survey and the teacher questionnaire
(which is sampled at the school level and is not identifiable to a particular classroom) were not
considered.
Students’ achievement on the items in TIMSS and prePIRLS are used to identify the knowledge
and skills associated with achievement at particular points on the achievement scale. Each of the
scales used in the studies has several benchmarks for reading, mathematics and science literacy.
The TIMSS and PIRLS International Study Center worked with various expert groups and subject
advisory committees to set benchmarks for reading, mathematics and science in terms of what
children should be achieving. On a scale of 0 to 100 with 500 mean, four benchmarks were set:
• Advanced International Benchmark (625);
• High International Benchmark (550);
• Intermediate International Benchmark (475);
• Low International Benchmark (400).
For our research, we consider students who do not achieve the Low International Benchmark
(i.e., students who score less than 400) to be experiencing LLOs in that domain.
For mathematics, the Low International Benchmark of 400 was defined for Grade 4 as follows:
Students have some basic mathematical knowledge. They can add and subtract whole numbers.
They have some recognition of parallel and perpendicular lines, familiar geometric shapes, and
map coordinates. They can read and complete simple bar graphs and tables.
For reading literacy, the Low International Benchmark of 400 was defined for Grade 4 as follows:
When reading literary texts, students can locate and retrieve an explicitly stated detail. When
reading informational texts, students can locate and reproduce explicitly stated information that
is at the beginning of the text.
76
The scales in each of the databases include five plausible values that are used to estimate
standard error associated with the achievement level. For our study, only performance on the
first plausible value was used to determine whether a student achieved the low international
benchmark. There is a small margin of statistical error in using this methodology, but it was
necessary in order to conduct the analyses. The relative size of the error is considered to be low.
TIMSS and prePIRLS collect a wide variety of contextual data. The variables chosen for our
analysis are those that are comparable with the information collected in other studies (such as
home background, household possessions, parental education), and those believed to be related
to students experiencing LLOs, such as relevant school-related factors. Contextual information
from the student questionnaire and parent questionnaires used in the analyses included the
following:
• Gender;
• Age (12 years or less; between 12 and 14 years; 14 years or older);
• Test language spoken at home;
• Home resources for learning index (number of books in the home, number of children’s
books in the home, number of home study supports, highest parental education level,
highest parental occupation level) was categorised into:
- many resources
- some resources
- few resources;
• Highest parental education levels;
• Student engagement with reading at school (prePIRLS scale);
• Preschool attendance;
• Engagement in numeracy activities before beginning primary school (TIMSS and PIRLS
International Study Center);
• Engagement in literacy activities before beginning primary school (prePIRLS);
• Competency of early numeracy tasks when beginning primary school (TIMSS and PIRLS
International Study Center);
• Competency of early literacy tasks before beginning primary school (prePIRLS);
Contextual information collected from the school questionnaire used in the analyses included:
• presence of a school library;
• presence of computers used for instruction;
• school location.
Methodology for Uwezo trend analysis
There is a distinct lack of assessment data available for trends analysis in ESAR. To analyse withincountry trends over time, assessments must be administered at least twice. It is also important
to consider that learning outcomes can only be compared between different implementations of
the same assessment that have the same key features, such as assessment framework, design,
target population, and conditions of administration across multiple cycles. Any comparison of
results between different assessments – or between different cycles of the same assessment
that lack the same key features across the different cycles – requires sophisticated linking
procedures and analyses that are beyond the scope of our report.
77
UWEZO data we obtained across three years (2009/2010, 2011, 2012) allowed us to examine
trends in student learning across time. We tracked performance data for each country for each
period of time over three years. English and mathematics performance was tracked in Kenya,
Tanzania and Uganda, whereas Swahili performance was only tracked in Kenya and Tanzania.
To conduct the analyses, several steps were taken to prepare the databases. First, the steps
detailed in Table 15 were carried out to ensure that the appropriate target population was being
tested. The datasets were then aggregated so that a single dataset could be used to examine
trends. Dummy variables were set up to indicate whether the student was able to complete each
stage of the assessment.
These dummy variables were then used to determine the percentage of students who could
complete each task correctly in each year for each domain. Simple logistic regressions were
then computed to determine whether differences in percentages across years were significant
and, if so, odds ratios were used to express the magnitude of these differences. More emphasis
was placed on the percentage of students at the extremes – students able to complete the most
difficult task or not able to complete any tasks. Changes in percentages within these extremes
are largely dependent on each other, and an increase in the percentage of children completing a
task at one level corresponds to an equivalent decrease in percentage across other tasks.
Where sizable differences existed, contextual variables were included in the logistic regression
model to determine whether factors such as gender and age differences of the groups in each
year might account for these differences.
78
Methodology for Chapter 3: Improving
learning outcomes in the ESA region:
Effective country-level practices
Literature review
The literature review we undertook is the main source of information for effective country-level
practices in improving literacy and numeracy learning outcomes of primary-school students in
the ESA region. The literature reviewed was complemented with information gathered during
our stock-taking of assessments (Methodology for Chapter 2), most of which was provided by
UNICEF ESARO and UNICEF country offices. In this section, we describe the literature review
process, which was based on the details outlined in the Inception Report on specific methodology
(ACER, 2014b).
Our search strategy sought to identify published as well as ‘grey’ literature about programmes
that were shown to have had an effect on student learning, as evidenced by learning outcomes
data. It also looked at monographic and demographic studies that used qualitative rather than
quantitative approaches. Literature and relevant databases held by different organisations
working in the region, particularly UNICEF, UNICEF ESARO, DfID and World Bank, were included
in the review. Additionally, peer-reviewed journals and Internet references were used to search for
relevant literature. Strategies that were used in literature searches included the following:
• Electronic searches of bibliographic databases. These initial searches were conducted by
ACER’s experienced information librarians, using our Cunningham Library catalogue and
online search engines such as Google Scholar to search for journal articles and reports
relevant to the scope of the review.
• Targeted searches of online holdings of international/regional agencies, research firms and
national ministries in the region. This included targeting known international, regional and
national agencies that have implemented programmes in improving learning for learners in
the region, particularly the disadvantaged. These included DfID, UNESCO, UNICEF, UNICEF
ESARO, UNICEF Country Offices in the region, and World Bank. Additionally, the publications
of relevant research bodies, such as the Research Triangle Institute (RTI) and International
Initiative for Impact Evaluation (3ie) were include. Websites of national ministries in the
region were also searched for any relevant publications.
• Regional databases such as African Journals Online (AJOL), which offers peer-reviewed
articles from southern scholars, and the Association for the Development of Education in
Africa’s (ADEA) online database.
• Citation chasing. This involved checking the references of relevant publications to identify
possibly relevant literature as well as forward-citation tracking using Scopus, or searching
through the list of papers/studies that cited relevant literature.
• Contacting relevant groups. This entailed collaboration between ACER, UNICEF ESARO and
UNICEF Country Offices to access additional literature and information.
Identification of effective country-level programmes with a focus on improving learning outcomes
of disadvantaged children in the region
79
The main sources for identifying programmes with a focus on improving learning outcomes in
literacy and numeracy of disadvantaged children were the literature review, as well as information
obtained during stock-taking – mainly from UNICEF ESARO and country offices. Any literature
that seemed relevant to the scope of this study was initially included in a large pool. Programmes
were then identified, closely examined and selected for further analysis of effective practices.
To select the programmes, we applied three main principles: The programme (1) aims to improve
learning outcomes in literacy and numeracy of (2) disadvantaged children, and (3) evaluation
mechanisms are place to measure children’s learning outcomes.
In Chapter 3, the definition of ‘disadvantaged children’ is based on the definition used in the
relevant programmes. A comparison of the type of disadvantage addressed in each programme
is provided in the main text of Chapter 3.
80
Appendix II
Detailed tables for Chapter 1: Stocktaking and comparative analysis of
assessments
Table 16. Assessment implementation by type of assessment
Country
International
Angola
Botswana
Burundi
Comoros
Eritrea
Ethiopia
Kenya
Lesotho
Madagascar
Malawi
Mozambique
Namibia
Rwanda
Somalia
South Africa
South Sudan
Swaziland
Tanzania
Uganda
Zambia
Zimbabwe
Total
√
Type of assessments
Regional
National
√
√
√
√
√
√
√
√
√
√
2
√
√
√
√
√
14
EGRA/EGMA
√
√
√
√
√
√
√
√
√
√
√
√
√
√
√
13
√
√
√
√
√
√
√
√
√
√
12
81
Table 17. ESAR countries with limited assessment activity in recent years
Country
Notes
Angola
Only known assessment activity is one EGRA implementation
Burundi
Only known assessment activity is one EGRA implementation and one PASEC
administration
Comoros
Only known assessment activity is one PASEC administration
Eritrea
Only known assessment activity is one MLA implementation
South Sudan No known assessment activity
82
83
√
√
Kenya
Lesotho
Lesotho
Malawi
Malawi
Mozambique Unknown Unknown
NASMLA
LNAEP
Assessment
of Grades 1,
2 and 3 in
Lesotho
Assessing
Learner
Achievement
MLA
National
Assessment
√
√
√
Unknown Unknown
√
√
√
Ethiopia
Unknown
√
Unknown
√
√
√
National
Learning
Assessment
(NLA)
√
Frequency
analyses
conducted/
mean scores
calculated
for cognitive
results,
disaggregated
by contextual
variables of
interest
Eritrea
Competency
levels/
benchmarks
established
MLA
IRT used
National
Country
Assessment
Type of
assessment
Unknown
Unknown
√
√
Frequency
analyses
conducted
on
contextual
data
Unknown
√
Unknown
√
√
Relationship
between
cognitive
performance
and
contextual
factors
explored via
analytical
techniques
Table 18. Analytical techniques used in the assessments from the stock-taking of assessments in ESAR
Unknown
Unknown
√
Trends in
cognitive
performance
computed
Unknown
Unknown
International
comparisons
of cognitive
data
reported
84
Rwanda
Somalia
South Africa
South Africa Unknown Unknown
Uganda
Zambia
Zimbabwe
LARS
MLA
Annual
National
Assessment
NALA
NAPE
NALA
ZELA
√
√
√
√
√
√
√
√
Unknown Unknown
Namibia
Competency
levels/
benchmarks
established
NSAT
IRT used
National
Country
Assessment
Type of
assessment
√
√
√
Unknown
√
√
√
Unknown
Frequency
analyses
conducted/
mean scores
calculated
for cognitive
results,
disaggregated
by contextual
variables of
interest
√
√
Unknown
√
Unknown
Frequency
analyses
conducted
on
contextual
data
√
Unknown
√
Unknown
Relationship
between
cognitive
performance
and
contextual
factors
explored via
analytical
techniques
√
Unknown
Unknown
Trends in
cognitive
performance
computed
Unknown
Unknown
International
comparisons
of cognitive
data
reported
85
Burundi
EGRA
Competency
levels/
benchmarks
established
√
√
Kenya
EGRA
EGRA, EGMA Kenya
√
√
√
√
EGRA
Ethiopia
(System-level
diagnostic) in
2011
√
Unknown
√
Frequency
analyses
conducted/
mean scores
calculated
for cognitive
results,
disaggregated
by contextual
variables of
interest
√
√
Unknown Unknown
IRT used
Ethiopia
EGRA
(Programme
evaluation)
EGRA
Ethiopia
(System-level
diagnostic) in
2010
Angola
EGRA
EGRA/EGMA
Country
Assessment
Type of
assessment
√
√
√
Unknown
√
Frequency
analyses
conducted
on
contextual
data
√
√
√
√
Unknown
√
Relationship
between
cognitive
performance
and
contextual
factors
explored via
analytical
techniques
√
√
√
Unknown
Trends in
cognitive
performance
computed
Unknown
International
comparisons
of cognitive
data
reported
86
√
√
Malawi
EGRA
(Programme
evaluation)
EGRA
Malawi
(System-level
monitoring)
√
√
Mozambique
EGRA
(Programme
evaluation –
APAL)
EGRA, EGMA Rwanda
√
Mozambique
EGRA
(Programme
evaluation
– Literacy
Boost)
√
Unknown
Malawi
Unknown Unknown
EGMA
Frequency
analyses
conducted/
mean scores
calculated
for cognitive
results,
disaggregated
by contextual
variables of
interest
Madagascar
Competency
levels/
benchmarks
established
EGRA
IRT used
EGRA/EGMA
Country
Assessment
Type of
assessment
√
√
√
√
Unknown
Frequency
analyses
conducted
on
contextual
data
√
√
Unknown
Relationship
between
cognitive
performance
and
contextual
factors
explored via
analytical
techniques
√
√
√
Unknown
Trends in
cognitive
performance
computed
Unknown
International
comparisons
of cognitive
data
reported
87
Unknown Unknown
Unknown
Unknown
EGRA
(Programme
evaluation)
EGRA
Zambia
(System-level
diagnostic)
√
EGRA, EGMA Zambia
Unknown Unknown
√
Uganda
EGRA
(Programme
evaluation)
Zambia
√
EGRA
Uganda
(System-level
diagnostic)
Frequency
analyses
conducted/
mean scores
calculated
for cognitive
results,
disaggregated
by contextual
variables of
interest
√
Competency
levels/
benchmarks
established
EGRA, EGMA Tanzania
IRT used
√
EGRA
EGRA/EGMA
Country
Somalia
Assessment
Type of
assessment
Unknown
Unknown
√
√
Frequency
analyses
conducted
on
contextual
data
Unknown
Unknown
√
√
√
Relationship
between
cognitive
performance
and
contextual
factors
explored via
analytical
techniques
Unknown
Unknown
√
Trends in
cognitive
performance
computed
Unknown
Unknown
International
comparisons
of cognitive
data
reported
88
Kenya
Tanzania
Uganda
Burundi
Comoros
Botswana
Kenya
Lesotho
Malawi
Uwezo
Uwezo
Uwezo
PASEC
PASEC
SACMEQ
SACMEQ
SACMEQ
SACMEQ
Regional
Country
Assessment
Type of
assessment
√
√
√
√
√
√
IRT used
√
√
√
√
√
√
Competency
levels/
benchmarks
established
√
√
√
√
√
√
√
√
√
Frequency
analyses
conducted/
mean scores
calculated
for cognitive
results,
disaggregated
by contextual
variables of
interest
√
√
√
√
Frequency
analyses
conducted
on
contextual
data
√
√
√
√
√
√
Relationship
between
cognitive
performance
and
contextual
factors
explored via
analytical
techniques
√
√
√
√
√
√
√
Trends in
cognitive
performance
computed
√
√
International
comparisons
of cognitive
data
reported
89
South Africa √
Swaziland
Tanzania
Uganda
Zambia
Zimbabwe
SACMEQ
SACMEQ
SACMEQ
SACMEQ
SACMEQ
SACMEQ
Total
International
Namibia
SACMEQ
South Africa √
TIMSS
58
Botswana
TIMSS
21
√
South Africa √
PIRLS,
prePIRLS
√
Botswana
PIRLS,
prePIRLS
√
√
√
√
√
√
Mozambique √
SACMEQ
IRT used
Regional
Country
Assessment
Type of
assessment
32
√
√
√
√
√
√
√
√
√
√
√
√
Competency
levels/
benchmarks
established
50
√
√
√
√
√
√
√
√
√
√
√
√
Frequency
analyses
conducted/
mean scores
calculated
for cognitive
results,
disaggregated
by contextual
variables of
interest
19
Frequency
analyses
conducted
on
contextual
data
33
√
√
√
√
√
√
√
√
√
√
√
√
Relationship
between
cognitive
performance
and
contextual
factors
explored via
analytical
techniques
27
√
√
√
√
√
√
√
√
√
√
√
Trends in
cognitive
performance
computed
6
√
√
√
√
International
comparisons
of cognitive
data
reported
Appendix III: Country case
studies
The aim of the two case studies we undertook for Zimbabwe and Rwanda was to obtain a
deeper understanding of specific practices implemented to measure and improve the literacy
and numeracy learning outcomes of primary school children in the long term. Both countries
have developed a national assessment system that provides data on student learning outcomes
in literacy and numeracy, as well as capturing important contextual background information that
allows the exploration of relationships between achievement and context factors. Furthermore,
the implementation of assessment systems in both countries helped local staff acquire the
knowledge and skills for future innovations and developments.
90
Zimbabwe
Country context
Zimbabwe is classified by the World Bank as a low-income country. It has a population of just
over 14 million people in 2013 (World Bank, 2015). Thirty per cent of the population live in urban
areas, the majority in Harare and Bulaywo (UNICEF, 2013b).
The Constitution of Zimbabwe, Amendment No. 20, recognises 16 official languages
(Government of Zimbabwe, 2013). These languages include Chewa, Chibarwe, English, Kalanga,
Koisan, Nambya, Ndau, Ndebele, Shangani, Shona, sign language, Sotho, Tonga, Tswana, Venda
and Xhosa. Amendment No. 20 states that each language must be treated equitably and that
government must create conditions for the development of the official languages.
Education
The Education Act 1987 makes provisions for three languages to be taught in all primary schools
from Grade 1: English, Shona and Ndebele. Primary education is designed to equip learners with
language skills in Shona and English or Shona and Ndebele (UNESCO, 2010). The Education
Amendment Bill 2005 was passed in February 2006 and proposes the teaching of ‘three main
languages of Zimbabwe mainly English, Shona and Ndebele and such other local language in
all schools up to form two on an equal time basis’ (MOESAC, 2006). The bill also states that
prior to Form 1 any language that is best understood by the pupils may be used in instruction
(MOESAC, 2006). Zimbabwe’s formal education structure includes seven years of primary
education (beginning at the age of six and ending at Grade 7), four years of lower secondary
education (Forms 1–4), and two years of upper secondary education (Forms 5–6). As of early
2015, external assessments are conducted in the form of the Grade Seven Certificate at the end
of the primary cycle, the O-Level examination at the end of the lower secondary cycle, and the
A-Level examination at the end of the upper secondary cycle.
91
After gaining independence in 1980, the Government of Zimbabwe expanded access to primary
school education, which resulted in the number of primary school enrolments more than doubling
in seven years. By 1982, primary enrolment rates were reported at almost 100 per cent (Nyanguru
and Peil, 1991). However, between 1982 and 2004 enrolment rates decreased and in 2008 the
provision of education services deteriorated dramatically because of the election period and
hyperinflation. During this time, student attendance fell to around 20 per cent, and teacher
attendance to about 40 per cent (UNICEF, 2008b).
Zimbabwe’s education system was ‘once arguably the best on the continent,’ but since 2000 the
education sector has experienced significant deterioration due to declining financial assistance
(UNICEF, p. 1, 2011). To replace the drop in government funding, a system of fees, levies and
incentives was imposed that has affected access to and quality of education, particularly for the
most disadvantaged children. In addition, the lack of funding has had an effect on school and
learning supervision, availability of planning and policy development related to school and system
governance, teacher in-service training and school environments in general (UNICEF, 2011).
In 2009, the sector slowly began to recover, with education made a priority in the new
government’s Short Term Emergency Recovery Programme (Government of Zimbabwe, 2009).
After a dramatic decrease in primary school completion rates between 1996 (82.6 per cent) and
2006 (68.2 per cent), completion rates rose to 82.4 per cent in 2009 (UNICEF, 2012).
However, there are still significant concerns about the provision of quality education for primary
school children in Zimbabwe. Demographic and Health Survey (DHS) statistics indicate that the
nation’s rural and poor citizens are substantially overrepresented in drop-out and repetition rates
(UNICEF, 2008a). O-level pass rates are still extremely low, and there remains limited access to
important material and non-material resources that support teaching and learning (MOESAC,
2009).
To address these shortcomings, the Ministry of Education, Sport, Arts and Culture (MOESAC)
launched the Education Transition Fund (ETF) in 2009, managed by UNICEF (UNICEF, 2011).77
The purpose of the ETF was to improve the quality of education through the provision and
delivery of essential teaching and learning materials for primary schools, and through high-level
technical assistance to MOESAC. ETF entered its second phase in 2011 with the overall goal of
continued support and revitalisation of the education sector. ETF was renamed the Education
Development Fund (EDF) in 2014.
EDF support in Phase II focuses on activities in the following areas linked with the Ministry
of Education’s Strategic Investment Plan (MOESAC, 2011): School and System Governance;
Teaching and Learning; and Second Chance Education. Key activities within these themes
include strengthening education delivery mechanisms; improving the quality of education
services; improving access, retention, completion and achievement of learners; and a continued
focus on the most vulnerable and out-of-school children (UNICEF, 2014). Access to education
and improvement of student learning outcomes have been confirmed by the 2013 Education
Management System (EMIS), the 2014 Zimbabwe Early Learning Assessment (ZELA) and the
2014 Multiple Indicator Cluster Surveys (MICS) (UNICEF, 2014).
77 In 2014, the Ministry of Education, Sport, Arts and Culture (MoESAC) was renamed the Ministry of Primary and Secondary
Education (MOPSE).
92
This case study explores emerging trends from ZELA in student learning outcomes and the
provision of textbooks and teaching materials procured through EDF. It also reviews the multiyear programme of an intensive capacity-building partnership with the Zimbabwe School
Examinations Council (ZIMSEC) and ACER. The capacity-building programme supports the longterm sustainability of ZELA through system strengthening in assessment, data management and
analysis. Kenneth Russell, EDF Manager at UNICEF Zimbabwe, shared his experience with the
ZELA Capacity-building Programme for this case study.
The latest available UNICEF annual country report for Zimbabwe is 2013. In it, multiple sources of
data suggest that children in Zimbabwe are better off in 2013 than they were in the previous five
years (UNICEF, 2013b). The report notes that there was a 95.6 per cent primary net enrolment
rate and a 52 per cent secondary net enrolment rate. The gender parity index was quoted at
1:01 and the primary completion rate at the time of the report was 86.7 per cent. Access to and
quality of education were reportedly enhanced through the provision of textbooks; training and
supervision of teachers in 35 per cent of primary and secondary schools; and improved water,
sanitation and hygiene (UNICEF, 2013b, p. 1).
In addition to ZELA (2012–2015), Zimbabwe participated in SACMEQ I (1995–1999) and
SACMEQ III (2005–2010). Zimbabwe played a significant role in the eventual development of
SACMEQ. Research generated from a collaboration in 1989 between Zimbabwe’s Minister for
Education and Culture and the Director of IIEP UNESCO led to dialogue that eventually resulted
in the development of the SACMEQ consortium (SACMEQ, 2013).
Zimbabwe Early Learning Assessment
The Zimbabwe Early Learning Assessment (ZELA) is a four-year programme commissioned by
UNICEF to support and enhance the national capacity to review, reform and re-orient the current
system of student assessment in Zimbabwe. It establishes a baseline to help determine whether
the EDF programme (2010-2015) has had the desired effects on children, their caregivers,
schools, and the education sector in general, and it examined the extent to which the changes
identified are attributable to the EDF programme interventions. ZELA’s defined target population
is students beginning Grade 3 of primary school (ACER and ZIMSEC, 2013).
Main purpose and components
The goal of the ZELA project is to monitor and evaluate the effects of the EDF programme
through the introduction of an early–grade learning assessment in language and mathematics.
ZELA measures student performance in language and mathematics. Information is also collected
at the school and student level. School head and pupil questionnaires collect information about
student background, teaching resources, funding and infrastructure.
The test domains are mathematics and language, including English as well as Ndebele and
Shona. Tests were developed in Zimbabwe in February 2012, January 2013 and January 2014,
by panels of ZIMSEC subject specialists and curriculum managers (ACER and ZIMSEC, 2015).
93
Main findings regarding effective strategies and factors
The ACER data, collected from three cycles of the ZELA, indicate socio-economic status is still
a strong predictor of performance, and is associated with large differences in assessment results
across Zimbabwe. Socio-economically advantaged pupils and schools tend to outscore their
disadvantaged peers by larger margins than between any other groups of pupils in English and
mathematics. There are large differences in pupils’ performance between provinces and between
urban and rural areas (ACER and ZIMSEC, 2015).
Key findings include the following:
• The percentage of students performing at or above the grade-appropriate level in English
after completing Grade 2 in Zimbabwe was 49 per cent in 2012, 54 per cent in 2013 and
51 per cent in 2014. The 2014 results were not statistically significantly different from
the previous years. The 2012 base-line study reported that the percentage of students
performing at or above the grade-appropriate level in mathematics after completing Grade
2 in Zimbabwe was 46 per cent. This increased substantially to 63 per cent in 2013 and
again increased significantly to 67 per cent in 2014.
• Girls have continued to outperform boys in English and mathematics from 2012 to 2014.
In 2014, more girls than boys reached the benchmark for English (by 9 percentage points)
and mathematics (by 6 percentage points). From 2012 to 2014, the performance of girls in
English was significantly higher in 2014 than in 2012 (by 3.8 percentage points). There was
a moderate positive trend in mathematics performance for both boys (by 11.9 percentage
points) and girls (by 11.6 percentage points) since 2012. These trends are similar to those
of other southern African nations. Findings indicate that gender differences do not change
much within southern African countries. Where girls perform better they tend to continue
performing better and where boys perform better they tend to continue performing better
(Satio, 2011).
• Students in urban schools significantly outperformed students in rural schools in both
English (by 42 percentage points) and mathematics (by 25 percentage points). More than
eight of 10 urban students reached the benchmark in both English and mathematics, while
only four of 10 rural students reached the English benchmark and six of 10 students
reached the mathematics benchmark.
• Students in registered schools outperform students in satellite schools in both English
and mathematics. Students in registered schools performed better on ZELA 2014 by 18
percentage points in English and 11 percentage points in mathematics.
• Socio-economically advantaged pupils and schools tend to outscore their disadvantaged
peers by larger margins than between any other groups of pupils. The percentage of
students performing at or above grade level in English was 34 per cent for the lowest socioeconomic status (SES) quartile and 77 per cent for the highest SES quartile (a difference
of 43 percentage points). In mathematics the difference was also clear, but smaller in
magnitude: 53 per cent of the low SES pupils performed at or above the grade level and 84
per cent of the high SES pupils—a difference of 31 percentage points (ACER and ZIMSEC,
2015).
Several relationships have been observed between student performance and student background,
teaching or infrastructure variables. These relationships are all correlational, and not necessarily
causal.
94
95
The school-level variance in performance was found to be relatively high, indicating that schools
vary substantially in average student performance. In line with the aims of the EDF programme,
one would expect to see a reduction in the proportion of school level variance over the EDF
programme cycle (ACER and ZIMSEC, 2015).
ZELA is in its evaluation phase in 2015, and it is too early to draw conclusions from the study
beyond some of the indicative trends noted earlier. The EDF programme distributed textbooks
and teaching materials to all schools in Zimbabwe. Based on the relatively low base some pupils
may be starting from, combined with increasing exposure to reading materials, one would expect
to see long-term advancements in pupil performance over the EDF programme cycle.
ZELA Capacity-building Programme
ZELA also targets system-level capacity. One of the key components of ZELA has been to
support and enhance national capacity in student assessment. In 2012, ACER worked with
ZIMSEC to construct four tests and two surveys. This activity was followed by the administration
of these tools in 500 schools in Zimbabwe and the analysis, standardisation and reporting of
pupil achievement levels in Zimbabwe through the ACER and ZIMSEC partnership. Training in
assessment and data analysis were conducted in 2012, and ZIMSEC took increasing responsibility
for these activities in each subsequent cycle of ZELA.
In 2013, ZIMSEC indicated that the training needs of its staff include the following topics:
• Analysis of the relationships between student background characteristics, teaching and
learning, and funding and facilities on pupil performance (using SPSS statistical analysis
software);
• Intensive and practical training on IRT (including use of ACER ConQuest);
• Knowledge and skills of school-based assessment (in theory and practice).
In 2014, an SPSS Roundtable was organized to reinforce the 2013 and 2014 capacity-building
activities and to ensure ZIMSEC colleagues were fundamentally involved in the analysis and
drafting of ZELA data. Intensive and practical training on IRT (including the use of ACER ConQuest
software) was provided during a three-week training programme, as well as technical assistance
on IRT. ACER also received a ZIMSEC delegation in Australia and introduced key ZIMSEC staff
to school-based assessment (SBA) prior to 2015 capacity-building activities in this focus area.
In 2015, the focus of capacity-building activities with ZIMSEC is SBA. SBA activities include
facilitated workshops with ZIMSEC and key government stakeholders, and a pilot research
project with schoolteachers in Zimbabwe. As with 2013 and 2014, an SPSS Roundtable will
be conducted with ZIMSEC during the Impact Evaluation report-writing stage. Similarly, an IRT
Roundtable will be conducted with ZIMSEC in order to build on the technical assistance and
workshops provided in previous ZELA cycles.
The expanded activities also include the placement of a technical assistance officer within ZIMSEC
for up to two months per year (Kenneth Russell, UNICEF Zimbabwe, personal communication,
16 April 2015).
96 96
ZELA Capacity-building Programme: Experience of
the EDF manager
Kenneth Russell, EDF Manager at UNICEF Zimbabwe, shared his experience with the ZELA
Capacity-Building Programme. The following is a summary of his responses to questions about
the implementation of the capacity-building support, along with some success stories and an
outline of the challenges encountered.
How is the ZELA Capacity-Building Programme implemented?
Most of the capacity-building activities were (or plan to be) delivered through facilitated
workshops. However, the placement of the technical assistance officer is different, and is one
of the distinctive strategies used by ZELA to help with capacity-building. It not only allows for
ready access and sustained support but would have helped to deepen relationships between the
technical officer and ZIMSEC as well as strengthen partnership among the entities (which might
survive beyond the project).
What are some success stories from ZELA’s Capacity-Building
Programme?
It is difficult to provide success stories from the Capacity-Building Programme without an
assessment of the effect of the support for capacity-building that has been provided. What we
know from our discussions with and the work of ZIMSEC is the following:
• ZIMSEC played a greater role in the analysis of 2014 data than they had done previously, as
well as the preparation of the report. This is due in part to the support they have received
in data analysis.
• ZIMSEC has spoken publicly about their increased capacity in IRT. This is a new area of
work and a new approach to analysis for ZIMSEC, but one which they are interested in
continuing to use for ZELA and their other assessments.
• ZIMSEC is at the forefront of national discussions on continuous assessment, and SBA
specifically, because of the support provided to them through ZELA. They were exposed to
good practices in Australia and had opportunities to reflect on how to apply some of these
lessons to Zimbabwe.
• ZELA has provided opportunities for ZIMSEC, as well as provincial and district staff
and teachers, to engage in developing items for the assessment. This helped to deepen
understanding of the participants, and helped the organisation to grow in how it designs
items for other assessment. A critical aspect of this area of capacity-building is the diversity
of those participating and hence the potential for domino effect in the system.
• Institutional capacity has also been enhanced through the provision of software such as
SPSS and ACER ConQuest, computers and motor vehicles. These enhance the organisation’s
access to technology to support its work, as well as its ability to monitor field activities and
supervise staff.
97 97
What are the main barriers you have encountered that limit
sustainable capacity-building, and what ways were considered to
overcome these barriers?
The major barrier to sustainable capacity-building is the ‘projectised’ approach taken with ZELA.
While necessary to test and experiment before making it institutional, such a critical project
creates expectations and practices that might not be sustainable when mainstreamed. This
project approach also resulted in the capacity-building activities being viewed as parallel to or
outside the normal functioning of the organisation. In so doing, capacity is built primarily in those
who are involved in the project despite their applicability and relevance to other aspects of the
organisation. ZELA invested heavily in a small number of core staff who have done great work,
but the effect of them leaving ZIMSEC would be potentially catastrophic for ZELA.
Another challenge to sustainability of built capacity is the concentration of investment in
capacity-building within ZIMSEC to the exclusion of other organisations that will be critical to
sustainability in the years ahead. While ZIMSEC has done a great job, during institutionalisation,
the implementation arrangements could be different. In such a case, there could be new players
playing critical roles for which they have not had the required capacity-building.
These barriers are the focus of the final year of ZELA as a project. Much of it will depend on the
institutional arrangements agreed for ZELA beyond the current phase (up to end of 2015).
We thank Kenneth Russell for sharing his experiences for this research.
98 98
Rwanda
Country context
Rwanda is the most densely populated country in Africa, with a population of over 11 million. Half
of its citizens are under the age of 18. Despite the country’s impressive economic growth over
the past two decades, since the 1994 genocide Rwanda remains one of the poorest countries
in the world, with 44 per cent of the population living below the poverty line. Approximately
80 per cent of the population live in rural areas. With increasing urbanisation since 1994, the
urban population is expected to grow to 30 per cent of the population by 2020. One of the main
development goals set out in Rwanda’s Vision 2020 and Economic Development and Poverty
Reduction Strategy is to move from an agriculture-based economy to ‘a knowledge-based hub
for business and information technology’ by 2020.78
Education
Primary education in Rwanda starts at the age of seven and comprises six years. Together with
three years of lower secondary education, Rwanda has nine years of compulsory basic education
(Rwanda Education Ministry, 2014, p. 1). In 2011, a strategy was launched to expand access to
education from nine to 12 years of basic education (UNICEF, 2013a, p. 2). The transition from
primary to lower secondary education is based on a national examination at the end of primary
education (Rwanda Education Ministry, 2014, p. 1).
78http://www.unicef.org/rwanda/overview.html
99
Equitable access to education and high-quality education are priorities for the government of
Rwanda, which aims to provide its citizen with the skills and knowledge required for the socioeconomic development of the country (Rwanda Education Ministry, 2014, p. 1). Since the school
year 2003/04, fees from primary to secondary education have been gradually abolished in an
effort to increase enrolment, retention and completion rates for basic education, especially for
vulnerable children (Rwanda Education Board, 2012, p. 11). Rwanda is one of the few African
countries on track to achieve seven of the eight Millennium Development Goals, one of which is
universal access to primary education by 2015.79 In 2013, primary school enrolment in Rwanda
reached 97 per cent (98 per cent for girls). However, the primary education completion rate
was still low in 2013 at 69 per cent (64 per cent for boys and 74 per cent for girls) (Rwanda
Education Ministry, 2014, pp. 12, 14).
The large increase in enrolment numbers poses enormous challenges for the education system,
especially for the provision of adequate learning spaces in primary education (Rwanda Education
Board, 2012, p. 11).
Another key challenge for Rwanda’s education system is improving the quality of education.
The government addresses the remaining disparities in access to education and improvement of
education quality in the Education Sector Strategic Plan (ESSP) for 2013/14–2017/18. The plan
was developed in consultation with UNICEF and other development partners. The plan focuses on
reducing the dropout rate, and improving access and retention for the most vulnerable children,
including children with special needs (UNICEF, 2013a, p. 21). To improve the quality and relevance
of education, the strategic priorities are curriculum development, quality standards, assurance
and assessments, textbook distribution, improving teaching and learning, and implementation
of a system for monitoring learning achievement at school level and national level (UNICEF,
2013a, p. 22). Key elements of UNICEF’s programme to support the government of Rwanda in
its strategy to increase quality education are curriculum review, teacher development and use of
learning achievement assessments (UNICEF, 2013a, p. 2).
Learning Achievement in Rwandan Schools (LARS)
An important development regarding the quality standards and assurance programme of education
in Rwanda is the 2011 introduction of Learning Achievement in Rwandan Schools (LARS).
Main purposes and components
The main purposes of LARS are to measure the level of achievement in literacy and numeracy
at the national level in order to determine factors associated with student achievement –
especially low achievement – and to monitor achievement over time. As a monitoring tool, LARS
provides the Ministry of Education with a reliable database on learning outcomes as a basis
for recommendations to policymakers and other stakeholders for future improvement (Rwanda
Education Board, 2012, p. 12).
79<http://www.unicef.org/rwanda/overview.html>.
100
To achieve these goals, LARS measures student achievement in literacy and numeracy at Grade
3 level in public schools, government-aided schools and private schools. Capturing completion of
the lower primary level, the target population consisted of students who had completed Grade 3
and were in the second term of Grade 4 (Rwanda Education Board, 2012, p. 13).
The literacy component of LARS focuses on writing and reading skills in the Kinyarwanda
language. The numeracy component captures skills in numeration and operations, the metric
system, and geometric figures (shapes), in conformity with guidelines from the national
mathematics curriculum (UNICEF, 2013a, p. 19). In order to identify the relevant indicators and
factors related to low-learning achievement in Rwandan schools, background data were collected
through questionnaires for students, parents, teachers and school administrators.
Main findings regarding effective strategies and factors
The LARS baseline report is based on a national representative sample of approximately 2,500
students in 60 schools across Rwanda (Rwanda Education Board, 2012, p. 15).
Key findings include the following:
• A significant percentage of students fail to meet curricular expectations: 37 per cent in
literacy and 46 cent in numeracy, compared to 55 cent of students meeting the expectations
in literacy and 27 cent in numeracy (Rwanda Education Board, 2012, p. 41). The percentage
of students failing to meet curricular expectations for numeracy is thus higher and more
variable than observed for literacy.
• Numeracy results vary significantly between provinces and between districts (Rwanda
Education Board, 2012, p. 55). Significant differences between some of the districts are
also reported for literacy (Rwanda Education Board, 2012, p. 43).
• Students in rural areas are disadvantaged in meeting curricular standards compared with
their peers in urban areas (Rwanda Education Board, 2012, p. 46). Achievement distribution
in both literacy and numeracy is relatively equal for girls and boys (Rwanda Education
Board, 2012, pp. 47, 55).
• A major impact on school level is also made through higher performing children of highincome parents.
• Another factor influencing achievement is the average teachers ‘or head teachers’ years
of experience (Rwanda Education Board, 2012, pp. 51, 61). Interestingly, students with
teachers with the least experience appear to perform better. One likely explanation
mentioned in the report is that new teachers are significantly more skilled than previous
cohorts, in the light of the rapid expansion of the Rwandan education system in response
to rapid increases in enrolment (Rwanda Education Board, 2012, p. 60).
Two main shortcomings affect the analysis of parent and classroom characteristics. First, the
response rate among parents is very low. Second, student data were not linked to teacher
and parent data at an individual level. Thus, students can only be linked through the average
of teacher and parent characteristics for their school. Results concerning the relationships of
parent and classroom background characteristics and student achievement therefore need to be
interpreted with caution (Rwanda Education Board, 2012, p. 49).
101
The LARS report mentions several reasons for the poor performance of students measured in
LARS 2011 (Rwanda Education Board, 2012, p. 41). One important factor is that ‘the children
being tested were born either during or immediately after the civil war, a period when parents’
attention was focused largely on matters of survival’ (Rwanda Education Board, 2012, p. 41).
Another challenge for the education system and classroom management in particular is the
rapidly growing enrolment rate, with high growth in overage children and children from low socioeconomic backgrounds as well as rural areas.80 Language of instruction (transition to English)
versus language spoken at home by the student and the teacher, and the language of test
(Kinyarwanda) are also seen as important, but have not been captured or analysed in LARS
(Rwanda Education Board, 2012, p. 41).
To improve student-learning outcomes in primary education in Rwanda, one major area that
needs to be addressed is the number of students failing curricular expectations in literacy and
numeracy. Another area is performance differences between districts. Both can be improved
by providing resources to low performing students, and schools and districts with the highest
proportion of low-performing students (Rwanda Education Board, 2012, p. 64).
The LARS report underlines the importance of further research to investigate and explain
the determinants of achievement, especially of low achievement. One example is identifying
cognitive strategies that need to be strengthened through in-depth analysis of items that students
consistently fail to resolve. This includes letter and number recognition, receptive vocabulary,
phonetic accuracy and fluency in reading components. This would help describe more precisely
students’ missing prerequisite literacy skills (Rwanda Education Board, 2012, p. 65).
In order to allow for the measurement of achievement and the relationships with important
background characteristics over time, LARS is implemented periodically in a three-year-cycle. An
innovative way of linking LARS with international benchmarks would be to include test items
from other regional or international assessments, as suggested in the LARS report (Rwanda
Education Board, 2012, p. 66)
LARS capacity-building component
One important accomplishment during the development of LARS was capacity-building. A team
at the Rwanda Education Board (drawn from various departments of REB, districts and school
teachers) was trained to design and conduct learning assessments. Important skills they acquired
include item/tools development, test administration, coding of questionnaires, data entry and
data analysis.
Both Rwanda and Zimbabwe focus on assessment design, monitoring and evaluation, gathering
background data at individual, school and system levels, and conducting innovative capacitybuilding programmes at the system level. Both ZELA and LARS include a development programme
to improve the capacity of education staff to analyse student learning outcomes. In addition,
both programmes appear to support innovation in curriculum reform.
80 In Rwanda, the Primary Net Enrolment Rate (NER) rose from 92.9 per cent in 2009 to 96.6 per cent in 2013, with the largest
increase between 2009 (92.9 per cent) and 2010 (95.4 per cent) (see Rwanda Education Ministry, 2014, p. 12).
102
Appendix IV: Detailed tables
and figures for Chapter 2:
Literacy and numeracy in
primary education in the ESA
region—students experiencing
LLOs and trends over time
Characteristics of students with LLOs
in literacy and numeracy in primary
education in the ESA region
The results of the comparison between the contextual profile of all Uwezo participants and the
contextual profile of students experiencing LLOs are presented below for each country, as well as
for each domain assessed in each country (see Table 19). For example, in Kenya 50 per cent of all
students in the population are female. However, of the sub-population of students who identified
as experiencing LLOs in English, 48 per cent were female. This means the proportion of females
experiencing LLOs is smaller than the proportion of females in the population, suggesting that
males are more likely to be experiencing LLOs. A bolded percentage indicates that the difference
is significant (in this instance, it means that the logistic regression with gender as the independent
variable and LLOs for English as the dependent variable was significant).
The results of the comparison between the contextual profile of all PASEC participants and the
contextual profile of students experiencing LLOs are presented below for each country, as well
as for each year level and each domain (see Table 20).
The results of the comparison between the contextual profile of all TIMSS students (Botswana)
and prePIRLS participants (Botswana and South Africa) and the contextual profile of students
experiencing LLOs are presented below (see Table 21 and Table 22).
103
Table 19a. Proportions of students experiencing LLOs for each contextual variable of
interest for Uwezo countries (Kenya)
Uwezo (2012)
Contextual variable
All
Gender
Females
Age
6 to 9
10 to 13
14 to 16
Socio-economic factors
Type of wall at home – Polythene
Type of wall at home - Iron sheet
Type of wall at home – Timber
Type of wall at home - Stone/Bricks
Type of wall at home – Mud
Type of wall at home - Burnt Bricks
Type of wall at home - Cement Bricks
Type of wall at home – Other
Household has access to electricity
Household owns a TV
Household owns a radio
Household owns a phone
Household has direct access to clean water
Household owns a car
Household owns a fridge
Household owns a motorbike
Mother’s level of education – None
Mother’s level of education - Some primary
Mother’s level of education - Some
secondary
Mother’s level of education - Post secondary
Out of school lessons
Child receives extra lessons/tuition
School type and school location
School type – Public
School type – Private
School type – Other
School Location – Urban
104
Kenya
English
Swahili
LLO
LLO
Maths
LLO
50%
48%
47%
49%
39%
43%
18%
34%
54%
12%
35%
54%
11%
32%
54%
14%
<1%
9%
11%
28%
50%
<1%
8%
11%
23%
58%
<1%
8%
11%
23%
58%
<1%
8%
12%
24%
55%
23%
28%
73%
70%
24%
16%
21%
70%
65%
18%
16%
21%
70%
65%
19%
19%
24%
71%
66%
20%
18%
58%
22%
21%
62%
16%
21%
62%
16%
21%
59%
19%
2%
1%
1%
1%
52%
43%
43%
46%
83%
16%
1%
89%
10%
1%
89%
10%
1%
89%
10%
1%
Table 19b. Proportions of students experiencing LLOs for each contextual variable of
interest for Uwezo countries (Tanzania)
Uwezo (2012)
Contextual variable
All
Gender
Females
50%
Age
6 to 9
35%
10 to 13
50%
14 to 16
16%
Socio-economic factors
Type of wall at home – Polythene
Type of wall at home - Iron sheet
Type of wall at home – Timber
Type of wall at home - Stone/Bricks
Type of wall at home – Mud
47%
Type of wall at home - Burnt Bricks
38%
Type of wall at home - Cement Bricks
12%
Type of wall at home – Other
3%
Household has access to electricity
17%
Household owns a TV
19%
Household owns a radio
66%
Household owns a phone
49%
Household has direct access to clean water
29%
Household owns a car
3%
Household owns a fridge
7%
Household owns a motorbike
9%
Mother’s level of education – None
20%
Mother’s level of education - Some primary
74%
Mother’s level of education - Some
5%
secondary
Mother’s level of education - Post secondary
<1%
Out of school lessons
Child receives extra lessons/tuition
School type and school location
School type – Public
97%
School type – Private
3%
School type – Other
School Location – Urban
17%
Tanzania
English
Swahili
LLO
LLO
Maths
LLO
50%
49%
50%
14%
67%
19%
16%
68%
16%
18%
68%
14%
49%
39%
10%
3%
15%
16%
65%
47%
27%
2%
6%
8%
21%
74%
4%
51%
39%
8%
2%
13%
15%
64%
44%
26%
2%
5%
8%
23%
73%
4%
53%
38%
7%
2%
12%
13%
62%
42%
24%
2%
4%
7%
26%
71%
3%
<1%
<1%
<1%
99%
1%
98%
2%
99%
1%
15%
13%
12%
105
Table 19c. Proportions of students experiencing LLOs for each contextual variable of
interest for Uwezo countries (Uganda)
Uwezo (2012)
Contextual variable
Gender
Females
Age
6 to 9
10 to 13
14 to 16
Socio-economic factors
Type of wall at home – Polythene
Type of wall at home - Iron sheet
Type of wall at home – Timber
Type of wall at home - Stone/Bricks
Type of wall at home – Mud
Type of wall at home - Burnt Bricks
Type of wall at home - Cement Bricks
Type of wall at home – Other
Household has access to electricity
Household owns a TV
Household owns a radio
Household owns a phone
Household has direct access to clean water
Household owns a car
Household owns a fridge
Household owns a motorbike
Mother’s level of education – None
Mother’s level of education - Some primary
Mother’s level of education - Some
secondary
Mother’s level of education - Post secondary
Out of school lessons
Child receives extra lessons/tuition
School type and school location
School type – Public
School type – Private
School type – Other
School Location – Urban
106
All
Uganda
English LLO
Maths LLO
49%
50%
50%
40%
41%
18%
20%
61%
20%
19%
60%
21%
12%
11%
72%
65%
12%
10%
9%
70%
62%
11%
11%
10%
71%
63%
11%
16%
69%
12%
17%
70%
10%
17%
70%
11%
3%
2%
2%
74%
26%
80%
20%
78%
22%
Table 20a. Proportions of students experiencing LLOs for each contextual variable of
interest for PASEC countries (Comoros)
PASEC (2008/2009)
Comoros
Grade 2
Contextual variable
Gender
Females
Age
Below normal age (5 for Grade 2, 9 for
Grade 5)
Above normal age (8 for Grade 2, 11 for
Grade 5)
Language spoken at home
Student speaks Shikomori at home
Student speaks Arabic at home
Student speaks French at home
Student speaks English at home
Student speaks Kirundi at home
Student speaks Swahili at home
Student speaks another language at home
Socio-economic factors
Home possession scale - Less than 3
Home possession scale - Between 3 and 5
Home possession scale - 6 or more
Father’s literacy
Mother’s literacy
Participates in farm work
Participates in house work
Participates in retail work
Work hinders students study at home
Work hinders student’s ability to go to
school
Work hinders student’s concentration at
school
School resources
Presence of school library
Presence of computer room
Presence of school toilets
School has electricity
School has drinking water facilities
Grade 5
All
French
LLO
Maths
LLO
All
French
LLO
Maths
LLO
52%
52%
55%
59%
70%
63%
0%
0%
0%
2%
1%
1%
36%
37%
36%
52%
55%
55%
97%
4%
4%
1%
97%
5%
3%
2%
91%
5%
3%
3%
96%
6%
6%
1%
97%
6%
4%
2%
97%
6%
3%
2%
2%
1%
7%
1%
2%
2%
52%
34%
14%
68%
61%
59%
54%
16%
24%
60%
32%
8%
65%
52%
61%
60%
19%
23%
52%
35%
14%
66%
58%
60%
56%
26%
17%
50%
34%
16%
64%
57%
59%
70%
14%
20%
58%
31%
11%
60%
56%
65%
72%
20%
21%
57%
34%
10%
64%
60%
65%
76%
19%
23%
22%
28%
22%
13%
13%
12%
15%
21%
20%
14%
20%
16%
9%
1%
77%
21%
66%
10%
<1%
75%
16%
63%
9%
<1%
74%
16%
62%
12%
2%
80%
27%
71%
14%
<1%
73%
14%
68%
13%
2%
72%
16%
62%
107
Table 20b. Proportions of students experiencing LLOs for each contextual variable of interest
for PASEC countries (Burundi)
PASEC (2008/2009)
Burundi
Grade 2
Contextual variable
All
Gender
Females
49%
Age
Below normal age (5 for Grade 2, 9 for
0%
Grade 5)
Above normal age (8 for Grade 2, 11 for
76%
Grade 5)
Language spoken at home
Student speaks Shikomori at home
Student speaks Arabic at home
Student speaks French at home
2%
Student speaks English at home
Student speaks Kirundi at home
95%
Student speaks Swahili at home
4%
Student speaks another language at home
1%
Socio-economic factors
Home possession scale - Less than 3
89%
Home possession scale - Between 3 and 5 11%
Home possession scale - 6 or more
<1%
Father’s literacy
60%
Mother’s literacy
51%
Participates in farm work
54%
Participates in house work
78%
Participates in retail work
10%
Work hinders students study at home
23%
Work hinders student’s ability to go to
16%
school
Work hinders student’s concentration at
18%
school
School resources
Presence of school library
3%
Presence of computer room
<1%
Presence of school toilets
92%
School has electricity
5%
School has drinking water facilities
34%
108
Grade 5
French
LLO
Kirundi
LLO
Maths
LLO
All
French
LLO
Maths
LLO
47%
46%
50%
48%
47%
56%
0%
0%
0%
0%
0%
0%
71%
71%
64%
90%
92%
93%
1%
1%
2%
2%
2%
1%
97%
3%
1%
95%
5%
<1%
96%
3%
1%
95%
4%
<1%
88%
3%
<1%
91%
3%
1%
88%
12%
0%
60%
49%
53%
77%
12%
19%
87%
13%
0%
61%
49%
53%
75%
13%
20%
88%
12%
0%
62%
51%
55%
81%
12%
21%
91%
9%
<1%
54%
43%
62%
80%
9%
24%
91%
8%
1%
52%
40%
58%
73%
7%
22%
91%
8%
1%
52%
42%
59%
75%
6%
24%
16%
16%
15%
19%
14%
14%
16%
19%
22%
15%
13%
15%
2%
<1%
93%
2%
34%
2%
<1%
93%
2%
35%
2%
<1%
95%
3%
36%
2%
<1%
92%
7%
40%
2%
<1%
89%
7%
35%
3%
1%
87%
6%
33%
Table 21. Proportions of students experiencing LLOs for each contextual variable of
interest for TIMSS
TIMSS (2011)
Botswana
Contextual variable
Grade 6
All
Maths
LLO
51%
46%
12 years or less
21%
11%
12 years but < 14 years
66%
68%
14 years or older
12%
22%
78%
69%
Home resources for learning - Many resources
1%
0%
Home resources for learning - Some resources
57%
44%
Home resources for learning - Few resources
42%
56%
Parent highest education level - University or higher
10%
3%
Parent highest education level - Post-secondary non-university
16%
8%
Parent highest education level - Upper secondary
13%
11%
Parent highest education level - Lower secondary
17%
19%
Parent highest education level - Some primary, lower secondary or no school
41%
58%
Attended a preschool
45%
32%
Numeracy activities prior to primary school - Often
18%
11%
Numeracy activities prior to primary school - Sometimes
53%
54%
Numeracy activities prior to primary school - Never or almost never
28%
35%
Numeracy competency when beginning primary school - Very well
14%
6%
Numeracy competency when beginning primary school - Moderately well
75%
68%
Numeracy competency when beginning primary school - Not well
11%
16%
Presence of school library
50%
45%
Presence of computers for instruction
70%
68%
74%
65%
Gender
Females
Age
Language spoken at home
Speaks test language at home
Socio-economic factors
Learning activities prior to attending school
School resources
School type and school location
Urban setting (Urban, suburban, medium-size city, small town)
109
Table 22. Proportions of students experiencing LLOs for each contextual variable of
interest for prePIRLS (2011)
Contextual variable
Gender
Females
Age
12 years or less
12 years but < 14 years
14 years or older
Language spoken at home
Speaks test language at home
Socio-economic factors
Home resources for learning - Many resources
Home resources for learning - Some resources
Home resources for learning - Few resources
Parent highest education level - University or higher
Parent highest education level - Post-secondary nonuni
Parent highest education level - Upper secondary
Parent highest education level - Lower secondary
Parent highest education level - Some primary, lower
secondary or no school
Learning activities prior to attending school
Attended a preschool
Literacy activities prior to primary school - Often
Literacy activities prior to primary school - Sometimes
Literacy activities prior to primary school - Never or
almost never
Literacy competency when beginning primary school Very well
Literacy competency when beginning primary school Moderately well
Literacy competency when beginning primary school Not well
Student engagement in reading lessons - Engaged
Student engagement in reading lessons - Somewhat
engaged
Student engagement in reading lessons - Not engaged
110
Botswana
Grade 6
Reading
All
LLO
South Africa
Grade 6
Reading
All
LLO
50%
31%
48%
36%
30%
63%
7%
19%
68%
12%
36%
54%
10%
31%
53%
16%
74%
73%
91%
88%
1%
62%
38%
9%
0%
53%
47%
3%
2%
65%
33%
10%
<1%
61%
39%
3%
16%
7%
17%
11%
14%
20%
10%
24%
38%
14%
41%
16%
38%
53%
19%
28%
46%
14%
76%
33%
8%
78%
83%
34%
62%
79%
30%
65%
10%
14%
4%
5%
25%
14%
31%
27%
43%
40%
44%
43%
32%
46%
25%
31%
25%
9%
47%
29%
58%
63%
45%
55%
18%
28%
8%
15%
Contextual variable
Botswana
Grade 6
Reading
All
LLO
School resources
Presence of school library
Presence of computers for instruction
School type and school location
Urban setting (Urban, suburban, medium-size city,
small town)
South Africa
Grade 6
Reading
All
LLO
49%
61%
40%
56%
41%
48%
28%
40%
75%
67%
58%
49%
Trends in literacy and numeracy learning outcomes
of children in primary education in the ESA region
The following figures show trends in performance for Swahili in Kenya (see Figure 8), mathematics
in Kenya (see Figure 9), English in Tanzania (see Figure 10), Swahili in Tanzania (see Figure 11),
English in Uganda (see Figure 12) and mathematics in Uganda (see Figure 13).
Figure 8. Trends in Swahili performance across time for students in Kenya (Uwezo)
Swahili performance in Kenya (Uwezo)
100%
90%
80%
70%
60%
8
7
8
10
11
11
15
14
14
15
13
13
50%
Syllables
Words
40%
30%
Nothing
Paragraph
52
53
53
2009-10
2011
2012
Story
20%
10%
0%
111
Figure 9. Trends in Mathematics performance across time for students in Kenya (Uwezo)
Mathematics performance in Kenya (Uwezo)
100%
90%
4
4
18
17
10
10
10
10
11
11
80%
70%
60%
50%
Nothing
Counting
Addition
Subtraction
40%
Multiplication
30%
20%
48
48
2011
2012
Division
10%
0%
Figure 10. Trends in English performance across time for students in Tanzania (Uwezo)
English performance in Tanzania (Uwezo)
100%
90%
80%
34
33
29
70%
60%
50%
40%
30%
Nothing
21
27
12
14
13
Paragraph
13
8
10
Story
20
20
21
2009-10
2011
2012
0%
112
Letters
Words
20%
10%
24
Figure 11. Trends in Swahili performance across time for students in Tanzania (Uwezo)
Swahili performance in Tanzania (Uwezo)
100%
90%
80%
70%
15
22
18
18
60%
10
50%
12
40%
18
22
11
12
10
11
30%
20%
Nothing
Letters
Words
Paragraph
Story
45
38
36
2011
2012
10%
0%
2009-10
Figure 12. Trends in English performance across time for students in Uganda (Uwezo)
English performance in Uganda (Uwezo)
100%
90%
22
24
23
27
25
80%
70%
60%
24
50%
40%
23
30%
20%
10%
14
17
16
10
15
9
23
27
2011
2012
Nothing
Letters
Words
Paragraph
Story
0%
2009-10
113
Figure 13. Trends in mathematics performance across time for students in Uganda
(Uwezo)
Mathematics performance in Uganda (Uwezo)
100%
90%
10
80%
16
70%
8
60%
50%
40%
12
14
9
30%
20%
10%
10
22
10
12
11
7
10
24
Counting
Numbers
12
Values
10
Addition
11
Subtraction
7
Multiplication
31
32
26
2009-10
2011
2012
0%
114
Nothing
115
Programme
Ethiopia
Malawi
Mozambique
Kenya
1
2
3
4
Reading to Learn
Literacy Boost (as
part of the Early
Literacy project in
Mozambique)
Literacy Boost
(as part of the
Sponsorship
Basic Education
Programme)
Literacy Boost
Early grade literacy/numeracy programmes
Country
Aga Khan
Foundation
Save the Children
Save the Children
Save the Children
Implemented by
Evaluation
Including children
affected by HIV/
AIDS and other
vulnerable children
in economically
disadvantaged
areas
Not specified
Randomized field
experiments report
(Lucas et al., 2014)
End-line report
(Mungoi et al., 2010)
Year 2 report (Dowd
and Mabeti, 2011)
Children of low SES End-line II report
(Friedlander et al.,
2012)
Disadvantaged
children targeted
Aga Khan Foundation Children with
low learning
achievements
in economically
disadvantaged
districts
Information not
available (private
donor)
Information not
available
Save the Children
Funded by
Table 23. Example programmes in ESAR with focus on improving learning outcomes in literacy and numeracy of disadvantaged
children in primary education
Appendix V: Detailed table for Chapter 3 –
country level practices
116
Kenya
Malawi
6
7
South Africa
Mozambique
Rwanda
9
10
Early Literacy and
Maths Initiative
(ELMI)
Early Childhood
Development (ECD)
programme
Disadvantaged
children targeted
ECD programme
implementation
was supported by
America Gives Back
and The ELMA
foundation
DfID, Innovation for
Education Fund
Save the Children
DfID
USAID
USAID
Preschool children,
including children
in remote areas
without access to
ECD programme
Young children
in communities
affected by HIV/
AIDS
Economically
disadvantaged rural
areas
Not specified
Marginalised
children in slums
and non-formal
settlements
Aga Khan Foundation Economically
disadvantaged
districts
Funded by
Save the Children
JET’s School
JET Education
Improvement
Services
Programme (Khanyisa
Education Support
Project)
Early Childhood Development programmes
8
RTI
Aga Khan
Foundation
Implemented by
Malawi Teacher
Creative Associates,
Professional
RTI, Seward Inc.
Development Support
(MTPDS)
PRIMR (Primary
Math and Reading
Initiative)
Reading to Learn
Programme
School improvement programmes
Uganda
5
Country
Rwanda mid-line
report (Save the
Children, 2014)
Randomized Impact
Evaluation (Martinez
et al., 2012)
Sustainable School
Improvement, report
(JET Education
Services, n.d.)
Project Monitoring
and Evaluation report
(Randolph et al.,
2013)
Final report (RTI,
2014)
Randomized field
experiments report
(Lucas et al., 2014)
Evaluation
Appendix VI: Main stock-taking
table
The main stock-taking table lists all assessments identified during this study that assess
student learning outcomes in literacy and numeracy in primary education in the ESA region.
The assessments are presented in the table according to the framework categories described in
Chapter 1.
Table 24: Main stock-taking table
NOTES:
Initiatives that have an asterisk (*) in the Name column are ones whose data are used in the
analysis discussed in Chapter 2.
The SACMEQ website (SACMEQ, 2013) was offline for the duration of this consultancy.
The information about SACMEQ is derived from materials downloaded from the site before
it went offline.
In a number of countries in ESAR, multiple implementations of EGRA/EGMA have been
conducted. In such cases, the table only includes several indicative implementations for
which complete or near complete documentation is available on the EdData website (see RTI
(n.d.)), the main repository for EGRA/EGMA information.
N/A indicates ‘Not applicable.
1. Angola
2. Bostwana
3. Burundi
4. Comoros
5. Eritrea
6. Ethiopia
5
7. Kenya
8. Lesotho
6
9. Madagascar
18
13
3
7
14
10. Malawi
11. Mozambique
12. Namibia
17
1
12
10 11
19
20
14. Somalia
4
15. South Africa
9
16. Swaziland
17. Tanzania
2
15
13. Rwanda
16
8
18. Uganda
19. Zambia
20. Zimbabwe
117
Angola
Name
EGRA
Organisation
/ institution
responsible
World Bank
Purpose
System-level diagnostic
Inception
2010
Frequency
N/A (one-off)
Target Population
Grade 3
Sample
139 schools, aiming for 36 students per school
Nationally representative
Cognitive domains
Reading (Portuguese)
Contextual
instruments
Student questionnaire
Teacher questionnaire
School head questionnaire
Parent questionnaire
Test Administration
School-based
One-on-one administration
Oral administration
Analysis
IRT analysis not used to scale cognitive data
Frequency analyses conducted/mean scores calculated for cognitive
results, disaggregated by contextual variables of interest
Frequency analyses conducted on contextual data
Relationship between cognitive performance and contextual factors
explored via analytical techniques (eg correlation, regression, multilevel
modelling)
Reporting and
dissemination
Reports not publicly available
Documents: (Ministry of Education of Angola, World Bank, & Russia Education Aid for Development
Program, 2011)
118
Botswana
Name
SACMEQ
Organisation
/ institution
responsible
SACMEQ
Purpose
System-level monitoring
Inception
SACMEQ I: 1995–1999
Frequency
5–6-year cycle; Botswana participated in SACMEQ II & III
Target Population
Grade 681
Sample
SACMEQ III: 160 schools, giving approx. 3,975 students
Nationally representative
Cognitive domains
Reading (English), mathematics, health knowledge
Contextual
instruments
Student questionnaire
Test Administration
School-based
Teacher questionnaire
School head questionnaire
Group administration
Paper-based administration
Analysis
IRT analysis used to scale cognitive data
Competency levels/benchmarks established
Frequency analyses conducted/mean scores calculated for cognitive
results, disaggregated by contextual variables of interest
Frequency analyses conducted on contextual data
Relationship between cognitive performance and contextual factors
explored via analytical techniques (eg correlation, regression, multilevel
modelling)
Trends in cognitive performance computed
Reporting and
dissemination
Working papers present international-level results
Reports present national-level results
Working papers, reports, databases available from SACMEQ website
Documents: (Hungi, 2011a, 2011m; Hungi et al., 2010; Makuwa, 2011; Monyaku & Mmereki, 2011)
Websites: (SACMEQ, 2013)
81 Teachers of Grade 6 reading, mathematics and health knowledge are also tested in SACMEQ.
119
Botswana
Name
TIMSS*
Organisation
/ institution
responsible
IEA
Purpose
System-level monitoring
Inception
1995
Frequency
4-year cycle; Botswana participated in 2007 & 2011
Target Population
Grade 8 (2007); Grade 6 and Grade 9 (2011)82
Sample
TIMSS 2011: 149 schools giving approx. 4,200 students (Grade 6);
150 schools giving approx. 5,400 students (Grade 9)
Nationally representative
Cognitive domains
Mathematics and science
Contextual
instruments
Student questionnaire
Teacher questionnaire (mathematics teacher questionnaire, science
teacher questionnaire)
School head questionnaire
Curriculum questionnaire
Test Administration
School-based
Group administration
Paper-based administration
Analysis
IRT analysis used to scale cognitive and contextual data
Competency levels/benchmarks established
Frequency analyses conducted/mean scores calculated for cognitive
results, disaggregated by contextual variables of interest
Relationship between cognitive performance and contextual factors
explored via analytical techniques (eg correlation, regression, multilevel
modelling)
Trends in cognitive performance computed
International comparisons of cognitive data
Reporting and
dissemination
International results reports, encyclopaedia and databases available for
download from the TIMSS and PIRLS/IEA websites
Documents: (M. O. Martin, Mullis, Foy, & Arora, 2012; M. O. Martin, Mullis, Foy, & Stanco, 2012; I. V. S.
Mullis, M. O. Martin, P. Foy, & J. F. (with Olson, Preuschoff, C., Erberber, E., & Galia, J.), 2008a; I. V. S.
Mullis, M. O. Martin, P. Foy, & J. F. (with Olson, Preuschoff, C., Erberber, E., Arora, A., & Galia, J.), 2008b)
Websites: (IEA, n.d.; TIMSS & PIRLS International Study Center, n.d.)
82 If it was expected that a country’s Grade 4/Grade 8 students would find TIMSS assessments too difficult, IEA encouraged the
country to test higher-grade children. Thus Botswana tested Grade 6 children with the TIMSS Grade 4 assessment and Grade 9
children with the TIMSS Grade 8 assessment.
120
Botswana
Name
PIRLS, prePIRLS*
Organisation
/ institution
responsible
IEA
Purpose
System-level monitoring
Inception
2001
Frequency
5-year cycle; Botswana participated in PIRLS and prePIRLS in 2011
Target Population
Grade 6 in PIRLS, and Grade 4 in prePIRLS
Sample
prePIRLS 2011: 149 schools, giving approx. 4,400 students
PIRLS 2011: 149 schools, giving approx. 4,200 students
Cognitive domains
Reading (English) (prePIRLS 2011)
Reading (English) (PIRLS 2011)
Contextual
instruments
Student questionnaire
Teacher questionnaire
School head questionnaire
Parent questionnaire
Test Administration
School-based
Group administration
Paper-based administration
Analysis
IRT analysis used to scale cognitive and contextual data
Competency levels/benchmarks established
Frequency analyses conducted/mean scores calculated for cognitive
results, disaggregated by contextual variables of interest
Relationship between cognitive performance and contextual factors
explored via analytical techniques (eg correlation, regression, multilevel
modelling)
International comparisons of cognitive data reported
Reporting and
dissemination
International results reports, encyclopaedia and databases available
from the TIMSS and PIRLS/IEA websites
Documents: (Joncas, 2011), (M.O. Martin & Mullis, 2012)
Websites: http://timssandpirls.bc.edu/methods/index.html
121
Burundi
Name
EGRA
Organisation
/ institution
responsible
World Bank
Purpose
System level diagnostic
Inception
2011
Frequency
N/A (one-off)
Target Population
Unknown
Sample
120 schools, giving approx. 1,800 pupils
Representativeness unknown
Cognitive domains
Kirundi
Contextual
instruments
Unknown
Test Administration
School-based
One-on-one administration
Oral administration
Analysis
Unknown
Reporting and
dissemination
Unknown
Documents: (RTI, 2014a)
122
Burundi
Name
PASEC
Organisation
/ institution
responsible
CONFEMEN
Purpose
System-level monitoring
Inception
1993
Frequency
Irregular (5 cycles, since inception) – Burundi participated in 20082009
Target Population
Grade 2, Grade 5
Sample
2008–2009 post-test: 180 schools, giving approx. 2,400 students
(Grade 2); 175 schools giving approx. 2,350 students (Grade 5)
Nationally representative
Cognitive domains
Reading (French and Kirundi) and mathematics
Contextual
instruments
Student questionnaire
Teacher questionnaire
School head questionnaire
Test Administration
School-based
Group administration
Paper-based administration
Analysis
IRT analysis not used to scale cognitive data
Competency levels/benchmarks established
Frequency analyses conducted/mean scores calculated for cognitive
results, disaggregated by contextual variables of interest
Relationship between cognitive performance and contextual factors
explored via analytical techniques (eg correlation, regression, multilevel
modelling)
International comparisons of cognitive data reported
Report prepared by PASEC-CONFEMEN
Reporting and
dissemination
Reports available from PASEC website
Workshop on results held for government representatives
Documents: (Ministère de l’Enseignement de Base et Secondaire et al., 2010)
Websites: (CONFEMEN, n.d.)
123
Comoros
Name
PASEC
Organisation
/ institution
responsible
CONFEMEN
Purpose
System-level monitoring
Inception
1993
Frequency
Irregular (5 cycles, since inception) – Comoros participated in 2009
Target Population
Grade 2, Grade 5
Sample
2009 post-test: 144 schools, giving approx. 1,900 students (Grade 2);
144 schools, giving approx. 195 students (Grade 8)
Nationally representative
Cognitive domains
Reading (French) and mathematics
Contextual
instruments
Student questionnaire
Teacher questionnaire
School head questionnaire
Test Administration
School-based
Group administration
Paper-based administration
Analysis
IRT analysis not used to scale cognitive data
Competency levels/benchmarks established
Frequency analyses conducted/mean scores calculated for cognitive
results, disaggregated by contextual variables of interest
Relationship between cognitive performance and contextual factors
explored via analytical techniques (eg correlation, regression, multilevel
modelling)
International comparisons of cognitive data reported
Report prepared by PASEC-CONFEMEN
Reporting and
dissemination
Reports available from PASEC website
Workshop on results held for government representatives
Documents: (Ministère de l’Éducation Nationale et de la Recherche & PASEC-CONFEMEN, 2010)
Websites: (CONFEMEN, n.d.)
124
Eritrea
Name
Monitoring Learning Achievement (MLA)
Organisation
/ institution
responsible
Ministry of Education, UNICEF
Purpose
System-level monitoring
Inception
2001
Frequency
Irregular – second MLA in 2008
Target Population
Grade 3, Grade 5
Sample
In 2008: 60 schools, giving approx. 2,300 students (Grade 3) and
2,000 students (Grade 5)
Nationally representative
Cognitive domains
English, mother tongue, mathematics
Contextual
instruments
Student questionnaire
Teacher questionnaire
School head questionnaire
Parent questionnaire
Test Administration
School-based
Group administration
Paper-based administration
Analysis
IRT analysis not used to scale cognitive data
Competency levels/benchmarks established
Frequency analyses conducted/mean scores calculated for cognitive
results, disaggregated by contextual variables of interest
Frequency analyses conducted on contextual data
Reporting and
dissemination
Reports not publicly available
Workshops at national and sub-national levels
Documents: (UNICEF Eritrea, n.d.)
125
Ethiopia
Name
EGRA
Organisation
/ institution
responsible
USAID, RTI, Ethiopia MoE
Purpose
System-level diagnostic
Inception
2010
Frequency
N/A (one-off)
Target Population
Grade 2, Grade 3
Sample
338 schools, giving approx. 13,000 students
Cognitive domains
Reading (Tigrinya, Afan Oromo, Amharic, Somali, Sidaamu Afoo, and
Hararigna)
Contextual
instruments
Student questionnaire
Teacher questionnaire
Head teacher questionnaire
Test Administration
School-based
One-on-one administration
Oral administration
Analysis
IRT analysis not used to scale cognitive data
Competency levels/benchmarks established
Frequency analyses conducted/mean scores calculated for cognitive
results, disaggregated by contextual variables of interest
Frequency analyses conducted on contextual data
Relationship between cognitive performance and contextual factors
explored via analytical techniques (eg correlation, regression, multilevel
modelling)
Reporting and
dissemination
Documents: (RTI, 2010)
126
Report available from EdData and Ethiopia MoE websites
Policy workshop for MoE representatives and other stakeholders
Ethiopia
Name
EGRA
Organisation
/ institution
responsible
Save the Children
Purpose
Program evaluation (Literacy Boost initiative)
Inception
2010–2012
Frequency
N/A (one-off)
Target Population
Grade 3 in treatment schools and control schools in Dendi district of
the Oromia region
Sample
Approx. 400 students
Cognitive domains
Reading (Afan Oromo)
Contextual
instruments
Student questionnaire
Test Administration
School-based
One-on-one administration
Oral administration
Analysis
IRT analysis not used to scale cognitive data
Frequency analyses conducted/mean scores calculated for cognitive
results, disaggregated by contextual variables of interest
Frequency analyses conducted on contextual data
Trends in cognitive performance computed
Reporting and
dissemination
Reports available from the EdData and Save the Children websites
Documents: (Cao et al., 2011; Friedlander et al., 2012; Hassen & Friedlander, 2012)
127
Ethiopia
Name
EGRA
Organisation
/ institution
responsible
USAID-Ethiopia, USAID-Washington, American Institutes for Research
(AIR)
Purpose
System-level diagnostic
Inception
2011
Frequency
N/A (one-off)
Target Population
Grade 2–Grade 4
Sample
330 schools, giving approx. 19,600 students
Cognitive domains
English
Contextual
instruments
Student questionnaire
Test Administration
School-based
Teacher questionnaire
School head questionnaire
One-on-one administration
Oral administration
Analysis
IRT analysis not used to scale cognitive data
Competency levels/benchmarks established
Frequency analyses conducted/mean scores calculated for cognitive
results, disaggregated by contextual variables of interest
Frequency analyses conducted on contextual data
Relationship between cognitive performance and contextual factors
explored via analytical techniques (eg correlation, regression, multilevel
modelling)
Reporting and
dissemination
Reports available from AIR website
Documents: (American Institutes for Research (AIR), 2012)
128
Ethiopia
Name
National Learning Assessment
Organisation
/ institution
responsible
USAID, National Educational Assessment And Examination Agency
(NEAEA)83
Purpose
System-level monitoring
Inception
2000
Frequency
3–4-year cycle, NLA III in 2007 & NLA IV in 2010/11
Target Population
Grade 4, Grade 8
Sample
NLA IV: 299 schools giving approx. 10,800 students (Grade 4); 291
schools, giving approx. 11,200 students (Grade 8)
Nationally representative
Cognitive domains
Mathematics, English, mother tongue, environmental science (Grade 4)
Mathematics, English, biology, chemistry, physics (Grade 8)
Contextual
instruments
Student questionnaire
Test Administration
School-based
Group administration
Paper-based administration
Analysis
IRT analysis used to scale cognitive data
Competency levels/benchmarks established
Frequency analyses conducted/mean scores calculated for cognitive
results, disaggregated by contextual variables of interest
Relationship between cognitive performance and contextual factors
explored via analytical techniques (eg correlation, regression, multilevel
modelling)
Trends in cognitive performance computed
Reporting and
dissemination
Reports available from NEAEA website
Documents: (Ministry of Education of Ethiopia (FDRE), 2008, 2013)
Websites: (National Educational Assessments and Examinations Association of Ethiopia, n.d.)
83 The next round of the NLA in Ethiopia will be supported by UNICEF, not USAID.
129
Kenya
Name
EGRA
Organisation
/ institution
responsible
USAID-Kenya, USAID-Washington, RTI, Aga Khan Foundation
Purpose
Program evaluation (EMACK initiative)
Inception
2007-2008
Frequency
N/A (one-off)
Target Population
Grade 2 in treatment and control schools in Malindi district
Sample
Approx. 400 Students
Cognitive domains
Reading (English, Kiswahili)
Contextual
instruments
Student questionnaire
Test Administration
School-based
One-on-one administration
Oral administration
Analysis
IRT analysis not used to scale cognitive data
Frequency analyses conducted/mean scores calculated for cognitive
results, disaggregated by contextual variables of interest
Relationship between cognitive performance and contextual factors
explored via analytical techniques (eg correlation, regression, multilevel
modelling)
Trends in cognitive performance computed (between baseline and endline)
Reporting and
dissemination
Report available from EdData website
Documents: (Crouch, Korda, & Mumo, 2009)
130
Kenya
84
Name
EGRA, EGMA
Organisation
/ institution
responsible
USAID-Kenya, USAID-Washington, DFID, RTI
Purpose
Program evaluation (PRIMR initiative)
Inception
2012–2013
Frequency
N/A (one-off)
Target Population
Grade 1, Grade 2 in treatment and control schools
Sample
Approx. 220 schools, giving approx. 4,400 students
Cognitive domains
Reading (English, Kiswahili), mathematics
Contextual
instruments
Student questionnaire
Teacher questionnaire
Head teacher questionnaire
School inventory
Classroom inventory
Classroom observation
Test Administration
School-based
One-on-one administration
Oral administration
Analysis
IRT analysis not used to scale cognitive data
Frequency analyses conducted/mean scores calculated for cognitive
results, disaggregated by contextual variables of interest
Relationship between cognitive performance and contextual factors
explored via analytical techniques (eg correlation, regression, multilevel
modelling)
Trends in cognitive performance computed (between baseline and endline)
Reporting and
dissemination
Reports available from EdData website
Documents: (Piper & Mugenda, 2013; RTI, 2012, 2014d)
84 A number of other implementations of EGRA/EGMA have been conducted in Kenya.
131
Kenya
Name
National Assessment System for Monitoring Learning Outcomes
(NASMLA)
Organisation
/ institution
responsible
The National Assessment Centre at the Kenya National Examinations
Council
Purpose
System-level diagnostic/monitoring
Inception
2010
Frequency
Uncertain
Target Population
Grade 3
Sample
328 schools, giving approx. 8,000 students
Nationally representative
Cognitive domains
Literacy (English) and Numeracy
Contextual
instruments
Student questionnaire
Teacher questionnaire
Head teacher questionnaire
School and classroom observation schedule
Test Administration
School-based
Group administration
Paper-based administration
Analysis
IRT analysis used to scale cognitive data
Competency levels/benchmarks established
Frequency analyses conducted/mean scores calculated for cognitive
results, disaggregated by contextual variables of interest
Relationship between cognitive performance and contextual factors
explored via analytical techniques (eg correlation, regression, multilevel
modelling)
Reporting and
dissemination
Reports (including recommendations) available from the Kenya National
Examinations Council website
Documents: (The National Assessment Centre, 2010a, 2010c)
Website: (The Kenya National Examinations Council, n.d.)
132
Kenya
Name
SACMEQ
Organisation
/ institution
responsible
SACMEQ
Purpose
System-level monitoring
Inception
SACMEQ I: 1995–1999
Frequency
5–6-year cycle; Kenya participated in SACMEQ I–III
Target Population
Grade 685
Sample
SACMEQ III: 193 schools, giving approx. 4,500 students
Nationally representative
Cognitive domains
Student tests in reading (English) , mathematics, and health knowledge
Contextual
instruments
Student questionnaire
Test Administration
School-based
Teacher questionnaire
School head questionnaire
Group administration
Paper-based administration
Analysis
IRT analysis used to scale cognitive data
Competency levels/benchmarks established
Frequency analyses conducted/mean scores calculated for cognitive
results, disaggregated by contextual variables of interest
Relationship between cognitive performance and contextual factors
explored via analytical techniques (eg correlation, regression, multilevel
modelling)
Trends in cognitive performance computed
Reporting and
dissemination
Working papers present international-level results
Reports present national-level results
Working papers, reports, databases available from SACMEQ website
Documents: (Hungi, 2011a, 2011m; Hungi et al., 2010; Makuwa, 2011; Wasanga, Ogle, & Wambua, 2012)
Website: (SACMEQ, 2013)
85 Teachers of Grade 6 reading, mathematics and health knowledge are also tested in SACMEQ.
133
Kenya
Name
Uwezo*
Organisation
/ institution
responsible
Twaweza
Purpose
System-level monitoring
Inception
2009/2010
Frequency
Annual
Target Population
6–16 years old
Sample
Uwezo 2012: Approx. 145,000 children
Cognitive domains
Reading (English and Kiswahili), numeracy
Contextual
instruments
Household observation
Test Administration
Household-based
Village observation
School observation
One-on-one administration
Oral administration
Analysis
IRT analysis not used to scale cognitive data
Frequency analyses conducted/mean scores calculated for cognitive
results, disaggregated by contextual variables of interest
Frequency analyses conducted on contextual data
Trends in cognitive performance computed
Results presented in national and regional reports
Reporting and
dissemination
Reports, datasets and other documentation available from the Uwezo
website
Results disseminated via radio and print media
Documents: (Uwezo-Kenya, 2013; Uwezo, 2014)
Website: (Twaweza, n.d.)
134
Lesotho
Name
Lesotho National Assessment of Educational Progress (LNAEP)
Organisation
/ institution
responsible
Examinations Council of Lesotho (ECoL), Ministry of Education and
Training
Purpose
System-level monitoring
Inception
2003
Frequency
1–2-year cycle
Target Population
Grade 3, Grade 6
Sample
Cycle 4: 184 schools, giving approx. 3,680 students at each grade
Nationally representative
Cognitive domains
Sesotho, English, mathematics
Contextual
instruments
Student questionnaire
Test Administration
School-based
Teacher questionnaire
School head questionnaire
Group administration
Paper-based administration
Analysis
IRT analysis not used to scale cognitive data
Frequency analyses conducted/mean scores calculated for cognitive
results, disaggregated by contextual variables of interest
Frequency analyses conducted on contextual data
Reporting and
dissemination
Report (including policy implications) available on the ECoL website
Documents: (“Lesotho National Assessment of Educational Progress (LNAEP) Survey Report, 2010,” n.d.)
Websites: (Examinations Council of Lesotho (ECoL), n.d.)
135
Lesotho
Name
Assessment of Grades 1, 2 and 3 in Lesotho
Organisation
/ institution
responsible
The Australian Council for Educational Research
Purpose
Pilot for system-level monitoring
Inception
2014
Frequency
N/A (one-off)
Target Population
Grades 1–3
Sample
16 schools, giving approx. 950 students
Cognitive domains
Sesotho, mathematics
Contextual
instruments
N/A
Test Administration
School-based
One-on-one or small group administration
Oral or tablet-based administration
Analysis
IRT analysis used to scale cognitive data
Frequency analyses conducted/mean scores calculated for cognitive
results
Reporting and
dissemination
Reports not publicly available
Documents: (Hungi, 2011a, 2011m; Hungi et al., 2010; Makuwa, 2011)
Website: (SACMEQ, 2013)
136
Lesotho
Name
SACMEQ
Organisation
/ institution
responsible
SACMEQ
Purpose
System-level monitoring
Inception
SACMEQ I: 1995–1999
Frequency
5–6-year cycle; Lesotho participated in SACMEQ II & III
Target Population
Grade 686
Sample
SACMEQ III: 182 schools, giving approx. 4,250 students
Nationally representative
Cognitive domains
Reading (English), mathematics, health knowledge
Contextual
instruments
Student questionnaire
Test Administration
School-based
Teacher questionnaire
School head questionnaire
Group administration
Paper-based administration
Analysis
IRT analysis used to scale cognitive data
Competency levels/benchmarks established
Frequency analyses conducted/mean scores calculated for cognitive
results, disaggregated by contextual variables of interest
Relationship between cognitive performance and contextual factors
explored via analytical techniques (eg correlation, regression, multilevel
modelling)
Trends in cognitive performance computed
Reporting and
dissemination
Working papers present international-level results
Reports present national-level results
Working papers, reports, databases available from SACMEQ website
Documents: (Hungi, 2011a, 2011m; Hungi et al., 2010; Makuwa, 2011)
Website: (SACMEQ, 2013)
86 Teachers of Grade 6 reading, mathematics and health knowledge are also tested in SACMEQ.
137
Madagascar
Name
EGRA
Organisation
/ institution
responsible
World Bank
Purpose
System-level diagnostic
Inception
2009
Frequency
N/A (one-off)
Target Population
Unknown
Sample
Unknown
Cognitive domains
Reading (Malagasy)
Contextual
instruments
Unknown
Test Administration
Unknown
Analysis
Unknown
Reporting and
dissemination
Unknown
Documents: (RTI, 2014a)
138
Malawi
Name
Assessing Learner Achievement
Organisation
/ institution
responsible
Malawi Institute of Education
Purpose
System-level monitoring
Inception
2005
Frequency
3–4-year cycle
Target Population
Grade 2, Grade 3, Grade 5, Grade 7
Sample
Unknown
Cognitive domains
Chichewa, English, mathematics, life skills
Contextual
instruments
Unknown
Test Administration
Unknown
Analysis
Unknown
Reporting and
dissemination
Unknown
Documents: (UNESCO, 2008, 2015)
139
Malawi
Name
EGMA
Organisation
/ institution
responsible
USAID-Malawi, RTI
Purpose
Program evaluation (baseline for Malawi Teacher Professional
Development Support initiative)
Inception
2010
Frequency
N/A (one-off)
Target Population
Grade 2, Grade 4
Sample
50 schools, giving approx. 1,000 students
Cognitive domains
Mathematics
Contextual
instruments
N/A
Test Administration
School-based
One-on-one administration
Oral administration
Analysis
IRT analysis not used to scale cognitive data
Frequency analyses conducted/mean scores calculated for cognitive
results, disaggregated by contextual variables of interest
Reporting and
dissemination
Reports available from EdData website
Documents: (USAID, 2011)
140
Malawi
Name
EGRA
Organisation
/ institution
responsible
Save the Children
Purpose
Program evaluation (Literacy Boost initiative)
Inception
2009–2010
Frequency
N/A (one-off)
Target Population
Grade 2 and Grade 4 at 24 treatment and control schools in Zomba
Sample
Approx. 850 students
Cognitive domains
Reading (Chichewa)
Contextual
instruments
Student questionnaire
Test Administration
School-based
One-on-one administration
Oral administration
Analysis
IRT analysis not used to scale cognitive data
Frequency analyses conducted/mean scores calculated for cognitive
results, disaggregated by contextual variables of interest
Frequency analyses conducted on contextual data
Trends in cognitive performance computed
Reporting and
dissemination
Reports available from EdData and Save the Children websites
Documents: (Dowd & Mabeti, 2011; Dowd, Wiener, & Mabeti, 2010)
141
Malawi
87
Name
EGRA
Organisation
/ institution
responsible
USAID-Malawi, RTI
Purpose
System-level monitoring
Inception
2010–2012 (baseline, midline, end-line)
Frequency
N/A (one-off)
Target Population
Grade 2, Grade 4
Sample
2012 end-line: 202 schools, giving approx. 5,200 students
Cognitive domains
Reading (Chichewa)
Contextual
instruments
Student questionnaire
Test Administration
School-based
Teacher questionnaire
Head teacher questionnaire
One-on-one administration
Oral administration
Analysis
IRT analysis not used to scale cognitive data
Frequency analyses conducted/mean scores calculated for cognitive
results, disaggregated by contextual variables of interest
Relationship between cognitive performance and contextual factors
explored via analytical techniques (eg correlation, regression, multilevel
modelling)
Trends in cognitive performance computed
Reporting and
dissemination
Reports available from EdData website
Documents: (Miksic & Harvey, 2012; Pouezevara, Costello, & Banda, 2013; RTI, 2011)
87 Several other implementations of EGRA/EGMA have been conducted in Malawi. The table only includes a couple of indicative
implementations for which complete or near complete documentation is available on the Eddata website (the main repository for
EGRA/EGMA information).
142
Malawi
Name
Monitoring Learning Achievement (MLA)
Organisation
/ institution
responsible
Ministry of Education, Science and Technology, UNICEF
Purpose
System-level monitoring
Inception
2012
Frequency
3-year cycle (intended)
Target Population
Grade 2, Grade 4, Grade 7
Sample
2012: 225 schools, giving approx. 3,400 students (Grade 2), 2,750
students (Grade 4), and 3,200 students (Grade 7)
Nationally representative
Cognitive domains
Chichewa, English, mathematics
Contextual
instruments
Student questionnaire
Teacher questionnaire
Head teacher questionnaire
Test Administration
School-based
Group administration
Paper-based administration
Analysis
IRT analysis not used to scale cognitive data
Competency levels/benchmarks established
Frequency analyses conducted/mean scores calculated for cognitive
results, disaggregated by contextual variables of interest
Relationship between cognitive performance and contextual factors
explored via analytical techniques (eg correlation, regression, multilevel
modelling)
Reporting and
dissemination
Reports not publicly available
Documents: (Ministry of Education, Science, & Technology of Malawi, 2014)
143
Malawi
Name
SACMEQ
Organisation
/ institution
responsible
SACMEQ
Purpose
System-level monitoring (national)
Inception
SACMEQ I: 1995–1999
Frequency
5–6-year cycle; Malawi participated in SACMEQ I–III
Target Population
Grade 688
Sample
SACMEQ III: 139 schools, giving approx. 2,800 students
Cognitive domains
Reading (English), mathematics, health knowledge
Contextual
instruments
Student questionnaire
Test Administration
School-based
Teacher questionnaire
School head questionnaire
Group administration
Paper-based administration
Analysis
IRT analysis used to scale cognitive data
Competency levels/benchmarks established
Frequency analyses conducted/mean scores calculated for cognitive
results, disaggregated by contextual variables of interest
Relationship between cognitive performance and contextual factors
explored via analytical techniques (eg correlation, regression, multilevel
modelling)
Trends in cognitive performance computed
Reporting and
dissemination
Working papers present international-level results
Reports present national-level results
Working papers, reports, databases available from SACMEQ website89
Documents: (Hungi, 2011a, 2011m; Hungi et al., 2010; Makuwa, 2011; Ministry of Education, Science, &
Technology of Malawi, 2011)
Website: (SACMEQ, 2013)
88 Teachers of Grade 6 reading, mathematics and health knowledge are also tested in SACMEQ.
89 Note that the SACMEQ website was offline for the duration of this consultancy. The information in this cell is based on material
downloaded from the site before it went offline.
144
Mozambique
Name
EGRA
Organisation
/ institution
responsible
Save the Children
Purpose
Program evaluation (Literacy Boost initiative)
Inception
2010-2011
Frequency
N/A (one-off)
Target Population
Grades 1–3 in treatment and control schools in Gaza
Sample
2011 end-line: approx. 550 children (preschool); approx. 430 students
(Grades 1–3)
Cognitive domains
Reading (Portuguese and Shangana)
Contextual
instruments
Student questionnaire
Test Administration
School-based
One-on-one administration
Oral administration
Analysis
IRT analysis not used to scale cognitive data
Frequency analyses conducted/mean scores calculated for cognitive
results, disaggregated by contextual variables of interest
Frequency analyses conducted on contextual data
Trends in cognitive performance computed
Reporting and
dissemination
Reports available from EdData website and Save the Children website
Documents: (Dowd & Fonseca, 2012)
145
Mozambique
Name
EGRA
Organisation
/ institution
responsible
USAID, International Business & Technical Consultants, Inc. (IBTCI),
Global Surveys Corporation (GSC Research)
Purpose
Program evaluation (USAID/Aprender a Ler (APAL) initiative)
Inception
2013
Frequency
N/A (one-off)
Target Population
Grade 2, Grade 3 in treatment and control schools in Nampula and
Zambézia provinces
Sample
2013 baseline: Approx. 3,500 students (baseline in 2013)
Cognitive domains
Reading (Portuguese)
Contextual
instruments
Student questionnaire
Student questionnaire
Teacher questionnaire
Head teacher questionnaire
School inventory
Classroom inventory
Classroom observation
Test Administration
School-based
One-on-one administration
Oral administration
Analysis
IRT analysis not used to scale cognitive data
Frequency analyses conducted/mean scores calculated for cognitive
results, disaggregated by contextual variables of interest
Frequency analyses conducted on contextual data
Reporting and
dissemination
Reports available from EdData website
Documents: (Raupp, Newman, & Revés, 2013)
146
Mozambique
Name
National Assessment
Organisation
/ institution
responsible
Instituto Nacional de Desenvolvimento de Educação
Purpose
System-level monitoring
Inception
2000
Frequency
Irregular – 2nd and 3rd implementations in 2006 and 2009
Target Population
Grades 3
Sample
Unknown
Cognitive domains
Mother tongue, Portuguese, mathematics
Contextual
instruments
Unknown
Test Administration
Unknown
Analysis
Unknown
Reporting and
dissemination
Unknown
Documents: (UNESCO, 2008, 2015; UNICEF Mozambique Country Office, 2015)
147
Mozambique
Name
SACMEQ
Organisation
/ institution
responsible
SACMEQ
Purpose
System-level monitoring
Inception
SACMEQ I: 1995–1999
Frequency
5–6-year cycle; Mozambique participated in SACMEQ II & III
Target Population
Grade 690
Sample
SACMEQ III: 183 schools, giving approx. 3,400 students
Nationally representative
Cognitive domains
Reading (English), mathematics, health knowledge
Contextual
instruments
Student questionnaire
Test Administration
School-based
Teacher questionnaire
School head questionnaire
Group administration
Paper-based administration
Analysis
IRT analysis used to scale cognitive data
Competency levels/benchmarks established
Frequency analyses conducted/mean scores calculated for cognitive
results, disaggregated by contextual variables of interest
Relationship between cognitive performance and contextual factors
explored via analytical techniques (eg correlation, regression, multilevel
modelling)
Trends in cognitive performance computed
Reporting and
dissemination
Working papers present international-level results
Reports present national-level results
Working papers, reports, databases available from SACMEQ website
Documents: (Hungi, 2011a, 2011m; Hungi et al., 2010; Makuwa, 2011)
Website: (SACMEQ, 2013)
90 Teachers of Grade 6 reading, mathematics and health knowledge are also tested in SACMEQ.
148
Namibia
Name
National Standardized Achievement Test (NSAT)
Organisation
/ institution
responsible
Directorate of National Examinations and Assessment (DNEA)
Purpose
System-level monitoring
Inception
2009
Frequency
Biannual
Target Population
Grade 5, Grade 7
Sample
Nationally representative
Cognitive domains
English, mathematics (Grade 5)
English, mathematics, natural science (Grade 7)
Contextual
instruments
Unknown
Test Administration
Unknown
Analysis
Unknown
Reporting and
dissemination
Unknown
Documents: (UNESCO, 2015)
Websites: (American Institutes for Research (AIR), n.d.; Nhongo, 2014; Sasman, 2011)
149
Namibia
Name
SACMEQ
Organisation
/ institution
responsible
SACMEQ
Purpose
System-level monitoring
Inception
SACMEQ I: 1995–1999
Frequency
5–6-year cycle; Namibia participated in SACMEQ I–III
Target Population
Grade 691
Sample
SACMEQ III: 275 schools, giving approx. 5,000 students
Nationally representative
Cognitive domains
Reading (English), mathematics, health knowledge
Contextual
instruments
Student questionnaire
Test Administration
School-based
Teacher questionnaire
School head questionnaire
Group administration
Paper-based administration
Analysis
IRT analysis used to scale cognitive data
Competency levels/benchmarks established
Frequency analyses conducted/mean scores calculated for cognitive
results, disaggregated by contextual variables of interest
Relationship between cognitive performance and contextual factors
explored via analytical techniques (eg correlation, regression, multilevel
modelling)
Trends in cognitive performance computed
Reporting and
dissemination
Working papers present international-level results
Reports present national-level results
Working papers, reports, databases available from the SACMEQ
website
Documents: (Hungi, 2011a, 2011m; Hungi et al., 2010; Makuwa, 2005, 2011)
Website: (SACMEQ, 2013)
91 Teachers of Grade 6 reading, mathematics and health knowledge are also tested in SACMEQ.
150
Rwanda
Name
EGRA, EGMA
Organisation
/ institution
responsible
USAID-Washington, USAID-Rwanda, Rwandan Ministry of Education
Purpose
System-level diagnostic
Inception
2011
Frequency
N/A (one-off)
Target Population
Grade 4, Grade 6
Sample
42 schools, giving approx. 840 students
Cognitive domains
Reading (English and Kinyarwanda), mathematics
Contextual
instruments
Student questionnaire
Teacher questionnaire
Head teacher questionnaire
School inventory
Classroom inventory
Classroom observation
Test Administration
School-based
One-on-one administration
Oral administration
Analysis
IRT analysis not used to scale cognitive data
Frequency analyses conducted/mean scores calculated for cognitive
results, disaggregated by contextual variables of interest
Frequency analyses conducted on contextual data
Relationship between cognitive performance and contextual factors
explored via analytical techniques (eg correlation, regression, multilevel
modelling)
Reporting and
dissemination
Reports available from EdData website
Documents: (DeStefano, Ralaingita, Costello, Sax, & Frank, 2012)
151
Rwanda
Name
Learning Achievement in Rwandan Schools (LARS)
Organisation
/ institution
responsible
Rwanda Education Board (REB),
Purpose
System-level monitoring
Inception
2011
Frequency
3-year cycle
Target Population
Grade 3
Sample
60 schools, giving approx. 2,500 students
Cognitive domains
Literacy (Kin-Rwanda) and numeracy
Contextual
instruments
Student questionnaire
Education Quality and Standards Department, UNICEF, UNESCO
Teacher questionnaire
School head questionnaire
Parent questionnaire
Test Administration
School-based
Group administration
Paper-based administration
Analysis
IRT analysis used to scale cognitive data
Competency levels/benchmarks established
Frequency analyses conducted/mean scores calculated for cognitive
results, disaggregated by contextual variables of interest
Frequency analyses conducted on contextual data
Relationship between cognitive performance and contextual factors
explored via analytical techniques (eg correlation, regression, multilevel
modelling)
Reporting and
dissemination
Reports not publicly available
Documents: (Rwanda Education Board, 2012), (UNICEF Rwanda Country Office, 2015)
152
Somalia
Name
EGRA
Organisation
/ institution
responsible
Concern Worldwide
Purpose
Program evaluation
Inception
2013–2014
Frequency
N/A (one-off)
Target Population
Grade 2, Grade 3, Grade 4 in Concern-supported schools in Mogadishu
Sample
2014 end-line: Five schools, giving approx. 321 students
Cognitive domains
Reading (Somali language)
Contextual
instruments
N/A
Test Administration
School-based
One-on-one administration
Oral administration
Analysis
IRT analysis not used to scale cognitive data
Frequency analyses conducted/mean scores calculated for cognitive
results, disaggregated by contextual variables of interest
Reporting and
dissemination
Reports not publicly available
Documents: (Beattie, 2014; Beattie & Grogan, 2013)
153
Somalia
Name
Monitoring Learning Achievement (MLA)
Organisation
/ institution
responsible
Ministry of Education, UNICEF
Purpose
System-level monitoring
Inception
N/A
Frequency
3-year cycle (intended)
Target Population
Grade 4, Grade 7 in Puntland and Somaliland
Sample
15 schools, giving approx. 1,000 students (Grade 4)
Cognitive domains
Somali and Mathematics (Grade 4)
Somali, Mathematics and Science (Grade 7)
Contextual
instruments
N/A
Test Administration
School-based
Group administration
Paper-based administration
Analysis
IRT analysis not used to scale cognitive data
Competency levels/benchmarks established
Frequency analyses conducted/mean scores calculated for cognitive
results, disaggregated by contextual variables of interest
Reporting and
dissemination
Reports not publicly available
Documents: (UNICEF Somalia, n.d.)
154
South Africa
Name
Annual National Assessment
Organisation
/ institution
responsible
Department of Basic Education (DBE), Ministry of Education of South
Africa
Purpose
System-level monitoring
Inception
2011
Frequency
Annual
Target Population
Grade 1–6, Grade 992
Sample
Census (i.e. all children in target population)
Cognitive domains
Language (English, Afrikaans and nine local languages), mathematics
(Grades 1–3)
Language (English and Afrikaans) and mathematics (Grades 4–6, Grade
9)
Contextual
instruments
N/A
Test Administration
School-based
Group administration
Paper-based administration
Analysis
IRT analysis not used to scale cognitive data
Competency levels/benchmarks established
Frequency analyses conducted/mean scores calculated for cognitive
results, disaggregated by contextual variables of interest
Reporting and
dissemination
Reports available from DBE website
Documents: (Department of Basic Education Republic of South Africa, 2011, 2012, 2013, 2014)
92 In South Africa’s Annual National Assessment, testing of Grade 7 and Grade 8 was piloted in 2014.
155
South Africa
Name
National Assessment of Learner Achievement (NALA)
Organisation
/ institution
responsible
Human Sciences Research Council
Purpose
System-level monitoring (national)
Inception
2008
Frequency
N/A (one-off)
Target Population
Grade 9
Sample
Unknown
Cognitive domains
Language, mathematics, natural sciences
Contextual
instruments
Unknown
Test Administration
School-based
Group administration
Paper-based administration
Analysis
Unknown
Reporting and
dissemination
Reports not publicly available
Documents: (UNESCO, 2008, 2015)
Website: (Human Sciences Research Council South Africa, n.d.)
156
South Africa
Name
SACMEQ
Organisation
/ institution
responsible
SACMEQ
Purpose
System-level monitoring
Inception
SACMEQ I: 1995–1999
Frequency
5–6-year cycle; South Africa participated in SACMEQ II & III
Target Population
Grade 693
Sample
SACMEQ III: 392 schools, giving approx. 9,100 students
Nationally representative
Cognitive domains
Reading (English), mathematics, health knowledge
Contextual
instruments
Student questionnaire
Test Administration
School-based
Teacher questionnaire
School head questionnaire
Group administration
Paper-based administration
Analysis
IRT analysis used to scale cognitive data
Competency levels/benchmarks established
Frequency analyses conducted/mean scores calculated for cognitive
results, disaggregated by contextual variables of interest
Relationship between cognitive performance and contextual factors
explored via analytical techniques (eg correlation, regression, multilevel
modelling)
Trends in cognitive performance computed
Reporting and
dissemination
Working papers present international-level results
Reports present national-level results
Working papers, reports, databases available from SACMEQ website
Documents: (Hungi, 2011a, 2011m; Hungi et al., 2010; Makuwa, 2011; Moloi, n.d.)
Website: (SACMEQ, 2013)
93 Teachers of Grade 6 reading, mathematics and health knowledge are also tested in SACMEQ.
157
South Africa
Name
TIMSS*
Organisation
/ institution
responsible
IEA
Purpose
System-level monitoring
Inception
1995
Frequency
4-year cycle; South Africa participated in 2011
Target Population
Grade 9 (2011)94
Sample
TIMSS 2011: 285 schools, giving approx. 12,000 students
Nationally representative
Cognitive domains
Mathematics and science
Contextual
instruments
Student questionnaire
Teachers questionnaires (Mathematics, Science)
School head questionnaire
Curriculum questionnaire
Test Administration
School-based
Group administration
Paper-based administration
Analysis
IRT analysis used to scale cognitive and contextual data
Competency levels/benchmarks established
Frequency analyses conducted/mean scores calculated for cognitive
results, disaggregated by contextual variables of interest
Relationship between cognitive performance and contextual factors
explored via analytical techniques (eg correlation, regression, multilevel
modelling)
Trends in cognitive performance computed
International comparisons of cognitive data reported
Reporting and
dissemination
International results reports, encyclopaedia and databases available
from the TIMSS and PIRLS/IEA websites
Documents: (M.O. Martin & Mullis, 2012; M. O. Martin, Mullis, Foy, & Arora, 2012; M. O. Martin, Mullis,
Foy, & Stanco, 2012; I.V.S. Mullis et al., 2008a, 2008b)
Websites: (IEA, n.d.; TIMSS & PIRLS International Study Center, n.d.)
94 If it was expected that a country’s Grade 4/Grade 8 students would find TIMSS assessments too difficult, IEA encouraged the
country to test higher-grade children. Thus South Africa tested Grade 9 children with the TIMSS Grade 8 assessment.
158
South Africa
Name
PIRLS, prePIRLS*
Organisation
/ institution
responsible
IEA
Purpose
System-level monitoring
Inception
2011
Frequency
5-year cycle; South Africa participated in 2006 and 2011
Target Population
Grade 5 (2006), Grade 4, Grade 5 (2011)95
Sample
prePIRLS 2011: 341 schools giving approx. 15,750 students
PIRLS 2011: 95 schools, giving approx. 3,500 students
Nationally representative
Cognitive domains
Reading (English, Afrikaans, 11 local languages) (prePIRLS 2011)
Reading (English, Afrikaans) (PIRLS 2011)
Contextual
instruments
Student questionnaire
Teacher questionnaire
School head questionnaire
Parent questionnaire
Curriculum questionnaire
Test Administration
School-based
Group administration
Paper-based administration
Analysis
IRT analysis used to scale cognitive and contextual data
Competency levels/benchmarks established
Frequency analyses conducted/mean scores calculated for cognitive
results, disaggregated by contextual variables of interest
Relationship between cognitive performance and contextual factors
explored via analytical techniques (eg correlation, regression, multilevel
modelling)
Trends in cognitive performance computed
International comparisons of cognitive data reported
Reporting and
dissemination
International results reports, encyclopaedia and databases available
from the TIMSS and PIRLS/IEA websites
Documents: (Howie, Staden, Tshele, Dowse, & Zimmerman, 2012; M.O. Martin & Mullis, 2012; I.V.S. Mullis,
Martin, Foy, & Drucker, 2012; Ina V.S. Mullis, Martin, Kennedy, & Foy P., 2007)
Websites: (IEA, n.d.; TIMSS & PIRLS International Study Center, n.d.)
95 To overcome the challenges presented by multiple native languages and languages of instruction, South Africa tested Grade 5
students in PIRLS in 2006, instead of testing the standard of Grade 4 students. In 2011, Grade 4 students were tested with the
easier prePIRLS assessment, and Grade 5 children were again tested with PIRLS.
159
Swaziland
Name
SACMEQ
Organisation
/ institution
responsible
SACMEQ
Purpose
System-level monitoring
Inception
SACMEQ I: 1995–1999
Frequency
5–6-year cycle; Swaziland participated in SACMEQ II & III
Target Population
Grade 696
Sample
SACMEQ III: 172 schools, giving approx. 4,000 students
Nationally representative
Cognitive domains
Reading (English), mathematics, health knowledge
Contextual
instruments
Student questionnaire
Test Administration
School-based
Teacher questionnaire
School head questionnaire
Group administration
Paper-based administration
Analysis
IRT analysis used to scale cognitive data
Competency levels/benchmarks established
Frequency analyses conducted/mean scores calculated for cognitive
results, disaggregated by contextual variables of interest
Relationship between cognitive performance and contextual factors
explored via analytical techniques (eg correlation, regression, multilevel
modelling)
Trends in cognitive performance computed
Reporting and
dissemination
Working papers present international-level results
Reports present national-level results
Working papers, reports, databases available from SACMEQ website
Documents: (Hungi, 2011a, 2011m; Hungi et al., 2010; Makuwa, 2011)
Websites: (SACMEQ, 2013)
96 Teachers of Grade 6 reading, mathematics and health knowledge are also tested in SACMEQ.
160
Tanzania
Name
EGRA, EGMA
Organisation
/ institution
responsible
USAID-Tanzania, RTI
Purpose
System-level monitoring
Inception
2013 (baseline)
Frequency
N/A (one-off)
Target Population
Grade 2
Sample
200 schools, giving approx. 2,300 students
Cognitive domains
Reading (English, Kiswahili), mathematics
Contextual
instruments
Student questionnaire
Teacher questionnaire
Head teacher questionnaire
School inventory
Classroom inventory
Classroom observation
Test Administration
School-based
One-on-one administration
Oral administration
Analysis
IRT analysis not used to scale cognitive data
Frequency analyses conducted/mean scores calculated for cognitive
results, disaggregated by contextual variables of interest
Relationship between cognitive performance and contextual factors
explored via analytical techniques (eg correlation, regression, multilevel
modelling)
Reporting and
dissemination
Reports available from EdData website
Documents: (Brombacher et al., 2014)
161
Tanzania
Name
SACMEQ
Organisation
/ institution
responsible
SACMEQ
Purpose
System-level monitoring
Inception
SACMEQ I: 1995–1999
Frequency
5–6-year cycle; Tanzania participated in SACMEQ II & III97
Target Population
Grade 698
Sample
SACMEQ III: 196 schools, giving approx. 4,200 students
Nationally representative
Cognitive domains
Reading (English), mathematics, health knowledge
Contextual
instruments
Student questionnaire
Test Administration
School-based
Teacher questionnaire
School head questionnaire
Group administration
Paper-based administration
Analysis
IRT analysis used to scale cognitive data
Competency levels/benchmarks established
Frequency analyses conducted/mean scores calculated for cognitive
results, disaggregated by contextual variables of interest
Relationship between cognitive performance and contextual factors
explored via analytical techniques (eg correlation, regression, multilevel
modelling)
Trends in cognitive performance computed
Reporting and
dissemination
Working papers present international-level results
Reports present national-level results
Working papers, reports, databases available from SACMEQ website
Documents: (Hungi, 2011a, 2011m; Hungi et al., 2010; Makuwa, 2011)
Websites: (SACMEQ, 2013)
97 Only Zanzibar of Tanzania participate in SACMEQ I.
98 Teachers of Grade 6 reading, mathematics and health knowledge are also tested in SACMEQ.
162
Tanzania
Name
Uwezo*
Organisation
/ institution
responsible
Twaweza
Purpose
System-level monitoring
Inception
2009/2010
Frequency
Annual
Target Population
6–16 years old
Sample
Approx. 10,5000 children
Nationally representative
Cognitive domains
Reading (English and Kiswahili), numeracy
Contextual
instruments
Household observation
Test Administration
Household-based
Village observation
School observation
One-on-one administration
Oral administration
Analysis
IRT analysis not used to scale cognitive data
Frequency analyses conducted/mean scores calculated for cognitive
results, disaggregated by contextual variables of interest
Frequency analyses conducted on contextual data
Trends in cognitive performance computed
Results presented in national and regional reports
Reporting and
dissemination
Reports, datasets and other documentation available from the Uwezo
website
Results disseminated via radio and print media
Documents: (Uwezo-Tanzania, 2013; Uwezo, 2014)
Website: (Twaweza, n.d.)
163
Uganda
Name
EGRA
Organisation
/ institution
responsible
William and Flora Hewlett Foundation, RTI
Purpose
System-level diagnostic
Inception
2009
Frequency
N/A (one-off)
Target Population
Grade 2, Grade 3 in Central and Northern provinces
Sample
50 schools, giving approx. 1,950 students
Cognitive domains
Reading (English, Luganda/Lango)
Contextual
instruments
Student questionnaire
Teacher questionnaire
Head teacher questionnaire
School inventory
Classroom inventory
Classroom observation
Test Administration
School-based
One-on-one administration
Oral administration
Analysis
IRT analysis not used to scale cognitive data
Frequency analyses conducted/mean scores calculated for cognitive
results, disaggregated by contextual variables of interest
Frequency analyses conducted on contextual data
Relationship between cognitive performance and contextual factors
explored via analytical techniques (eg correlation, regression, multilevel
modelling)
Reporting and
dissemination
Report available from EdData website
Documents: (Piper, 2010)
164
Uganda
Name
EGRA
Organisation
/ institution
responsible
Save the Children
Purpose
Program evaluation (Literacy Boost initiative)
Inception
2010 (baseline), 2012 (midline)
Frequency
N/A (one-off)
Target Population
Grade 3 in treatment and comparison schools in Amuru and Nwoya
districts
Sample
2012 midline: approx. 530 students
Cognitive domains
Reading (Luo), one mathematics subtask
Contextual
instruments
N/A
Test Administration
School-based
One-on-one administration
Oral administration
Analysis
IRT analysis not used to scale cognitive data
Frequency analyses conducted/mean scores calculated for cognitive
results, disaggregated by contextual variables of interest
Trends in cognitive performance computed
Reporting and
dissemination
Reports available from Save the Children website
Documents: (Friedlander, Candiru, & Dowd, 2010; Guajardo et al., 2010)
165
Uganda
Name
National Assessment of Progress in Education (NAPE)
Organisation
/ institution
responsible
Uganda National Examinations Board (UNEB)
Purpose
System-level monitoring
Inception
1996
Frequency
1–3-year cycle, most recently in 2010
Target Population
Grade 3, Grade 6
Sample
2010: 1,098 schools, giving approx. 21,900 students (in each of Grade
3 and Grade 6)
Nationally representative
Cognitive domains
Literacy (English and local languages), numeracy
Contextual
instruments
Student questionnaire
Test Administration
School-based
Head teacher questionnaire
Group administration
Paper-based administration
Analysis
IRT analysis not used to scale cognitive data
Competency levels/benchmarks established
Frequency analyses conducted/mean scores calculated for cognitive
results, disaggregated by contextual variables of interest
Frequency analyses conducted on contextual data
Reporting and
dissemination
Reports available from UNEB website
Documents: (Uganda National Examinations Board, 2010)
Website: (Uganda National Examinations Board, n.d.)
166
Uganda
Name
SACMEQ
Organisation
/ institution
responsible
SACMEQ
Purpose
System-level monitoring
Inception
SACMEQ I: 1995–1999
Frequency
5–6-year cycle; Uganda participated in SACMEQ II & III
Target Population
Grade 699
Sample
SACMEQ III: 264 schools, giving approx. 5,300 students
Nationally representative
Cognitive domains
Reading (English), mathematics, health knowledge
Contextual
instruments
Student questionnaire
Test Administration
School-based
Teacher questionnaire
School head questionnaire
Group administration
Paper-based administration
Analysis
IRT analysis used to scale cognitive data
Competency levels/benchmarks established
Frequency analyses conducted/mean scores calculated for cognitive
results, disaggregated by contextual variables of interest
Relationship between cognitive performance and contextual factors
explored via analytical techniques (eg correlation, regression, multilevel
modelling)
Trends in cognitive performance computed
Reporting and
dissemination
Working papers present international-level results
Reports present national-level results
Working papers, reports, databases available from SACMEQ website
Documents: (Hungi, 2011a, 2011m; Hungi et al., 2010; Makuwa, 2011)
Websites: (SACMEQ, 2013)
99 Teachers of Grade 6 reading, mathematics and health knowledge are also tested in SACMEQ.
167
Uganda
Name
Uwezo*
Organisation
/ institution
responsible
Twaweza
Purpose
System-level monitoring
Inception
2009/2010
Frequency
Annual
Target Population
7–16 years old
Sample
Approx. 92,000 children
Nationally representative
Cognitive domains
Reading (English and local languages), numeracy
Contextual
instruments
Household observation
Test Administration
Household-based
Village observation
School observation
One-on-one administration
Oral administration
Analysis
IRT analysis not used to scale cognitive data
Frequency analyses conducted/mean scores calculated for cognitive
results, disaggregated by contextual variables of interest
Frequency analyses conducted on contextual data
Trends in cognitive performance computed
Results presented in national and regional reports
Reporting and
dissemination
Reports, datasets and other documentation available from the Uwezo
website
Results disseminated via radio and print media
Documents: (Uwezo-Uganda, 2013; Uwezo, 2014)
Website: (Twaweza, n.d.)
168
Zambia
Name
EGRA, EGMA
Organisation
/ institution
responsible
USAID, RTI
Purpose
Pilot for system-level diagnostic
Inception
2011
Frequency
N/A (one-off)
Target Population
Grade 2, Grade 3 in the Central, Copperbelt, Luapula, and Northern
regions
Sample
33 schools, giving approx. 800 students
Cognitive domains
Reading (Bemba), mathematics
Contextual
instruments
Student questionnaire
Teacher questionnaire
Head teacher questionnaire
School inventory
Classroom inventory
Classroom observation
Test Administration
School-based
One-on-one administration
Oral administration
Analysis
IRT analysis not used to scale cognitive data
Frequency analyses conducted/mean scores calculated for cognitive
results, disaggregated by contextual variables of interest
Frequency analyses conducted on contextual data
Relationship between cognitive performance and contextual factors
explored via analytical techniques (eg correlation, regression, multilevel
modelling)
Reporting and
dissemination
Report available from EdData website
Documents: (Collins et al., 2012; RTI, 2015a)
169
Zambia
Name
EGRA
Organisation
/ institution
responsible
USAID, EDC
Purpose
Program evaluation
Inception
2012
Frequency
N/A (one-off)
Target Population
Unknown
Sample
1,400 students in 6 provinces
Cognitive domains
Reading (English, ChiNyanja, ChiTonga, IciBemba)
Contextual
instruments
Unknown
Test Administration
School-based
One-on-one administration
Oral administration
Analysis
Unknown
Reporting and
dissemination
Reports not publicly available
Documents: (Collins et al., 2012; RTI, 2015a)
170
Zambia
Name
EGRA
Organisation
/ institution
responsible
USAID, RTI
Purpose
System-level diagnostic (as part of the National Assessment Survey)
Inception
2014
Frequency
2-year cycle
Target Population
Grade 2
Sample
850 schools, 8,500 students
Nationally representative
Cognitive domains
Reading (Bemba, Nyanja, Luvale, Lunda, Silozi, Kikoande, Tonga)
Contextual
instruments
Unknown
Test Administration
School-based
One-on-one administration
Oral administration
Analysis
Unknown
Reporting and
dissemination
Reports not publicly available
Documents: (Collins et al., 2012; RTI, 2015a)
171
Zambia
Name
National Assessment of Learning Achievement
Organisation
/ institution
responsible
Examinations Council of Zambia
Purpose
System-level monitoring
Inception
1999
Frequency
2-year cycle
Target Population
Grade 5, Grade 9
Sample
2008: Approx. 400 schools, approx. 8,000 students
Nationally representative
Cognitive domains
Grade 5 : English, mathematics, life skills
Grade 9 : English, mathematics, environmental sciences
Contextual
instruments
Student questionnaire
Test Administration
School-based
Teacher questionnaire
Head teacher questionnaire
Group administration
Paper-based administration
Analysis
IRT analysis not used to scale cognitive data
Competency levels/benchmarks established
Frequency analyses conducted/mean scores calculated for cognitive
results, disaggregated by contextual variables of interest
Frequency analyses conducted on contextual data
Reporting and
dissemination
Reports not publicly available
Results of Grade 5 in 2008 were disseminated at provincial level. Also,
remedial materials were developed for the areas that were found to be
challenging for teachers and learners based on the test item analysis
Documents: (Examinations Council of Zambia, 2015; RTI, 2015a, 2015c; Sakala & Chilala, 2007; UNICEF
Zambia Country Office, 2015)
172
Zambia
Name
SACMEQ
Organisation
/ institution
responsible
SACMEQ
Purpose
System-level monitoring
Inception
SACMEQ I: 1995–1999
Frequency
5–6-year cycle; Zambia participated in SACMEQ I–III
Target Population
Grade 6100
Sample
SACMEQ III: 157 schools, giving approx. 2,900 students
Nationally representative
Cognitive domains
Reading (English), mathematics, health knowledge
Contextual
instruments
Student questionnaire
Test Administration
School-based
Teacher questionnaire
School head questionnaire
Group administration
Paper-based administration
Analysis
IRT analysis used to scale cognitive data
Competency levels/benchmarks established
Frequency analyses conducted/mean scores calculated for cognitive
results, disaggregated by contextual variables of interest
Relationship between cognitive performance and contextual factors
explored via analytical techniques (eg correlation, regression, multilevel
modelling)
Trends in cognitive performance computed
Reporting and
dissemination
Working papers present international-level results
Reports present national-level results
Working papers, reports, databases available from SACMEQ website
Documents: (Hungi, 2011a, 2011m; Hungi et al., 2010; Makuwa, 2011)
Websites: (SACMEQ, 2013)
100 Teachers of Grade 6 reading, mathematics and health knowledge are also tested in SACMEQ.
173
Zimbabwe
Name
SACMEQ
Organisation
/ institution
responsible
SACMEQ
Purpose
System-level monitoring
Inception
SACMEQ I: 1995–1999
Frequency
5–6-year cycle; Zimbabwe participated in SACMEQ I and III
Target Population
Grade 6101
Sample
SACMEQ III: 155 schools, giving approx. 3,000 students
Nationally representative
Cognitive domains
Reading (English), mathematics, health knowledge
Contextual
instruments
Student questionnaire
Test Administration
School-based
Teacher questionnaire
School head questionnaire
Group administration
Paper-based administration
Analysis
IRT analysis used to scale cognitive data
Competency levels/benchmarks established
Frequency analyses conducted/mean scores calculated for cognitive
results, disaggregated by contextual variables of interest
Relationship between cognitive performance and contextual factors
explored via analytical techniques (eg correlation, regression, multilevel
modelling)
Trends in cognitive performance computed
Reporting and
dissemination
Working papers present international-level results
Reports present national-level results
Working papers, reports, databases available from SACMEQ website
Documents: (Hungi, 2011a, 2011m; Hungi et al., 2010; Makuwa, 2011)
Websites: (SACMEQ, 2013)
101 Teachers of Grade 6 Reading, Mathematics and Health Knowledge are also tested in SACMEQ.
174
Zimbabwe
Name
Zimbabwe Early Learning Assessment (ZELA)
Organisation
/ institution
responsible
ZIMSEC
Purpose
System-level monitoring
Inception
2012–2015 (baseline, two cycles)
Frequency
N/A (one-off)
Target Population
Grade 3
Sample
2014 cycle: 500 schools, giving approx. 16,000 students
Cognitive domains
English, mathematics, Ndebele and Shona
Contextual
instruments
Student questionnaire
Test Administration
School-based
School head questionnaire
Group administration
Paper-based administration
Analysis
IRT analysis used to scale cognitive data
Competency levels/benchmarks established
Frequency analyses conducted/mean scores calculated for cognitive
results, disaggregated by contextual variables of interest
Relationship between cognitive performance and contextual factors
explored via analytical techniques (eg correlation, regression, multilevel
modelling)
Trends in cognitive performance computed
Reporting and
dissemination
Reports not publicly available
Documents: (The Australian Council for Educational Research & Zimbabwe School Examination Council,
2013a, 2013c, 2015)
175
176
178