IMPACT EVALUATION
The Impact Evaluation assessed whether, after a full year
(2014) of implementation of the USAID / ApaL Project,
there was an impact on student reading skills and, if so,
what the cost of this intervention was. A randomized
control trial (RCT) was created to measure these effects
under two treatment packages and a control, or
non-treatment group.
Wordscorrectly
correctly read
minute
(cwpm)
Words
readper
per
minute
(cwpm)
Midterm 1 SEP 2013 vs Midterm 2 SEP 2014
Midterm 1 Sept 2013 vs Midterm 2 Sept 2014
9
Full
8.1
8
7
Medium
6.5
6
5
4
Full
3.8
Medium
3.3
Control
2.8
Control
3.2
10 POINTS TO WEIGH
AND CONSIDER:
3
1 In intervention schools, findings show improvement in
Percentage of students reading 45+
rd Grade)
Percentage
reading
cwpmof(3students
45+ cwpm (3rd Grade)
student performance in all competencies measured by the
Early Grade Reading Assessment (EGRA).
2 Differences between the results obtained by students in
treatment schools and in Control schools are always significant as are, in many cases, the diferences between Full and
Medium treatment groups.
3 The cost-effectiveness analysis confirms that the Full
treatment had a larger impact on student performance
relative to the Medium treatment group, even when the
difference in cost of US$ 2.75 per year per student is
considered.
4 The impact of the project is greater in the third grade.
Nonetheless, third graders are still far from the target of
reading 45 words per minute required to enable them to
understand what they read.
5 The training offered by ApaL strengthened teaching of
reading, as evidenced by the improvement in pedagogical
practices observed in classrooms of treatment schools.
Percentage of teachers who obtained
Percentage
of teachers who
the maximum score
obtained the maximum score
FULL
MEDIUM
MEDIUM FULL
CONTROL
CONTROL
75
80
70
72,1
60
C
o
m
p 50
o
s
i
t 40
e
51,3
43
75
54,9
51,3
24,5
20
23,1
10
10,9
0
Planning and
Teaching
Decoding
Sequencing Comprehension
1
0
Meio Termo 1
Meio Termo 2
■FULL ■ MEDIUM ■ CONTROL
8
7
6
FULL
7.4%
5
4
3
MEDIUM
6.3%
2
1
0
CONTROL
1.7%
■FULL ■ MEDIUM ■ CONTROL
6 The project changed procedures in school
administration, such as the use of school management
routines and tools.
7 Lower rates of absenteeism of students, teachers and
school principals were observed in treatment schools,
especially in Full treatment schools.
8 The material distributed by ApaL at a low cost of
US$ 3.20 per student per year seems to enhance
student performance.
9 Events such as the Dia da Leitura are a beacon to
attract people to schools and encourage the community
to participate in the educational development of children.
10 The difference between the gains of students
73,1
47,8
s
c
o 30
r
e
s
2
Interactions
in Medium and Full schools and their counterparts in
Control of schools is always significant, indicating that the
amount invested in treatment schools (US$ 13.33 or the
lower amount of US$ 10.58 per child ) was sufficient to
cause an impact. The Full treatment alternative appears
to produce greater gains for each US$ 1 spent than the
Medium treatment.
IMPACT EVALUATION
WHAT LIMITS LEARNINGAbsentee
?
rates of students,
The lower performance level of girls in all of EGRA
subtasks had already been observed at the 2013
baseline and at the 2013 midline (end of school year)
evaluations. These difference persisted in 2014. Of the
1,704 third grade students assessed on the ability to
read 20 of the 30 common words in a minute, 12.9% of
boys were able, but only 7.6% of girls were. The ability
to correctly read 45 or more words of connected
text was demonstrated, at the end of third grade, by
6.1% of male students, but only by 4.2 % of female
students. Although overall performance on higher level
reading skills is dismal, the gender gap is significant and
Delay in the start of the school
worrisome.
day
Delay in the start of the school day
(in
minutes)
(in minutes)
Absentee
rates
of and
students,
teachers
and
SDs
PDs
teachers and SDs and PDs
STUDENTS
TEACHERS
SDs/PDs
70%
60%
62%
59%
50%
52%
40%
30%
34%
32%
31%
25%
20%
25%
20%
10%
0%
FULL
MEDIUM
CONTROL
EGRA SCORES
Boys and Girls – Treatment
EGRA SCORES
Groups
Boys and Girls
– Treatment Groups
Treatment Boys
Treatment Girls
11,9
Fluency
14,9
Control 52
8,2
CWPM
6,4
31,1
CLPM
Medium 47
26,2
7,6
7,0
CAP
Full 34
9,0
8,9
Oral Comp
0,0
0
10
20
30
40
50
5,0
60
10,0
15,0
20,0
25,0
30,0
35,0
■ TREATMENT BOYS ■ TREATMENT GIRLS
FIVE RECOMMENDATIONS
1 Define and implement strategies and incentives such 4 Send the message that reading is a priority by orgaas contests, awards and seek parental involvement to
reduce the high level of student, teacher and school principal tardiness and absenteeism.
nizing reading contests, giving prizes for teachers/schools
whose students make progress towards the 45 word per
minute mark by the end of third grade.
project is implemented to provide insights into the reasons why boys consistently outperform girls.
school supervision and provide support to teachers to
improve reading instruction.
2 Conduct special studies in communities where the
3 Develop and incorporate the project strategies and
procedures to reduce the gender gap that currently exists in schools where the project is being implemented.
5 Encourage specific, constructive and non-threatening