The Rebirth of Education
Chapter 1
Chapter 1: Learning Goals for the new Millennium
Success in expanding enrollments and extending grade completion in the
developing world that has been rapid and amazingly uniform across countries.
But, the pace of learning—the increase skills, competencies, knowledge—per year
enrolled in many developing countries around the world is very low so that children
make little progress from year to year. The “learning profile” that connects years
enrolled with learning progress is just too flat.
The consequence is that cohorts emerge into adulthood inadequately prepared to
participate fully and successfully in the world they will face. Many developing countries,
even those meeting goals for schooling are not meeting goals for education. This low
quality of education means that most lack even the basics, the average are far behind
global standards, and even the best students are far behind the global frontier.
The challenge of the next millennium is to build on the success in expanding
schooling for all to reach learning for all.
Introduction
The world is very near accomplishing a landmark goal of the 20th century:
universal basic schooling. This triumph sets the stage for tackling the next goal:
universal basic education, in which each child is equipped to participate fully in their
social, political and economic futures.
Although universal schooling was adopted as a goal, no one has ever believed that
the goal of schooling was schooling. Schooling is a merely a technique, and one among
many, for the accomplishment of education. The goals of education have always been
learning achievement goals: the mastery of ideas, concepts, skills, beliefs. However,
reducing the goal that every child be educated to a goal that every child be schooled had
powerful advantages. This transformed difficult learning goals into relatively
straightforward problems of logistics. But schooling goals can coincide with learning
goals, or they can diverge, depending on how much children actually learn as they
progress through school.
What fraction of, say, 12 year olds can “fill in the blank.” Literally. What
fraction of 12 year olds can complete a sentence by writing an appropriate word in the
blank? What fraction of 6 year olds? 16 year olds? The increase in ability to fill in the
blank as children get older is an example of an learning achievement profile.
Alternatively, what fraction of 12 year olds can “fill in the blank” figuratively—that is,
what fraction can think creatively, what fraction understand citizenship, what fraction can
analyze literature, what fraction can appreciate their heritage. There are many goals of
basic education, both skills and values, and for any goal (or combination of goals) there is
an empirical relationship between a measure of mastery of that goal and all children of a
Preliminary and incomplete:
Not for citation
1
The Rebirth of Education
Chapter 1
given age. This is a cohort learning achievement profile that answers the question, what
are the education accomplishments of a cohort of 15 year olds?
Cohort learning achievement is linked to schooling through the familiar measures
of progression of the cohort through schooling and the learning achievement profile by
grade. Arithmetically, the proportion of 15 year olds who can do division is just the
product of the grade attainment of 15 year olds (what fraction had no schooling, dropped
out in grade 3, are now in grade 10) and the proportion at each level of grade attainment
who can do division.
The most pressing educational problem in most of the developing world is
that learning achievement profiles by grade completion are just too darn flat.
Children are just not learning enough, even though they are capable of doing so. The flat
learning profile means that schooling goals of grade progression are not translating into
educational goals. As I will show, even many middle-income countries who have long
since met the Millennium Development Goal of universal primary schooling have more
than three quarters of their youth entering their adulthood without adequate learning
achievement, ill-equipped to participate in the future they will face.
The challenge of the next century is to produce universal education so that each
new cohort of youth entering into adulthood are adequately equipped for the complex and
rapidly changing world they will face over their lifetime. The rest of the book asks the
question of how the educational systems of developing countries can meet this challenge.
.
I)
Schooling: The success of the century
Advocacy never stops to smell the roses, but some dreams really do come true, if not
quite as imagined. On December 10, 1948 the United Nations General Assembly adopted
the Universal Declaration of Human Rights of which Article 26(1) declared:
Everyone has the right to education. Education shall be free, at least in the
elementary and fundamental stages. Elementary education shall be compulsory.
Technical and professional education shall be made generally available and higher
education shall be equally accessible to all on the basis of merit.
In 1948 universal free elementary education was a pious declaration with no hope at
all of being accomplished in the near future. This was followed by a long series of
international conferences that declared not just the goal of universal education, but also
specific target years, with the target year slipping as time went by. Conferences in the
early 1960s declared a target of 1980. The World Conference on Education for All in
Jomtien Thailand in 1990 declared a target year of 2000 for “universal primary education
and completion.” The Millennium Development Goals set a target and date: Ensure that
by 2015 children everywhere, boys and girls alike, will be able to complete a full course
of primary schooling.
Preliminary and incomplete:
Not for citation
2
The Rebirth of Education
Chapter 1
Advocacy is always unrealistic on time frames1, but the world is in fact now very near
to universal primary enrollment, that every child have at least enrolled in schooling, and
is edging nearer to universal primary completion. The vast majority of countries will
meet the MDG target for universal primary completion, and very few countries will miss
it by much.
This expansion of schooling has been a massive global transformation, as schooling
had grown even in the rich countries, but particularly in the developing world. The latest
estimates Barro and Lee (2010) show the average completed years of schooling of the
population over 15 in the developing world has more than tripled in the last 60 years
from 2.1 years (at which point 60 percent of the population had no schooling at all) to 7.1
years in 2010. The amazing point from Figure 1.1 is that average developing country
today has higher levels of schooling in 2010 (7.1 years) that the average advanced
country had in 1960 (6.8 years).
Years of schooling, population
aged 15 to 64
Figure 1.1: A massive expansion in completed years of schooling
12.0
10.7 11.0
10.0
8.8
8.0
6.0
7.7
6.2
7.1
6.8
6.2
5.2
4.3
4.0
2.0
9.6
2.1
2.6
3.4
Advanced countries
Developing
0.0
1950 1960 1970 1980 1990 2000 2010
Year
Source: Barro and Lee (2010), table 3.
This implies that the levels of grade completion in even very poor countries are
higher today than the levels in rich countries in 1970. Ghana’s aged 15 and over
population in 2010 has 7.8 years of schooling completed, which surpasses where the
average of 7.3 in the UK was in 1970. Even countries thought of as education laggards,
like Bangladesh and India are high compared to many European countries in 1970.
Amazingly, even countries thought of as basket cases like Haiti have populations aged 15
and over with more years of schooling completed than advanced countries like France or
Germany in 19702.
1
Clemens (2004) in the Long Walk to School illustrates that meeting the targets always implied rates of the
expansion of schooling systems far in excess of what any country had ever achieved.
2
Using age 15 and over emphasizes the spread of “basic” education while the advanced countries had
many youth still in school, but even comparing the “adult” population age 25 and over still puts France’s
4.95 in 1970 near Haiti’s 4.89 in 2010.
Preliminary and incomplete:
Not for citation
3
The Rebirth of Education
Chapter 1
Figure 1.2: Poor countries today have higher levels of schooling of their populations
aged 15 and above than rich countries did in 1970
Source: Barro and Lee 2010.
Progress in schooling has been amazingly uniform. While progress on some
development goals, like poverty reduction, economic growth, or governance, has been
spotty, progress on the “human development” part of the “human development index”
has been nearly universal. The Human Development Report by the UNDP in 2010 is a
retrospective on 20 years of tracking the Human Development Index (HDI) and it shows
that since 1970 there has been massive progress on the schooling dimensions across the
board with nearly every country—rich, poor, rapid growing, stagnating, democratic, nondemocratic--made enormous strides in increasing enrollment rates.
This isn’t to say the goal is complete, but as the latest reports on the MDG are
revealing in that the remaining pockets of non-completion are social dysfunctions. That
is, remained non-enrollment is concentrated among the “marginalized” (EFA 2010), girls
who are “doubly disadvantaged” (Lockheed and Lewis 2008), or the “fragile states.”
While the reports continue to emphasize the remaining challenges (that is, after all, what
advocacy does), the message that emerges from all of the cumulating statistics that
measure schooling is one of huge success. Universal primary completion has become the
norm worldwide, both among parents and communities and among nation-states. This is
an amazing accomplishment, a watershed achievement in the long history of mankind.
Preliminary and incomplete:
Not for citation
4
The Rebirth of Education
II)
Chapter 1
Schooling without enough Learning
The problem is hidden in plain sight, right in the Millennium Development Goals
themselves. The Millennium Development Goal is “universal primary education” but the
target is universal completion of primary school. But that elision of goal and target
assumes that meeting the target (schooling) meets the goal (education), but every
educated person knows that happens when you assume3.
Education prepares the young to be adults. The goal of basic education is to equip
children with the skills, abilities, knowledge, cultural understandings, and values they
will need to adequately participate as adults in their society, their polity and their
economy. The goal of parents, communities, and societies has always been an education
goal and hence learning goals. Schooling goals are just a means, one input into the
process of a real education.
The old saying that “what gets measured gets done” is not right, as not everything that
gets measured gets done, but what is truer is “what does not get measured does not get
done.” An education goal is that a cohort should emerge from youth into adult with an
education that has equipped them with the kills/values/competencies/abilities/dispositions
needed for their adulthood. Any self-respecting development agency can publish reams
of tables showing myriad aspects of schooling: enrollments, expenditures, grade
progression, completion. But on education there is next to nothing.
What links a schooling goal and a true education goal is what I call the “learning
profile”—the link between years in school or grade completion and a measure of any
skill—which can be basic decoding skills like arithmetic or reading to more sophisticated
like applying those to concrete problems to more complex cognitive skills like critical
thinking or creativity or could include skills like the ability to work in a team, or
communicate effectively with others. I will emphasize early and often that although the
empirical data I will use mostly refer to easily assessed basics like mathematics and
reading the same principles apply broadly to learning.
Figure 1.3 is the illustration of a set of hypothetical learning trajectories of four
students showing the progress of their mastery of skills in some domain as they persist
through schooling. A schooling goal is measured as movement along the horizontal
axis—another year in school moves the child along no matter what learning progress they
have made. A learning goal is measured along the vertical axis. Whether meeting the
schooling goal meets an education goal depends on the learning trajectory or learning
profile. In this case Bill meets neither a schooling nor learning goal, Jack and Jill meet
both a schooling and a learning goal and Mary reaches a schooling goal—but at a level of
mastery too low to meet a learning goal. Schooling goals—whether it be basic education
or schooling for medicine, getting a PhD, or even training to be a pilot or hairdresser-have always been set by first focusing on the learning goal—what is it that students need
to know, to a learning profile or trajectory—how much time will it take students to
master that material. But whether the learning goals are met in the duration allotted for
3
Since every educated person can spell out assume.
Preliminary and incomplete:
Not for citation
5
The Rebirth of Education
Chapter 1
the schooling hinges on the actual learning trajectory of individual students, which when
averaged over a number of students is a learning profile.
Figure 1.3: Illustration of a learning trajectory as the empirical relationship
between skill mastery and duration of exposure to instruction
Measure of mastery
Grade
Completion Goal
(e.g. MDG)
Jill
Jack
Learning Goal
Learning
trajectories of
individual
students
Mary
Bill
Year in school
Before showing any empirical learning profiles, I want to build some intuition of
what we would expect learning profiles would look like if things were working as
planned.
First, what do we mean when we say something is a “second grade” concept or that in
a given curriculum a given topic or skill is “taught in fourth grade.” Take a very
narrowly defined and basic skill like recognizing letters of the alphabet or single digit
arithmetic. The idea of grade based curricula is that there is a very steep learning profile
for individual skills. So if a curriculum says “one digit addition” (e.g. 9+8=?) is covered
in grade 1 then one might expect that most children who had not completed grade 1
would not be able to do this but that all or most children who had completed grade one
would be able to correctly do single digit arithmetic.
Second, individual skills are part of a broader domain of skills, like mathematics,
surgery, tennis, playing the piano. Mastery over the overall domain assesses a number of
skills or even the ability to integrate and apply those basic skills to practical situations.
This overall mastery averages over individual skills might be expected to evolve more
smoothly.
Let me give a simple, super-stylized, example from the domain of arithmetic.
Suppose the curriculum was such that in first grade students learned number recognition,
Preliminary and incomplete:
Not for citation
6
The Rebirth of Education
Chapter 1
in second grade one digit addition, in third grade multiple digit addition, in fourth grade
multiple digit multiplication, and in fifth grade multiple digit division. Suppose, again in
a super-stylized hypothetical that assumes away all the real complexity of assessment,
suppose there were an assessment instrument that one had one item each the perfectly
captured actual student mastery. What would we expect the learning profiles to look
like? Something like Figure 1.4—rapid skill acquisition of each individual skill in the
grade in which it is taught (which some students acquiring early and others only
mastering later) with eventual near complete mastery of each. A measure of the entire
domain would show steady progress and hence a child finishing grade 5 would “know”
how to do all five of these arithmetic operations. Figure 1.4 shows what a learning
profile of specific items plus “overall domain” (of these five arithmetic operations) would
look like under the simple assumption 10 percent of students know it before grade level,
80 percent learn in the grade in which it is taught, and five percent pick it up in each of
the two grades following the grade in which it is introduced. The basic pattern is steadily
progressing mastery over an increasing range of skills within a domain as skills are
introduced and mastered in a sequenced curriculum. The same type of exercise could be
done for other domains, such as reading, that require sequential mastery of a series of
skills to produce an overall ability—such as the ability to read with comprehension.
Figure 1.4: An entirely hypothetical learning profile of sequential mastery of
arithmetic operations to reach universal mastery by grade 6
The point is that for a learning or education goal to be achieved there has to be a
pattern of learning that takes students from their initial levels of ability to the goal over
the time spent in school. Tragically, this is not at all what learning profiles look like in
most developing countries.
I will use data from three different studies from India to illustrate the problem with
the learning profile. Why India? Mainly because there have been three very recent
pioneering efforts to measure student learning performance in India, each of which
Preliminary and incomplete:
Not for citation
7
The Rebirth of Education
Chapter 1
provides not just the usual reporting of mastery at a single grade but actual learning
profiles that track performance across grades. Each of these three studies has advantages
and disadvantages in coverage and technique, but together the three studies paint a clear
and coherent picture of incredibly shallow learning profiles, with mastery of even of the
most rote of basics weak and even weaker and less rapid progress in conceptual
understanding. I am using India to illustrate the conceptual points and then will show in
the next section on cross-national comparisons that India’s results on learning
achievement are not atypical, even of countries at much higher income levels. So, if you
are worried about education in India, fully enjoy; if you are worried about education in
any other developing country, stick with me and I’ll get back to the country that interests
you in section III.
II.A) The Flat Learning Profile in Basics: Study in Andhra Pradesh
The researchers Karthik Muralidharan and Venki Sundararaman, working with the
government India state of Andhra Pradesh (henceforth AP) the Azim Premji Foundation
and the World Bank, have carried out one of the most impressive studies of schooling and
education ever over the last few years in the. This study is striking in several respects.
First, it has been carried out at a massive scale with hundreds and hundreds of schools
across different districts in AP. Second, it has examined not just one possible
“intervention” to raise quality but a whole variety of interventions—from performance
pay to increased school grants. Third, it has used the latest techniques of
randomization—assigning schools randomly to the various “treatments” versus another
set of “control” schools so that its findings have powerful claims to have identified the
actual causal impacts of the treatments on learning. Fourth, the study developed
(together with the organization Education Initiatives, on which more below) and used a
sophisticated test that is able to both assess students “rote” learning as well as their
deeper conceptual understanding. Fifth, by testing students in multiple grades and
tracking students over time the study has produced a series of both “cross-sectional”
learning profiles—averages of skill mastery across grades—as well as actual student
learning trajectories—tracking individual students over time. Given the richness of this
research we will have many occasions to draw on its findings, but even before talking
about the causal impact of the tested policy alternatives, one rich set of findings are just
that the learning profiles look like.
[These are very preliminary and may change and Karthik has allowed me to use these
but please to not cite or reproduce without Karthik’s permission].
Figure 1.5 shows the learning profiles of individual questions involving mastery of
simple arithmetic concepts. Start with just adding two single digit numbers—the
question 9+8=? The best, though not so good, news is that 35 percent of children can
answer this “grade 1” skill, so there clearly was some learning. The bad news is that in
the grade after this skill was first taught 65 percent of children could not answer this
simple question. The worst news is that of the 65 percent that did not learn this simple
arithmetic operation by second grade—only half learned it in the next three years. By the
Preliminary and incomplete:
Not for citation
8
The Rebirth of Education
Chapter 1
time these students finish primary school only 61 percent can do the simplest addition
with carry4.
There is less than universal (or even near universal) mastery of a grade 1 or 2 concept
by the end of primary schooling, and unfortunately this vastly exaggerates how many
actually understand arithmetic operations students have acquired as opposed to having
simply memorized responses by rote. When presented with an equation that requires
even modest manipulation less than 10 percent in 5th grade can handle it.
Figure 1.5: Learning profiles across grades 2 to 5 for four specific arithmetic
operations of increasing difficulty from Andhra Pradesh India
697
+505
-------
Source: Adapted from AP study
Figure 1.6 shows similar item specific learning profiles across grades for
questions involving fractions. In fifth grade only half of the students have mastered even
what is considered a “grade 2” concept of being able to associated fractions with areas.
Again, the percent of students able to apply these concepts in even moderately complex
ways, like understanding a sentence with fractions is even lower.
That the particular “double digit” addition is answered correctly more often is likely because it is two very
small numbers in each column.
4
Preliminary and incomplete:
Not for citation
9
The Rebirth of Education
Chapter 1
Figure 1.6: Learning profiles across grades 2 to 5 for questions about fractions
from Andhra Pradesh India
What fraction is shaded?
Which has 1/5 apples?
Rama ate 1/3 an Rita ate ¼
of a chocolate, who ate
more?
Which is larger, 2/3rds of
4500 or 1/6th of 6000?
Preliminary and incomplete:
Not for citation
10
The Rebirth of Education
Chapter 1
There are two striking things about the learning profiles in figures 1.5 and 1.6. First,
there is no hugely noticeable “spike” in mastery. That is, although based on the AP
curriclar objectives questions are classed as “grade 2” or “grade 3” or “grade 4” concepts,
there is no particular evidence they are learned in these grades as indicated by a
substantial difference across grades. Second, these profiles are “flat” as the fraction of
students who appear to master these concepts (or more accurately, answer these particular
questions per year is very small.
Table 1.1. is the tabular counter-part of the graphic learning profiles showing the
evolution of the “fraction correct” and how it increases from year to year. Take a
question like being able to read the weights listed on various boxes and identify which is
the lightest. Only half of students in fifth grade could do this, up from 30 percent already
by 2nd grade. This means that in three years of instruction only about 1 in five students
gained the ability to answer this question.
Another example is how many students were able to correctly identify which of the
figures shown were a “triangle”—which only 35 percent of fifth graders could so, up fron
19 percent of second grades. This question shows almost no progress fron second to third
to fourth grades. Of the 80 percent of children who could not answer this question in 2nd
grade only 4 percent (1 in 25) appear to have gained the ability in all of third grade. This
is really stunning as it suggests that of class of 25 children who go through all of the
instruction of third grade only one child will pick up on this concept in the entire year.
Preliminary and incomplete:
Not for citation
11
The Rebirth of Education
Chapter 1
Table 1.1: Examples of increase in fraction of students answering questions
correctly
Question
Which of the following is lightest box given
below?
Tick all the triangle given below?
Grade
Fraction Percent of
Correct student
Percent
who did not
population know who
who learned learned
2
.3
3
.39
9%
13%
4
.41
2%
3%
5
.52
11%
19%
2
.19
3
.22
3%
4%
4
.25
3%
4%
5
.35
10%
13%
2
.19
3
.21
2%
2%
4
.23
2%
3%
5
.32
9%
12%
Source: AP study.
Just to emphasize again the difference of “rote” and “conceptual” mastery I should
point out the question is harder that it might at first appear. First, the triangle in the
question is pointed downward while in most of the examples of triangles children would
have seen in their examples were pointed upward. So the child has to realize that rotating
a figure does not change is classification as a “triangle.” Second, the figure shows
figures that are “triangular” but not a “triangle”—such as the cone. This requires the
child to understand that in the context of geometry “triangle” has a formal definition
which does not always correspond to “common sense” notions of “triangular.” This is
also shown in the low levels of understanding of the word “perpenidicular.”
The study from AP also aggregates these individual questions into overall measures
of learning profiles in a broad domain like “mathematics” which will also show very flat
overall learning profiles, but I hoped to introduce the notion of a “learning profile” with
individual questions so that the idea was the clearest.
Preliminary and incomplete:
Not for citation
12
The Rebirth of Education
Chapter 1
II.B) Flat Learning Profiles in Basics: The ASER experience
The study above was limited to just a few districts of one state there has been in India
a new and massive exercise that, like all assessments, has its strengths and weaknesses.
The ASER (Annual Status of Education Report) exercise has been carried out by the
Indian NGO Pratham and now the ASER Centre for the five years from 2005 to 2009
(and is planned to continue for the next five years). This assessment implements a very
simple instrument that only assesses very basic reading and arithmetic skills, but what the
instrument might lack in test sophistication is made up for by several features of the
exercise.
First, the ASER exercise surveys all children in a village, both in school and out
of school. For a variety of reasons, nearly all assessments of learning are done only for
children who are in school, which gives a potentially very over optimistic picture of an
age group’s learning achievement as out-of school (or behind grade-for-age) children are
not tested. As we shall see, even if the system is designed so that a child of age 15
“should” be in grade/class/standard 10, in India about half have either dropped out or are
in a lower grade. The grade learning profile and the cohort profile, say the learning of
those who are aged 10 or 12 or 15 are very different.
Second, the ASER exercise is massive, repeated, and available. The ASER report
covers all of rural India (sampling and surveying in urban areas is much more complex
and that was only done in 2007). The report produces estimates for almost all districts in
India (a unit with an average population of around 1.5 million). Hence the sample is
massive, over 600,000 different children are tested annually. The repetition of the
exercise year after year allows one to cross check the reliability and validity of the
estimates. And the data is available to me, which is, to me, a very important
characteristic of a data set.
I start with the ASER 2008 exercise because in that year they not only tested
simple “academic” skills like reading and writing but also practical skills like telling time
from a clock and simple calculations with money. This allows us to assess some minimal
competencies of different types. The ASER instrument is, and is meant to be, amazingly
simple (it takes an enormous amount of sophistication to be this simple and still be
useful). The test instrument for the reading exercise is on one page and has some letters,
some simple words, some simple sentences, and then a reading passage meant to be at the
level expected of children at the grade 2 level (meant to be equivalent to the grade 2
textbook passages). The child is shown the words and then progresses up or down, to
their level of achievement, with the highest that they can read the level 2 passage and the
lowest being that they cannot recognize the letters. Of course, given the many mother
tongues of children in India the test instrument is available in all of the relevant local
languages and children are given their choice of language.
The arithmetic instrument is similarly simple, children are asked if they can
recognize two digit numbers, and then move up or down. The arithmetic skills tests
subtraction of two digit numbers and the highest level tested is the division of three digit
Preliminary and incomplete:
Not for citation
13
The Rebirth of Education
Chapter 1
number by a single digit number (e.g. 824 divided by 6), which requires the ability to
understand division with carry.
The ASER 2008 also covered two practical skills: telling time and handling
money. For telling time, children were asked to give the time from pictures of two
clocks. With money children were asked questions like, if this hand has two 5 rupee
coins and the other two 10 rupee notes, which hand has more money?
I have traveled with people doing the ASER exercise in Uttar Pradesh (the largest
state of India) and in practice determining child competence the given simple levels
(recognizes nothing, letters, words, sentences, passage) was pretty clear. Moreover, it
was quite powerful to see children who were 11 years old and claimed to be in class 3 in
school who were not able to read simple words or even name letters. The out of school
tests also corresponded to my school visits, in that children who had been taught to do
arithmetic actually were unable to do it in novel settings without guidance as they really
had not understood the concepts (more on this below).
These skill sets are the very lowest of what one could conceivably define as
minimally adequate preparation for the 21st century5. I argue below that children need (a)
deeper conceptual understanding than simple decoding skills and (b) require new skill
sets such as the ability to think creatively and critically, the ability to identify and solve
problems, and the ability to communicate effectively to and work with others. But
certainly no one will argue that a child is ready for life the 21st century if they cannot read
a simple passage or tell time—and universal primary education in India does not give
even those skills.
What do the results of the ASER exercise show? Table 1.1 shows the results for
various Indian states (I show three bottom, three middle and two top states) and the all
(rural) India average. Children at the age near the end of primary school (ages 10 and 11)
are radically under equipped for the world—only 27 percent of children in India can read
a simple passage, do division, tell time, and handle money. In the states with low
learning achievement less than one in five children can do all four. Even in the famously
high performing states in India that have achieved universal enrollment, the less than half
of primary school completion aged children (10 to 11) can do all four tasks6.
The official “minimum levels of learning” (MLL) in India defines various skills children should have
acquired by class and class II includes (among many other skills) “read aloud rhymes, songs, and simple
stories” and for class V “divides a four digit number by a two digit number.” The specification was that 80
percent of the children should master 80 percent of the skills specified in the MLL.
6
While these numbers from ASER are low, similarly low levels of learning have been found in
independent assessments by a number of different researchers. Pandey, Goyal, and Sundararaman (2010)
find in their baseline in 2006 for a study in Uttar Pradesh that of children in grades 2, 3 and 4 only 13
percent could read a sentence and words and only 4.5 percent could do division. Another baseline study
done in rural Andhra Pradesh found that in class 3 only 4 percent of children could answer 20 divided by 5
and only 5 percent could subtract 491-58. Both of these suggest even worse performance than ASER.
5
Preliminary and incomplete:
Not for citation
14
The Rebirth of Education
Chapter 1
Table 1.2: Cohort assessment of skill mastery shows that nearly three quarters
primary completion aged children in India do not master four basic skills, nor
do more than a third of 15-16 year olds
Age 10 and 11
State
Reading
Level 2
and
Division
Tell
Time
and
Handle
Money
Aged 15 and 16
Do all
four
Reading
Level 2
and
Division
Tell
Time
and
Handle
Money
Do all
four
Three low learning achievement states
KARNATAKA
UTTAR PRADESH
WEST BENGAL
14.1%
20.3%
24.4%
48.6%
41.9%
42.5%
12.4%
17.3%
18.6%
45.9%
55.4%
59.7%
82.8%
78.2%
81.2%
43.8%
52.2%
55.7%
Four states near the middle
GUJRAT
ORRISA
ANDHRA PRADESH
RAJASTHAN
24.8%
36.2%
35.7%
33.1%
55.4%
65.4%
46.7%
55.6%
21.5%
32.2%
25.3%
28.4%
62.4%
64.3%
69.5%
74.5%
85.9%
85.1%
84.3%
87.8%
59.8%
62.3%
65.2%
70.6%
All India
31.7%
54.8%
26.9%
67.5%
85.7%
64.5%
86.2%
88.6%
95.8%
94.2%
83.7%
87.3%
Three high performing states
KERALA
46.9% 85.7% 44.6%
58.7% 68.0% 49.7%
Source: Author’s calculations with ASER data.
Himachal PRADESH
Even by the ages at which children are typically nearing the end of their
schooling, 15 and 16, there is still a huge fraction of children who are incapable of these
simple competencies. On average in India less than two thirds (64.5%) of children aged
15-16 in 2008 could handle all four of these skills. The difference between the states is
now even more dramatic: in the low achievement states only about half of youth can do
all four, while in the highest states these skills are practically universal (interestingly, the
practical skills, which children may have acquired out of school, are more common).
Figure 1.2 shows an estimated learning achievement profile by grade, which
shows the percent of children predicted from the data to be able to do division by each
Preliminary and incomplete:
Not for citation
15
The Rebirth of Education
Chapter 1
grade7. What is striking is how shallow the profile is in UP. An increase in one year of
grade attainment is associated with only a 6.1 percentage point increase in the children
who can do division in UP versus a 9.5 percent increase per grade attained in Himachal.
Figure 1.7: The ability to do division increases more rapidly with grade attainment
in high performing states like Himachal Pradesh than low performing states like
Uttar Pradesh
Predicted values from regression
94.9%
Predicted Percent of Students Who Can Do
Division
100.0%
85.4%
90.0%
75.9%
80.0%
66.4%
70.0%
56.9%
61.4%
55.3%
49.1%
38.0%
43.0%
36.8%
28.5%
30.7%
19.0%
24.6%
18.4%
9.5%
12.3%
6.1%
60.0%
47.5%
50.0%
40.0%
30.0%
20.0%
10.0%
Uttar Pradesh
Himachal Pradesh
0.0%
0
1
2
3
4
5
6
7
8
9
10
Highest Grade Achieved
Source: Author’s calculations with ASER 2008 data
I am not obsessing about division because I love arithmetic. The learning
achievement profile for the other skills is similarly too flat. Figure 1.3 shows the learning
achievement profile by grade in UP for four basic skills: reading, arithmetic, telling time,
and handling money. There are two striking facts which emerge.
7
This is the result of a simple descriptive linear regression of a binary indicator of whether a child can do
division or not on the students grade, age, mother’s schooling, household assets, and village and district
indicators.
Preliminary and incomplete:
Not for citation
16
The Rebirth of Education
Chapter 1
Percent of students that can perform
Figure 1.8: Learning achievement profiles by grade attainment for each of four
basic skills, government schools in Uttar Pradesh
100.0%
90.0%
80.0%
70.0%
60.0%
50.0%
can do division
can read level 2
40.0%
can tell time
30.0%
can money
20.0%
10.0%
0.0%
0
1
2
3
4
5
6
7
8
9
10
Highest grade attained
Source: Author’s calculations with ASER 2008 data.
First, at grade 5, the completion of “primary” schooling (in the way universal
primary education was originally pursued) children are nowhere near adequately
equipped for life. Only 22 percent can do division, only 41 percent can read a very
simple story, only half can tell time. Interestingly, the purely practical skill of handling
money (even though it involves arithmetic) is picked up even by those with little or no
schooling. This is a reminder that the learning achievement profile by grade attained is
descriptive about children’s competencies, but does not imply the skills were acquired in
school or anything causal about what would happen to a given child who got an extra
year of school. Even those who complete “elementary school” at grade 8 there are still
massive deficits in these very basic skills: 48 percent cannot do division, 24 percent
cannot read a simple story, 22 percent cannot tell time. The only skill that is nearly
universal is handling money—but remember that 37 percent of children can do that with
no schooling at all.
The second striking aspect of the profiles how flat they are. Take reading. Only
30 percent of children in grade 4 can read a simple passage. This is already shocking as
nearly everything about the organization of the schools, method of teaching, and
curriculum assumes that grade 4 children can read. But seven out of ten cannot read a
level 2 text in grade 4. The percent who can read increases to 41 percent in grade 5.
Some children appear to have learned to read sometime between fourth and fifth grade—
but only 10 percent of all children. So of the 70 percent of the children who could not
read in grade 4 only one in seven acquired this skill during an entire academic year. This
is truly shocking. Somehow in the available hours of instructional time in an entire year
of grade 4 only one in seven children who were not able to read acquired that truly
fundamental skill.
Preliminary and incomplete:
Not for citation
17
The Rebirth of Education
Chapter 1
Table 1.3 shows the complete detail of the levels of reading for those children
enrolled in grades 4 and 5 in 2008 in UP. Think of this as a transition matrix (the data
doesn’t actually track individual children it tracks cohorts) in which attending school
increases the level of performance from one to the other. So of the children in 20.8
percent of grade 4 children who could recognize letters but not read words, if we rule out
digression, then the 14.5 percent in grade 5 who can only recognize letters, some are
children who in grade 4 could do nothing and can now read letters, some are children
who stayed in the same category, and some in that category in grade 4 could in grade 5
now read words. The combined category of “cannot read words” fell from 28.3 percent
of children in grade 4 to only 19.2 percent of children in grade 5. So, there is some
progress, but amazingly slow. That is, of the 28.3 percent of children who arrived to
grade 4 not being able to read words only one in four (7.1/28.3) learned to read words (or
better) in an entire year of schooling.
Table 1.3: Comparing reading ability of children enrolled in
fourth and fifth grades in Uttar Pradesh shows slow
incremental progress
Level of reading
4 Level or
5 level or
mastery
below
below
Nothing (does not
7.6%
4.6%
recognize letters)
Recognizes letters, but 20.8%
28.3%
14.5%
19.2%
cannot read words
Reads words, but
17.9%
46.3%
14.5%
33.7%
cannot read a
paragraph
Can simple sentences
23.1%
69.4%
22.6%
56.3%
but cannot read a story
Can read a short, level
30.6% 100.0%
43.7%
100.0%
2 story
Source: Author’s calculations with ASER 2008 data.
The result of this cumulative slow progress is that one in five children who report
being enrolled in grade five in UP cannot even read simple words. This means that
everything else that is happening in school for these children is unlikely to make sense, as
nearly all other school work in grade 5 involves some reading.
II.B) Flat learning profiles in concepts: Educational Initiatives Study
The problem is of course even worse than the ASER numbers reveal, as they are
tests of mastery over extremely basic skills in reading and arithmetic. Moreover, even
rote or “mechanical” learning could master those skills without any deeper “conceptual”
understanding. A very recent study by the Indian think-tank Educational Initiatives
Preliminary and incomplete:
Not for citation
18
The Rebirth of Education
Chapter 1
probed many of the same skills in India, but probed deeper to examine conceptual
mastery. Two simple examples will illustrate the point.
One test question asked class 4 students to write the answer to a multiplication
problem written in exactly the standard way (Table 1.4). More than two thirds (67.1
percent) of tested Indian fourth graders could answer that question correctly. So, you
might think class 4 students “understand” multiplication. But when asked a procedurally
much easier question but in a non-standard way that required students to understand
conceptually that multiplication is repeated division (see Table 1.4) only 30.4 percent of
students could answer that question correctly. So less than half of the students who could
“do” multiplication, in the sense that they could do the procedure when the problem was
contextualized such that the student could see easily what procedure was called for (e.g.
numbers arrayed in columns) appeared to understand even the rudimentary concept that
multiplication is repeated addition.
Table 1.4: Examples of questions that test rote or mechanical or procedural
learning versus conceptual mastery.
Class 4: Maths
Write the answer.
Math
43
x 2
Sample Question 25: This is a procedural
question that checks for the process of
multiplication. 67.1% of students answered
this correctly.
Fill in the appropriate number in the
box.
3 x
= 3 + 3 + 3 + 3
Sample Question 26: This is a conceptual
question that checks whether the student is
able to link multiplication with repeated
addition of a number. 30.4% of students
answered this correctly.
Source: Educational Initiatives (EI), p 29.
A second example has to do with the simple concepts of length and measurement.
Nearly all Indian textbooks teach the concept of measurement using an example in which
an object, such as pencil, is laid next to a ruler with the base of the object at zero and the
student is taught to read off the length (see Figure 1.5). However, if the students are
presented with the object displaced from zero this throws then completely off. The single
most common answer to the question in Figure 1.5b about the length of the pencil is
6cm—even as late as grade 8. So students have learned that “length is the number
associated with the tip of the object” and less than a quarter in grades 4 and 6 (length is
Preliminary and incomplete:
Not for citation
19
The Rebirth of Education
Chapter 1
typically taught in class 3) actually understand the concept of length and measurement.
Moreover, once stuck without conceptual understanding, low capability can persist. Even
by grade 8 (nearing the completion of schooling for many students) only 34.7 percent of
students get the right answer, versus 38.8 percent who continue to believe the pencil is 6
cm long.
Table 1.5: The plurality of students do not understand “length” and measurement
Class 4, 6, 8
Math
cm
The length of the line in the figure above is 4 cm.
How long is the pencil shown in the picture? (Use the
ruler shown in the picture.)
Class 4:
5 cm (23.0%), 6 cm
(46.0%)
Class 6:
5 cm (22.1%), 6 cm
(41.7%)
Class 8:
5 cm (34.7%), 6 cm
(38.8%)
cm
.
This leads to situations like the paired comparisons from class 6 in Table 1.6 in
which the procedural question, asked in exactly the familiar way from the textbook, is
mostly answered correctly but even simple conceptual questions elicits answers worse
than random guessing. So multiplying a two digit number times a three digit number in
the left hand column is computationally complex, but even without multiple choice 48
percent of grade 6 students could do the calculation. However, knowing that multiplying
by 18 is adding up sequences of 18s appears to be rare—as the correct answer was given
slightly less frequently than one would have expected from random guessing among the
four options presented.
Preliminary and incomplete:
Not for citation
20
The Rebirth of Education
Chapter 1
Table 1.6: Procedural correctness without conceptual mastery—the performance in
questions that require conceptual mastery to answer simple questions in nonstandard ways is often worse than random guessing
ANALYSIS OF STUDENT PERFORMANCE IN CLASS 6 QUESTION PAIRS
TESTING
‘LEARNING WITH UNDERSTANDING’
Rote based/Procedural
Questions
%
Correct
Write the answer.
%
Correct
25 x 18 is more than 24 x 18. How
much more?
713
x 24
47.9%
What is the perimeter of this
shape?
15
cm
Understanding/Conceptual
questions
8
cm
A. 1
B. 18
C. 24
D. 25
A thin wire 20 centimeters long is
formed into a rectangle. If the
width of this rectangle is 4
centimeters. What is its length?
47.9%
20
cm
______cm.
21.3%
16.7%
A.
B.
C.
D.
5 centimeters
6 centimeters
12 centimeters
16 centimeters
Source: Educational Initiatives, p. 30.
The Education Initiatives study confirmed two of the main points of the AP and
ASER studies. Children at the completion of primary schooling have extremely low
mastery of even basic procedural aspects of reading, writing and arithmetic. The learning
profile is very flat. But the EI study added a new dimension of concern: that even many
of the children who would be measured by ASER (or similar assessment tools) as having
“understood” basics do not really have conceptual mastery, only the ability to mechanical
reproduce only what they have learned by rote, when presented in exactly the same
context8.
II.D) The New Basic Skills
The skills examined so far as the easiest to assess but are only a small, if essential,
part of a complete education. In fact, most people would agree that to be fully prepared
8
This sensitivity of results to question structure and presentation perhaps explains why at times
government’s results on examinations appear to show better results—they give questions that are exactly
like those provided in the text and class and hence probe for pure memorization and ability to mimic
previous test questions.
Preliminary and incomplete:
Not for citation
21
The Rebirth of Education
Chapter 1
for their lives children need a much richer array of skills and competencies. After all, a
child of 15 in 2010 will only end his or her working life at age 65 in the year 2060,
which, if the recent past is any guide, will be an enormously more socially and
technologically complex and demanding world than today. While literacy and numeracy
are essential, they are far from enough.
For instance, the labor economists Frank Levy and Richard Murnane proposed a
set of “new basic skills” that they argued that even entry level workers in the US
economy requires which included the ability to work in teams (including with people
whose views or background differs from your own), the ability to create and make
presentations (which requires the ability to understand and analyze data) and computer
literacy. Many other education experts have proposed other needed skill sets that go
beyond those skills just to have adequate earnings but also to participate in civic and
political life.
I realize that any discussion of the results of student learning using quantitative
instruments potentially creates controversy, but it is useful to emphasize what is and is
not controversial (or at least widely controversial.) That education is about learning is
not controversial. That some part of that learning is a mastery of skills like reading with
understanding or arithmetic operations is not controversial. That conceptual mastery is a
better basis for being able to answer even basic questions is also not controversial. That
the ultimate goal of skill mastery is that people lead more fulfilling lives is not
controversial. No one equates the quality of education with performance on a
standardized assessment or ability to answer multiple choice questions. If I were
asserting that measures of learning achievement like those of ASER or EI were complete
measures of the goals of education that would not be controversial, that would be just
plain silly: of course they are not.
We will return to this question later in more detail, but the complicated questions are
not actually about assessment per se, they are about the interface of pedagogy and
assessment. That is, the field of education is rife with controversy about the correct
pedagogical approach: whether schooling should be “child centered” or “activity based”
or “back to basics” or any one of the many slogans and fads that wax and wane. But even
if one has a “constructivist” approach to pedagogy, emphasizing that each child learns in
their own way and at their own pace to “construct” the concept of length and the
measurement of length, it isn’t controversial that a pencil is either 5 centimeters long or it
isn’t. While alternative pedagogical approaches may use different assessment techniques
as part of the teaching/learning process (including the possibility of eschewing
assessment altogether), there is no debate that at least certain aspects of learning can be
assessed.
III)
How close is the world to a Millennium Learning Goal?
In this section we move from looking at the learning trajectories of individual
students or learning profiles across grades to the end of basic education. These are
obviously linked as any child’s current level of learning is the result of his or her entire
Preliminary and incomplete:
Not for citation
22
The Rebirth of Education
Chapter 1
trajectory (including both learning before school, in school, and out of school. What
most international assessments do is compare the skills of a cohort—either at a given
grade in school or at a given age. These give a different way of assessing the learning
and allow comparisons across countries over large domains of the goals of education.
In this section, relying on research I have done with co-authors, I do three things.
First, I review the available internationally comparable data on learning quality,
particularly the results using the PISA (Programme for International Student Assessment)
and TIMMS (Trends in International Mathematics and Science Study) and discuss the
implications of those studies for average, low, and high performance in developing
countries. Second, I present estimates of what fraction of youth are not meeting a
proposed (low) standard for a MLG that use assumptions to get from grade attainment
and a single test to the overall cohort achievement. Third, I show the implications of the
very low average learning levels for the top of the learning distribution, the best students.
The conclusion is that even countries that have achieved universal schooling are
far from having adequate learning and this problem is not limited to “the poor” or some
disadvantaged groups only but rather the typical student emerges from their schooling
with low actual learning.
III.A) The International Evidence on Achievement
I am no expert on educational testing, but neither are you9. There are many
esoteric details in constructing assessments, but a few simple characteristics of the major
assessments that are central to understanding the available statistics (Koretz 2008). First,
what is the domain or skill set that assessments are trying to measure the mastery of?
Second, how well do those assessments capture that? Third, how are those measures
scaled?
Both PISA and TIMSS assess overall performance in large domains, the PISA
covers language, mathematics and science, the TIMSS covers only mathematics and
science (hence the MS in TIMSS). PISA is intended to capture the ability of students to
apply learning from these domains to applied, real world, contexts. TIMSS is intended to
whether students acquired conceptual mastery over the mathematics and science
curriculum.
Whether these tests are reliable and valid (and other desirable psychometric
properties) in measuring the mastery of the domains they intend to measure I will leave to
the many and distinguished experts who designed these instruments (and who are not
reading this book). I will take for granted, for now, that they are.
The third issue is the scaling of these tests. Any assessment has questions, gets
answers, assigns points to the answers for each question, assigns an importance to each
question and comes up with a number. The actual number is pretty arbitrary as it can be
9
Unless by the odd chance you happen to be one of the very small handful of people in the world who
really are expert, if so, sorry, I apologize, you are.
Preliminary and incomplete:
Not for citation
23
The Rebirth of Education
Chapter 1
transformed into any other number while preserving all the actual information (rankings
and differences). So whether the absolute number is scaled zero to 100, or 200 to 800
(the SAT), or 1 to 36 (the ACT) or whatever, is meaningless. Both the TIMSS and the
PISA have chosen to norm their results so that the average score of OECD countries is
500 and the student standard deviation is 100, this is a more or less arbitrary scaling but
as good as any other.
These assessments produce a distribution of results across students as within
every country there is a wide difference in performance better higher and lower
performing students. In comparing the measured distribution of student learning
outcomes across two countries, there are three possible measures. First, how different at
the averages (or other measures of the central tendency). Second, if one chooses as a
minimal threshold one can measure the fraction of students below that threshold (this is
akin to using “poverty lines” to compare countries distributions of incomes). Similarly to
a millennium development goal setting a target for the minimal schooling levels, one
could define a threshold of learning that was considered minimally adequate globally and
calculated the achievement of a Millennium Learning Goal. Third, one can look at the
“high performing” students and compare what fraction of students are above any given
threshold.
Figure 1.9: Distributions of performance on assessments across students and across
countries, comparing implications for the middle, bottom and top
Difference
in means
Fraction of low
performers (below
MLG)
Fraction high
performers
III.B) Averages across countries
Preliminary and incomplete:
Not for citation
24
The Rebirth of Education
Chapter 1
I start with the PISA assessment results from 2006. They state: “PISA assesses
how far students near the end of compulsory education have acquired some of the
knowledge and skills that are essential for full participation in society. In all cycles, the
domains of reading, mathematical and scientific literacy are covered not merely in terms
of mastery of the school curriculum, but in terms of important knowledge and skills
needed in adult life.” That is, PISA is not trying to capture just the simple decoding skills
of reading or procedural skills of arithmetic, but how these prepare children for
adulthood. The PISA’s target population for assessment is “15-year-old students
attending educational institutions located within the country, in grades 7 or higher.” The
sampling is therefore explicitly student based not cohort based.
Table 1.8 compares only selected countries from the PISA. I ignore most of the
OECD and Eastern European countries which, although they are the bulk of the
participating countries, mostly do well enough on these exams that I, as a development
expert don’t worry about them. I focus on the developing countries who participated.
Even for the mostly middle income developing countries who participated, it is striking
just how low the learning levels are relative to a typical OECD country10. One way to
illustrate this is to ask the hypothetical of where the average student would rank if
arrayed in the performance of students from a typical OECD country like Denmark.
Take a large middle income country like Brazil. The average in Brazil is 370 which is
worse than all but 1 of 20 students in Denmark.
10
In the table I have chosen Denmark as a typical OECD country because almost no one ever has
strong feelings about Denmark.
Preliminary and incomplete:
Not for citation
25
The Rebirth of Education
Chapter 1
Table 1.8: Achievement on the PISA assessment in 2006 for Mathematics and
Reading, for enrolled 15 year old students in selected countries
Mathematics
country:
math
Denmark
OECD
Argentina
Brazil
Chile
Colombia
Mexico
Indonesia
Thailand
Tunisia
Mean
Std.
Dev.
513
498
381
370
411
370
406
391
417
365
Where the
countries
average
student would
rank in the
distribution in
Denmark
85
92
101
92
87
88
85
80
81
92
Country Student
Standard
Deviations
behind Denmark
6.0%
4.6%
11.5%
4.6%
10.4%
7.6%
12.9%
4.1%
1.16
1.39
1.00
1.45
1.08
1.34
1.00
1.45
8.9%
12.8%
28.0%
11.0%
17.3%
12.8%
19.3%
10.0%
0.95
0.97
0.49
0.99
0.85
1.32
0.91
1.15
Reading
Denmark
OECD
Argentina
Brazil
Chile
Colombia
Mexico
Indonesia
Thailand
Tunisia
494
492
374
393
442
385
410
393
417
380
89
99
124
102
103
108
96
75
82
97
Source: PISA Volume 2, Tables 6.1,c (reading), 6.2a,c (mathematics). Cols V, VI,
and VII assume a normal distributions.
The TIMSS assessments of mathematics produce very similar overall patterns,
both for the countries who participated in both (e.g. Indonesia, Chile) and for the
countries who did TIMSS but not PISA (e.g. Ghana, Philippines, Egypt). First, all
countries are at least a full student standard deviation below the OECD mean of 500.
Second, the 50th percentile student from these countries is at a very low percentile in the
American distribution—from at most 12th percentile for Indonesia to only the 5th for the
Philippines.
Preliminary and incomplete:
Not for citation
26
The Rebirth of Education
Chapter 1
Table 1.9: The mathematics assessments of TIMSS in Grade show similar low
levels of achievement in developing countries
Country
Mean
Score
USA
Indonesia
Tunisia
Egypt
Orissa (not TIMSS,
grade 9)
Morocco
Chile
Rajasthan (not
TIMSS, grade 9)
Philippines
Ghana
Where country
X’s average
student would
be among USA
students:
Student
Standard
Deviation
504
411
410
406
401
80
89
60
93
387
387
381
378
276
Country student
standard
deviations below
the OECD mean
(500)
12.3%
11.3%
10.8%
9.9%
1.0
1.5
1.0
68
83
7.2%
6.4%
6.2%
1.7
1.4
87
91
5.1%
0.2%
1.4
2.5
Source: TIMSS 2003, Exhibit D.1 and D.2 for countries. Das and Zajonc (2009) for
two Indian states of Orissa and Rajasthan. Percentiles in cols III, IV, and IV are
assuming a normal distribution for USA scores.
The comparison with TIMSS results allows some linkage between the learning
profiles from India in the previous section and the internationally comparisons. While
India has not recently participated in a international assessment exercise (though some
states plan participation in 2011), there were exact TIMSS questions included on an
assessment in two states, Rajasthan and Orissa. Using those data Das and Zajonc (2009)
created as best they could a “TIMSS-comparable” measure. Their results suggest India is
a pretty typical developing country with average scores of 381 and 401 of those still
enrolled in India in grade 9.
This means those shockingly flat learning profiles for India led to roughly similar
performance as countries like Egypt or the Philippines.
Two economists of education, Eric Hanushek and Ludger Woessmann have done
the most complete job of combining all of the available internationally tests into a single
comparable measure of learning. How far behind on their composite measure are
students in (lower) secondary schools developing countries? Scaling their overall
country averages by the OECD student standard deviation figure 1.10 shows that
typically developing countries are one or more student standard deviations below the
OECD average. What is amazing, given the India specific data shown above about lack
Preliminary and incomplete:
Not for citation
27
The Rebirth of Education
Chapter 1
of mastery of both basics and conceptual mastery is that Indian students by this metric are
very near the top of developing countries (of course this is only of enrolled students, not
cohort assessment).
Figure 1.10: Students in most developing countries are at least an OECD student
standard deviation behind the OECD
OECD Student Standard Deviations from 500
-2.5
-2.0
-1.5
-1.0
-0.5
0.0
0.5
South Africa
Peru
Morocco
Ghana
Philippines
Botswana
Brazil
Saudi Arabia
Albania
Indonesia
Tunisia
Argentina
Chile
Lebanon
Mexico
Egypt
Zimbabwe
Turkey
Colombia
Nigeria
India
Iran
Jordan
Uruguay
Thailand
Malaysia
Denmark
United States
Singapore
Source: Hanushek and Woessmann 2009.
The first finding from international comparisons of enrolled students is that most
developing countries are between one and two full student standard deviations below the
typical OECD country.
III.B) International Comparisons: Meeting a Millennium Learning Goal?
The consequence of low average learning is that many students do not exceed
even very low levels of performance. Table 1.10 draws again on the PISA 2006 and
shows the fraction of students who were at level I of proficiency or below. This is
defined as:
Preliminary and incomplete:
Not for citation
28
The Rebirth of Education
Chapter 1
At competence level 1, students can answer questions involving familiar contexts
where all relevant information is present and the questions are clearly defined.
They are able to identify information and, carry out routine procedures according
to direct instructions in explicit situations. They can perform actions that are
obvious and follow immediately from the given stimuli.
For mathematics this is the third of six levels and corresponds to a score less than 420.7,
for reading this is of five levels of proficiency and is a score below 407.47. In Denmark
only 13 percent of 15 year old students are at or below this level in mathematics. In
marked contrast, over 70 percent of students are at or below level I proficiency in
Tunisia, Colombia and Brazil and over half of students are below this level in every
developing country reported.
In reading, this is 16 percent in Denmark but in the developing countries in the
table near or above half of all children do not reach this level (except for Chile and
Thailand).
Table 1.10: Most students in developing countries do not reach
beyond very basic levels of proficiency
Country
Mathematics at level 1
or below (420.07)
Reading at proficiency
level 1 or below
(407.47)
13.6%
21.3%
64.1%
72.5%
55.1%
71.9%
56.5%
65.7%
53.0%
72.5%
16.0%
20.1%
57.9%
55.5%
36.3%
55.7%
47.0%
58.3%
44.6%
59.0%
Denmark
OECD
Argentina
Brazil
Chile
Colombia
Mexico
Indonesia
Thailand
Tunisia
Source: PISA Volume 2, Tables 6.1,c (reading), 6.2a,c
(mathematics).
Learning goals should not be based solely on the learning of those currently enrolled,
but the skill set of the entire cohort—including those who never enrolled or dropped out.
How close are developing countries to a cohort-based learning goal, that every child
emerge from their basic schooling experience equipped for life?
I will make some calculations but no one knows. In the push for schooling,
education and learning got pushed to one side. The world has reams and reams of
schooling data and almost no learning data at all. Almost no country in the world can
Preliminary and incomplete:
Not for citation
29
The Rebirth of Education
Chapter 1
track the learning achievements of its students over time. Almost no country in the world
has cohort, as opposed to enrolled student based, measures of learning achievement. The
estimates of cross-nationally comparable measures of learning deficits that I will present
in this section are weak and incomplete—and that is shocking.
In previous work with Deon Filmer and Amer Hasan, we have calculated how
many 15 year olds in various countries are currently meeting a learning goal. To arrive
at these estimates required many assumptions.
First, what would be a minimally acceptable learning target? I have no interest in
deciding this question for any one country or for many countries. But, the PISA does
define levels of proficiency for mathematics and reading and we adopted level 1
proficiency as an illustrative “low” learning goal. As a “high” learning goal we
calculated what proportion of 15 year-olds were above the OECD mean value of 500.
Second, as we have emphasized, all international tests are school based which
means that to know the learning levels of an entire cohort some assumptions have to be
made about the learning of the untested students. As the PISA tests 15 year olds who
were in different grades we could use the learning levels across the tested grades and then
we just did the simplest possible thing a extrapolated linearly the distribution of
achievement (by extrapolating the mean and keeping the coefficient of variation
constant). Fortunately for us, the results are quite robust to the assumptions and the
Indian data (both ASER and EI) suggest linear extrapolations do not do too much
violence to actual learning profiles by grade.
The results of these calculations are sobering. Take Mexico, which is in many
ways on the verge of being a “developed” country. Schooling is essentially universal
through the primary level (there are some drop-outs, even in early grades) and the
average level of education of the population aged 15 and above is nearing 9 years. But
half of the 2003 cohort of 15 year olds was in proficiency level 1 or below for
Mathematics, 39 percent for Reading, 38 percent for Science. Suppose Mexico imagined
it was going to compete against Korea. In Korea those numbers are 2 percent,0 percent,
and 2 percent.
Things are worse in Brazil, Turkey, and Indonesia—all middle-income
countries—where, averaged across the three topics, over half of the cohort is below a
potential learning goal. In mathematics, each of these three countries have two thirds
arriving at age 15 below a standard that is essentially universal in Korea. In the United
States there has been widespread concern about the quality of education since the Nation
at Risk report in 1983 and the USA is widely and unfavorably compared to many
countries. Yet in the USA—only 3 percent of students in Science, 5 in Reading, and 9 in
Mathematics do not reach these thresholds.
Table 1.11: Even middle income countries with high average levels of schooling-who are meeting the MDG target-have one-third to two thirds of a 15 year old
Preliminary and incomplete:
Not for citation
30
The Rebirth of Education
Chapter 1
cohort not meeting a low learning goal
Country
Percent of a cohort of 15 year olds not
meeting a low Millennium Learning Goal
(level 1 proficiency or below) in 2003
Mathematics Reading Science Average
Brazil
78
57
64
66
Turkey
67
50
57
58
Indonesia
68
45
39
51
Mexico
50
39
38
42
Uruguay
39
31
31
34
Thailand
34
19
26
26
Greece
17
8
7
11
USA
9
5
3
6
Japan
3
5
3
4
Korea
2
0
2
1
Average
Cohort
years of
completion
schooling, of grade 5
2005
7.19
80.3
6.44
90.4
5.58
94.4
8.44
92.7
8.08
6.82
97.6
9.9
12.1
11.2
11.5
Source: Filmer, Hasan, and Pritchett, 2006, tables 2, 3 and 4. Barro and Lee
(2010) for average years, Filmer 2010 for cohort completion.
The TIMSS exercise tests children in specific grades, grade 4 and grade 8, rather
than an enrolled cohort and includes a variety of developing countries. This means that if
one is to calculate a cohort Learning Goal deficit, again, there have to be assumptions
made. The first part, the grade attainment of a cohort, is widely available based on
household surveys. In order to get the achievement at each grade I do the simplest
possible simulation. I take the mean score given for grade 8 (grade 9 for Rajasthan) and
the extrapolate it backwards and forward using an grade increment calculated as the
linear increment to get from a minimum of 100 on enrolling in school to the score
actually observed. I simply assume that the standard deviation of student scores is
constant across grade (although the calculations are quite robust to assuming the
coefficient of variation in constant instead). Then, using the assumption that scores
follow a normal distribution, I can calculate the fraction of students at each level of grade
attainment who would be above any given threshold. I choose a potential learning goal in
the TIMSS assessment of 420 for three reasons. This is roughly a typical country student
standard deviation on the TIMSS below the OECD score. Second, this is near the
threshold for proficiency level I in PISA (although the two instruments are not
comparable). Third, the only country with both an MLG calculation and a TIMSS 2003
score is Indonesia and choosing 420 gives an estimate of 56 percent for Indonesia, which
is modestly better than the 68 percent estimated from TIMSS, but not wildly off. Let me
be the first to admit these calculations are weak and the last to defend them. But, again, a
weakness in these calculations is strength for my overall argument about the lack of
information on learning: without these calculations there is just zero information about
the achievement of cohorts, only the enrolled students.
Preliminary and incomplete:
Not for citation
31
The Rebirth of Education
Chapter 1
The calculations from TIMSS are from poorer countries than the participants in
PISA and show even more striking calculations. Ghana has 98 percent of a cohort not
achieving an MLG of 420, Philippines 67 percent, and Rajasthan, with all caveats about
not full TIMSS comparability, 75 percent.
III.C) The best are not the brightest
Before moving to the next chapters about how to address the very low levels of
learning, it is worth stressing that the problem is not the quality of education for “the
poor” or “vulnerable groups”—the low quality of education actually goes all the way to
the top, both of the socioeconomic scale and of performance.
As I have fretted about elsewhere (Viarengo and Pritchett 2009) in a world of
global competition success in at least some sectors may depend on the number of “superstars” that are among the globally best. But the low average performance combined with
often also low variance across students implies that this means that the number of
potential superstars from these countries is very, very small. We calculate that even in
Mexico, with over 100 million people all of the students emerging each year who are in
the global top ten percent in mathematics performance could fit in one smallish
auditorium (between 3000 and 6000, depending on assumptions).
Hanushek and Woessmann (2009) have also calculated the proportion of students
in the global top ten percent.
Figure 1.11: Developing countries are producing very small proportions of students
that are in the global top ten percent
Source: Adapted from Hanushek and Woessmann 20009.
Preliminary and incomplete:
Not for citation
32
The Rebirth of Education
Chapter 1
Column IV of Table 1.4 shows the proportion of students at level 5 proficiency or
above. This is about 13 percent of Danish (and OECD) students in mathematics. But in
the developing countries one percent or less of students in all these countries reach level
5 proficiency (except Chile and Thailand at 1.4 and 1.4 percent). In Indonesia only .1
percent of students—one in a thousand—is at level 5 proficiency in reading. Even a
stellar student from Indonesia, a student from the 95th percentile who would be
competitive for admission to the top universities, would be very near the middle of the
pack of Danish students (57th percentile).
Table 1.12: There are few highly proficient students in developing
countries on the PISA
country: math
Denmark
OECD
Argentina
Brazil
Chile
Colombia
Mexico
Indonesia
Thailand
Tunisia
At level 5 or
above on
mathematics
(606.99)
13.7%
13.3%
1.0%
1.0%
1.4%
0.4%
0.9%
0.4%
1.3%
0.5%
At level 5 or above in
reading
(625.61)
5.9%
8.6%
0.9%
1.1%
3.5%
0.6%
0.6%
0.1%
0.3%
0.2%
Source: PISA 2006 Volume 2, Tables 6.1,c (reading), 6.2a,c
(mathematics).
Filmer (2010) compares the scores of the socio-economic elite in developing
countries and shows that even the “elite” are far behind the typical, or even poor, student
in richer countries.
So, while I may concentrate on universal goals even the educational or economic
“elite” of most developing countries are far from excellence. The problem in developing
countries is only that students from poor households are not learning nor that the
inequality in schooling quality is high—the problem goes deeper and affects the middle
class and elite so that even the “best” in poor countries are faring badly.
Conclusion
There has been massive success in putting butts in seats—enrollments and grade
completion have expanded massively around the globe—even in very poor countries.
However, the accumulating evidence from around the globe is that this has not lead to
Preliminary and incomplete:
Not for citation
33
The Rebirth of Education
Chapter 1
nearly the degree of learning as had been hoped. Students show very little mastery of
even the basics, much less the ability to go beyond the rote and actually understand
concepts, much much less the ability to use their conceptual mastery applied to novel
practical problems, much much much less even having laid the foundation for “new basic
skills” that are not substitutes for, but mean to build off of, the old basic skills.
I argue it is time to re-orient from schooling based goals to learning based goals. No
This is a “re”-orientation to what students, parents, communities and societies have
always cared about: learning through education. While schooling was just an
organizationally easy proxy for learning no one ever deliberately had the cruel notion that
students should be in school for school’s sake, even if they were learning little or nothing.
What is to be done?
Preliminary and incomplete:
Not for citation
34