Assessing Argentina’s Preparedness for the Knowledge Economy:
Measuring Student Knowledge and Skills in Reading, Mathematical and Scientific Literacy
-Evidence from PISA 2000
Husein Abdul-Hamid, Ph.D.
University of Maryland University College
[email protected]
TABLE OF CONTENTS
I. INTRODUCTION......................................................................................................... 3
II. THE LITERATURE ON DETERMINANTS OF LEARNING ....................................... 4
III. BACKGROUND ON PISA ........................................................................................ 9
IV. PAST RESEARCH ON ASSESSING ACHIEVEMENT IN ARGENTINA ............... 11
V. ANALYTIC DESIGN AND METHODOLOGY .......................................................... 13
Data............................................................................................................................................... 14
VI. RESULTS AND FINDINGS ...................................................................................... 1
Between and within school variance ........................................................................................... 22
School educational resources ...................................................................................................... 22
Quality of Teachers...................................................................................................................... 23
Technology: Quality and not quantity......................................................................................... 23
Student-related factors................................................................................................................. 24
Family factors .............................................................................................................................. 25
VII. CONCLUSION AND POLICY IMPLICATION ....................................................... 27
REFERENCES.............................................................................................................. 32
2
I. INTRODUCTION
In the global economy, knowledge is developed and applied in novel ways. The
information revolution provides new opportunities for access to knowledge, making the use of
ideas and the effective application of technology more critical determinants of productivity than
physical abilities. In order to compete, countries must develop a high capacity, flexible
workforce that can respond to changing demands. The knowledge economy is generating skill
requirements such as problem solving and the capacity to understand and implement information
and productivity-enhancing technologies. Countries need access to global knowledge in order to
raise productivity, competitiveness and living standards. Equipping people to deal with these
demands requires a new learning model that encompasses the entire life cycle: lifelong learning
(World Bank 2003a).
Argentina is starting to emerge from one of the worst financial crises in its history. Its
relatively high levels of human capital are needed now more than ever to put the country back on
a growth path. Participating in the global economy is predicated on the strengths of its
workforce. The quality of that workforce depends on the outcomes of the education and training
system, and the capability of that system of instilling the skills and competencies demanded by
global industry today. International student assessments provide the opportunity to assess
progress towards building a workforce that is capable of making a country competitive in the
global knowledge economy.
International assessments have been used in the past to rank countries’ education
systems, and increasingly for research on growth and the determinants of success in a crosscountry context. But these assessments are also very useful for national level study, for in-depth
analysis of the determinants of learning, and for generating policy recommendations for
improving the national learning system. While most international assessments are curriculumbased, the Programme for International Student Assessment (PISA) provides the opportunity to
assess young people’s readiness for the world of work. It provides a good test of the skills for
the knowledge economy and for lifelong learning, issues that are of the utmost importance for
middle-income countries.
The results of the PISA assessment for 2000 have been used to understand how well
prepared Argentina is for the global knowledge economy and for lifelong learning. The research
is based on analyzing the results for 15-year-olds in reading, math and science. In this research
we document and analyze Argentina’s performance in an international context, and assess the
extent to which Argentina is prepared to meet the challenges of the knowledge economy. The
specific questions that this research attempts to answer include:
• What are the factors that determine effective teaching and learning of the skills and
competencies for the knowledge economy in Argentina, and how do the results compare
to other countries?
• What skills do students in Argentina possess that will facilitate their capacity to adapt to
rapid change?
• Are students in Argentina able to analyze, reason and communicate their ideas – essential
for the knowledge economy – effectively?
• Do students in Argentina have the capabilities to continue learning throughout their life?
3
•
•
Are some ways of organizing schools and school learning more effective than others for
teaching and learning the knowledge economy skills?
What influence does the quality of school resources have on student outcomes?
Socioeconomic background? Institutional factors?
The results go further than just benchmarking Argentina in the international context. The
results also allow one to determine in what areas Argentina is strong in meeting the demands of
the knowledge economy, and in what areas more work is needed to better prepare youth for the
new economy. The intended contribution to this paper is to find and indicative evidence on the
association between different variables and by no mean will show causality between the
educational characteristics and performance.
Whereas existing research has looked at the determinants of learning using national
assessments, this research is intended to go beyond the curriculum-based outcomes and analyze
the extent to which workers in Argentina possess the skills and competencies deemed necessary
to function effectively. This is a task that is made easy with the availability of PISA. It provides
a measure of reading, mathematics and science achievement that is comparable for a nationally
representative sample, and is comparable across countries.
II. THE LITERATURE ON DETERMINANTS OF LEARNING
Past research on the determinants of learning has relied on national level assessments of
learning outcomes. That research has proven very useful for helping revise curriculum in
countries and for better targeting assistance. Hanushek and Luque (2003) indicated that attention
to the quality of human capital in different countries naturally leads to concerns about how
school policies relate to student performance. Using the Third International Mathematics and
Science Study (TIMSS), the results of their analyses of the educational production functions
within a range of developed and developing countries show general problem with efficiency of
resource usage similar to those found previously in the United States. These effects did not
appear to be dictated by variations related to income level of the country or level of resources in
the schools and the conventional view that school resources are relatively more important in poor
countries also failed to be supported. Past research shows that in general, in developing countries
education inputs are more important than socioeconomic origin in explaining academic
outcomes, while in developed countries the contrary seems to be the case: family background
and home factors are more important than the school and teacher characteristics in explaining
achievement (Heyneman and Loxley 1983; Fuller 1987; Baker, Goesling and Letendre 2002).
The latter studies focus on inputs rather than institutional factors.
Significant research (for example see Greenberg 2004) highlighted the value of school
and home climate on achievement. On the relationship between school climate and performance,
Greenberg (2004) in her analysis of the United States’ National Assessment of Educational
Progress (NAEP) 2000 data, has indicated that students in schools with the highest student
behavior values had higher mean mathematics scores than students in schools in the middle or at
the bottom of the student behavior distribution. Similar relationship existed between parental
involvement and mathematics achievement and between school morale and mathematics
achievement.
4
Heneveld and Craig (1996), Patrinos and Psacharopoulos (1995) and Lockheed and
Verspoor (1991) present several factors that are important for the development of effective
schools in developing countries. Necessary basic inputs seem to include the following: (i)
instructional materials such as textbooks, supplementary teachers’ guides and materials and
library books; (ii) a curriculum with appropriate scope and sequence, and content related to pupil
experience; (iii) time for learning (the number and length of school days); and (iv) teaching
practices (such as active student learning to include discussion and group work). On the relative
effectiveness and efficiency of secondary schools, Lockheed and Jimenez (1996), in a review of
the empirical evidence from five developing countries, find that after controlling for
socioeconomic factors private schools are more effective than public schools; private schools are
more efficient than public schools; and that the greater effectiveness and efficiency of private
schools is due to differences in school-level management. In all cases they find that private
school students score higher in tests of math and language. In terms of efficiency, calculations
based on school expenditure data indicate that, on average, the unit costs for private schools are
lower than for public schools. For the same unit cost, private schools provide as much as three
times more learning, as do the public schools.
Using the TIMSS data at the aggregate cross-country level, Hanushek and Kimko (2000)
have shown that educational quality as measured by comparative tests in mathematics and
science has a consistent, stable and strong influence on economic growth. Direct measures of
labor force quality from international mathematics and science test scores are strongly related to
growth. Indirect specification tests are generally consistent with a causal link: direct spending on
schools is unrelated to student performance differences; the estimated growth effects of
improved labor force quality hold when East Asian countries are excluded; and, finally, home
country quality differences of immigrants are directly related to United States earnings if the
immigrants are educated in their own country but not in the United States. The last estimates of
micro productivity effects, however, introduce uncertainty about the magnitude of the growth
effects. The analysis explicitly considered the quality of the labor force as measured by
comparative tests of mathematics and scientific skills. The estimated impact of quality on
growth indicated that one standard deviation in mathematics and science skills translates into
more than one percentage point in average annual real growth. If, for example, quality enters
through raising the steady state level of income and growth reflects a process of conditional
convergence, the growth equation results are much larger than the corresponding results for
individual earnings. The growth model results, however, implied that the externalities must be
significantly stronger for quality than for quantity. The estimated growth effect of one standard
deviation of quality is larger than would be obtained from over nine years in average schooling.
Moreover, in absolute terms, this effect is roughly equal to estimates of average rates of
technological progress over this period. They concluded that labor force quality differences are
important for growth; that these quality differences are related to schooling (but not necessarily
the resources devoted by a country to schooling); and that quality of schooling has a causal
impact on growth. At the same time, the simple estimates of cross-country growth relationships
appear to overstate the causal impact of quality. The precise cause or magnitude of this
overstatement is unclear. There is also a distinct policy dilemma, because standard school
resource policies do not relate to the variations in quality that have been identified.
5
Barro (2001) using TIMSS distinguished between quantity of education (measured by
years of school attainment) and quality of education (measured by scores on internationally
comparable examinations). He fond that the quantity of schooling, measured as the school
attainment of males at secondary and higher levels, has a positive and statistically significant
relationship with growth. On the quality of education, Barro (2001) finds that science scores
have a statistically significantly positive effect on growth. The implication is that a onestandard-deviation increase in scores (by 0.08) would raise growth rate on impact by 1.0 percent
a year. By contrast, a one-standard-deviation increase in school attainment would increase
growth rate by 0.2 percent a year. Thus, he concludes that quality and quantity of education
matter, but quality matters much more.
Lee and Barro (1997), again using TIMSS, investigated the determinants of educational
quality in a panel data set that includes output and input measures for a broad number of
countries. The results show that family inputs and school resources are closely related to school
outcomes, as measured by internationally comparable test scores, repetition rates, and drop-out
rates. Family characteristics (income and education of parents) have strong effects on student
performance. The findings also indicate that more school resources – especially smaller class
sizes but probably also higher" teacher salaries and greater school length – enhance educational
outcomes.
In an important paper, Woessmann (2003), using TIMSS, suggests that international
differences in educational institutions explain the large international differences in student
performance in cognitive achievement tests. An econometric student-level estimation based on
data for more than 260,000 students from 39 countries reveals that positive effects on student
performance stem from centralized examinations and control mechanisms, school autonomy in
personnel and process decisions, competition from private educational institutions, scrutiny of
achievement, and teacher influence on teaching methods. A large influence of teacher unions on
curriculum scope has negative effects on student performance. The findings imply that
international differences in student performance are not caused by differences in schooling
resources but are mainly due to differences in educational institutions. Taking all countries into
consideration, he finds that the following factors positively impact science and mathematics
learning:
• Central examinations
• Centralized control of curriculum and budget matters
• School autonomy in process and personnel
• Teacher incentives (teaching methodology)
• Limited influence of unions
• Scrutiny of student performance
• Parental interest
• Intermediate level of administration
• Competition from private sector
At the national level, Fertig (2003) used OLS and quantile regressions to analyze the
determinants of German students’ achievement in the PISA 2000. Among the negative suggested
factors were:
• Schools without regular tests
6
•
•
•
•
•
Too much regulation of schools – that is, schools did not have enough autonomy to
decide upon important school relevant issues
Poor school conditions
Not enough access to modern information technology for the students
Non-native students (especially in reading)
High student-teacher ratio and shortage of teachers
Fertig and Schmidt (2002) provided, based on the individual-level data of the PISA 2000
study, a detailed econometric analysis of the way that reading test scores are associated with
individual and family background information and with characteristics of the school and class of
the 15 year old respondents to the survey. Based on quantile regressions, they interpreted the
national performance scores conditional on these observable characteristics, as the reflection of
different education systems. Their findings suggest that United States students, particularly
those in the lower quantiles, are served relatively unsatisfactorily by their system of education.
Moreover, part of the potential for improvement seems to involve measurable aspects which
could be altered and monitored easily. Overall, family background and school characteristics
play a more important role for success in PISA 2000 than previously recognized in the debate.
Furthermore, from a policy perspective the results indicate that countries directly improve the
performance of their school system by investing into tangible aspects of the system. In particular,
school conditions including teacher provision account for a sizeable fraction of student's
individual success in PISA 2000. Thus, school quality apparently does matter. Moreover, it
seems to be the students in the bottom of the performance distribution who suffer most if their
education environment is lacking. Consequently, before policy makers seek deep philosophical
distinctions underlying the education systems in the more successful countries of PISA 2000,
they should turn to improvements in the tangible aspects of the system.
Wolter (2002) analyzed the sibling size and birth-order effect on educational achievement
in Switzerland on the basis of PISA data. They showed that besides the usual factors like
education, wealth or the occupational status of parents, family configurations can play an
important role in explaining differences between students. They found an overall modest size and
birth-order effect. The sibling size effect, however, is a product of a substantial and significant
negative size effect for families with lower socio-economic status and foreign origin and a
positive sibling size effect in small, native families with a high socio-economic status compared
to single-child families with the same background. Thus, subgroups of the population seem to be
confronted with binding budget constraints, although education is free. The hypothesis that
parents of larger families spend on average less time with their children is also tested and shows
the expected negative effect of the sibling size. They presented an extended version of the sibling
size model that can account for these effects and discuss the consequences these results might
have for social and educational policy.
In one of the few if not the only study of a developing country, Abdul-Hamid (2003)
investigated the factors that affected student performance in Jordan using TIMSS 1999 data. He
found advantages for the home, family, and demographics in determining student achievement.
Parent’s education, especially for those who finished university, has played a significant role in
achievement. Parents that considered making available the education materials at home has also
been found to be correlated with achievement in children. It has also been noticed that school
governance and demographic factors have played an important role in determining achievement.
7
These factors mattered not only for achievement but also for exposure to certain teaching
behaviors, such as problem solving and critical thinking to the advantage of private and urban
schools. Among other factors were:
• Experience of teachers, as well as their qualifications and age are important. It is normal
to see that experience is associated with better achievement. Young teachers (below 39
years of age) and those with reasonable years of experience have proven to be more
likely to produce better students in mathematics. By investigating relationships and
model elimination techniques one can justify that it has to deal with some of the skills
and quality of new mathematics teachers that match with the new mathematics
curriculum (just introduced by the modernized curriculum). On the other hand, older
teachers have been correlated with better achievement in science. With the drawbacks of
the science curriculum, a quick reasoning to this observation is that the old science
curriculum (and practices) is still more dominant in teaching and older teachers are better
at it.
• Confidence in the ability to teach the assigned subjects is also a significant factor.
Teachers who were very confident of their abilities on average scored higher in
mathematics than those who were not sure about their qualifications and abilities in all
areas of the subject matter. Creative teaching and the use of technology when available
were also important.
• Teaching methods indicated that students taught by teachers who focus on memorizing
facts and rote learning are associated with lower achievement. In underachieving
countries, science teachers still focus on memorizing facts and theories, some practical
examples, some laboratory use, and very low focus on creative and scientific thinking.
The same situation applies to mathematics teachers, as they concentrate on memorizing
facts and solving problems with very small attention to projects.
• Teacher training is highly correlated with achievement. On average, having a degree in
mathematics is related to higher student achievement in mathematics. This could be a
serious issue if the qualifications of teachers are still below the guidelines that are
required by the national education law.
• Student’s attitude and confidence in their ability to focus and give attention to school
work after hours contribute to better achievement. Family encouragement is also
associated with higher scores. Some of the negative characteristics are laziness,
memorizing notes, and being involved in paid work after hours.
• Curriculum and teaching practices highlighted the need for regular standard tests and
improving reasoning and critical thinking material and methods. Giving standard tests
and explaining reasoning are associated with higher math and science achievement. In
many underachieving countries students can graduate from basic education without going
through a general test. The only general exam is at the end of high school.
One can conclude from these findings that factors that determine achievement include
home and school resources, curriculum and styles that expose students to certain teaching
behaviors, such as problem solving and critical thinking. These are the skills that build the
foundation for lifelong learning that are needed in a knowledge economy.
8
III. BACKGROUND ON PISA
PISA’s assessment focuses on young people’s ability to apply their knowledge and skills
to real-life problems and situations, rather than on how much curriculum-based knowledge they
possess. PISA does not limit itself to assessing the knowledge and skills of students but also
asks students to report on their own self-regulated learning, their motivation to learn, and their
preferences for different types of learning situations. As students begin to make the transition to
adult life, they need to know not only how to read, or understand particular mathematical
formulas or scientific concepts, but also how to apply this knowledge and these skills in the
many different situations they encounter in their lives. As technical and advanced knowledge is
implemented in the industry and heavily used in the everyday activities of consumers, only those
who are literate in such knowledge can use it to their advantage.
The emphasis is on whether students, faced with problem situations that might occur in
real life, are able to analyze, reason and communicate their ideas, arguments or conclusions
effectively. The term literacy is attached to each domain to reflect the focus on these broader
skills. In the way that the term is used, it means much more than the traditional meaning of being
able to read and write. The OECD considers that mathematics, science and technology are so
pervasive in modern life that it is important for students to be ‘literate’ in these areas as well.
Reading, mathematical and scientific literacy skills are measured with assessment tasks,
called ‘units’, that typically contain some text describing a real-life situation and a series of two
or more questions for students to answer about the text. For the mathematical and scientific
literacy components of the assessment, the text typically presents situations in which
mathematical or scientific problems are posed or mathematical or scientific concepts need to be
understood. In all three domains the ‘text’ is not necessarily prose text, but can be a diagram,
table or chart, for example.
PISA, an initiative of the Organisation for Economic Cooperation and Development
(OECD), is part of an ongoing program of reporting on indicators in education, which first
appeared in the annual, Education at a Glance, about a decade ago. Over this period, the OECD
has successfully developed indicators of human and monetary resources invested in education
and of how education systems operate. PISA arose because there was a need for regular and
reliable information on educational outcomes across countries, especially measures of students’
skills. Because it is part of an ongoing program of reporting, an aim of PISA is to monitor trends
in performance over time and give countries the opportunity to assess their systems against other
countries. Also, it gives them the opportunity to assess their education policies and reforms by
measuring students’ skills. PISA also includes countries at different levels of economic
development. Including non-OECD and developing countries gives countries the opportunity to
identify gaps and the obstacles in teaching and learning systems.
PISA was implemented for the first time in 2000, but not all countries participated at the
same time. Among other findings (OECD 2001), the survey shows that:
•
On average, 10 percent of 15-year-olds in the world’s most developed countries have toplevel reading skills, being able to understand complex texts, evaluate information and build
hypotheses, and draw on specialized knowledge. In Australia, Canada, Finland, New
9
Zealand and the United Kingdom, the figure is between 15 and 19 percent. Developing
countries, such as Peru (0.1 percent), Brazil (0.6 percent), Mexico (0.9 percent), and
Argentina (1.7 percent) did not do as well.
•
At the other end of the scale, an average of 6 percent of 15-year-olds – and in some countries
more than twice that proportion – fall below Level 1, PISA’s lowest level of reading
proficiency. A further 12 percent only make it to Level 1, which requires students to
complete very basic reading tasks such as locating a simple piece of information or
identifying the main theme of a text. Young people in these categories show serious gaps in
the foundation of literacy skills needed for further learning, impairing their ability to benefit
from further educational opportunities at school or beyond. This is especially true for
students in Mexico, Argentina and Brazil, where 16 percent, 23 percent and 23 percent fall
below level 1.
•
Japan and Korea are the top performers in mathematical and scientific literacy – defined as
the capacity of students to use the mathematical and scientific knowledge acquired in school
in a world that increasingly relies on technological and scientific advances. Among the worst
performers were Peru, Brazil, Chile, Argentina and Mexico.
•
High overall performance can go hand in hand with an equitable distribution of results.
Performance averages in the three subject areas show some countries – notably Finland,
Japan and Korea – maintaining a comparatively narrow gap between the highest and poorest
performers while still attaining high average levels. In Germany, one of the countries with
the largest gap between the highest and lowest performing students, the average performance
is below the OECD average, with much of this variation accounted for by differences
between schools. Overall, variations in student performance and the extent of variation
between schools tend to be greater in countries that differentiate at an early age between
types of program and school.
•
In many countries, boys are falling far behind in reading literacy. In every country surveyed,
girls were, on average, better readers than boys. Significant differences between countries
reflect varying abilities to provide a learning environment or broader context that benefits
both genders equally. In all participating countries, males are more likely than females to be
at Level 1 or below in reading – in the case of Finland, the best performing country, over
three times as likely. In developing countries, for example in Argentina, Brazil and Mexico,
females also scored higher than boys but the difference between boys and girls was smaller
than all other countries except Korea.
•
Students show wide differences in their general engagement with school, including big
variations in attitudes to reading and even more so to mathematics. In 20 countries, more
than one in four students consider school a place where they do not want to go. The
proportion of reluctant students is highest in Belgium (42 percent), followed by Canada (37
percent), France (37 percent), Hungary (38 percent), Italy (38 percent) and the United States
(35 percent). Developing countries such as Mexico, Brazil and the Czech Republic, have the
highest percentages of 15-year-olds who agree or strongly agree that reading is a favorite
hobby than all other countries except Portugal.
10
•
Higher average spending per student tends to be associated with higher average
performance in the three areas of literacy, but does not guarantee it.
•
Students from privileged social backgrounds tend to perform better, but differences are less
pronounced in some countries than in others. Canada, Finland, Iceland, Japan, Korea and
Sweden display above-average levels of reading literacy and a below-average impact of
social background on student performance. In the Czech Republic, Germany, Hungary and
Luxembourg, it is the opposite.
•
Results vary widely across schools, but there are countries in which the large majority of
schools achieve high standards. In countries where differences among schools are widest, a
significant part of these differences tends to be associated with the socio-economic
composition of schools.
Initial findings from country comparisons are alarming. For example, Mexico and Brazil,
the only two countries in the Latin America and Caribbean region to participate in the first phase
of PISA 20001 have performed on average below other participating countries. The top 20
percent achievers in Argentina, Chile, Mexico and Brazil scored on average below those in other
countries and similarly for those at the level between 20 percent and 50 percent. Among the
characteristics of those at the top 10 percent are: (a) the ability to evaluate information and build
hypothesis, (b) the ability to draw on specialized knowledge, and (3) the ability to accommodate
concepts contrary to expectations. Lacking these important skills that are necessary for the
knowledge economy is alarming, especially in countries with the vision toward this goal.
IV. PAST RESEARCH ON ASSESSING ACHIEVEMENT IN ARGENTINA
The analysis of PISA for Argentina marks the first occasion that this important
information is used at the country level in a middle-income developing country. It thus
contributes to a small but growing international literature. Assessing readiness for the
knowledge economy has been confined to macroeconomic indicators (see, for example, Dahlman
and Aubert 2001), and quality of education research has relied on national assessments based on
the curriculum (World Bank 2002). Use of international assessments is more limited (World
Bank 2003b), but has so far been based on curriculum-based tests.
One of the only other major international tests in which Argentina participated is the
1997 UNESCO comparative study of language and mathematics skills in primary school
throughout the region. According to the assessment, which is based on the performance of third
and fourth year students (primary school) in 13 countries, Argentina is clearly one of the better
performers. Thus, Argentina emerges at the very top together with Cuba, Brazil and Chile. Yet,
1
PISA 2000 was conducted in 32 countries: Australia, Austria, Belgium, Brazil, Canada, Czech Republic, Denmark,
Finland, France, Germany, Greece, Hungary, Iceland, Ireland, Italy, Japan, Korea, Latvia, Liechtenstein,
Luxembourg, Mexico, the Netherlands, New Zealand, Norway, Poland, Portugal, Russian Federation, Spain,
Sweden, Switzerland, United Kingdom and the United States. This is followed by PISA 2000 Plus, which was
conducted in 13 additional countries: Albania, Argentina, Bulgaria, Chile, China, Hong Kong, Indonesia, Israel,
Lithuania, Macedonia, Peru, Romania and Thailand.
11
achievement in Argentina varies across socio-economic and geographical lines: students in large
cities and urban areas score consistently higher than those in rural areas.
Argentina’s education system succeeded in creating a stock of human capital. But the
system has not succeeded in creating the human capital which will enable Argentina to develop
into a knowledge-based economy. While participation in secondary and tertiary education is
high, completion rates are unacceptably low. Curriculum at all levels is too heavily focused on
rote learning and repetition and the quality of instruction is poor.
With the enactment of the Federal Law of Education in 1993, Argentina established for
the first time a system for monitoring the quality of education: The National Learning
Assessment System (in Spanish), targeted towards pupils in the 3rd, 6th and 9th grade (primary)
as well as 3rd polimodal (secondary). The system consists of various tests grouped by areas,
such as mathematics and language. The language tests have been designed to approximate the
students’ conceptual understanding and reading comprehension. The mathematics tests have
been constructed to estimate students’ ability to solve problems, conduct mathematical
operations and read graphs and tables. Still, the actual contents of the tests change on a yearly
basis, which in turn makes comparison over time difficult. While scores were higher in 2000
than 1996 levels, the data indicates an overall decline in performance after 1998 in both language
and mathematics. In addition, while Argentina performs relatively well in the region, it tends to
better in areas of more rote learning and worse in analysis and interpretation (Naronzini and
others 2001).
The education system is, to a large extent, based on static routines rather than the
development of skills in line with the needs of the knowledge economy. Moreover, data show
that 6th grade students are especially poor at problem solving, analysis of situations and
interpretation of results (45-50 points), whereas they do significantly better in more standardized
fields such as using algorithms (69 points). Outdated curricula may be one of the core reasons
why Argentina lags behind with regard to the development of higher order skills. The private
sector has complained, for example, about the excessive dominance of rote learning and
repetition.
Several quantitative studies of the determinants of learning in Argentina have taken
place. McEwan (2001) looks at 7th grade Spanish and math achievement. Catholic schools are
somewhat more effective than public schools in producing student outcomes, but the differences
are not as pronounced as in Chile. Llach, Montoya and Roldán (1999) add 12th grade
achievement to their analysis. They find, controlling for socioeconomic status, that private
school attendance is positively associated with achievement, but at the middle school level there
are no significant differences. However, private school attendance is positively associated with
lower dropout and repetition, and promotion rates. With respect to mathematics, Eskeland and
Filmer (2002) found that school autonomy is significantly correlated to the test scores in urban
areas. Although parent participation has no significant impact by itself, it does have a positive
correlation when present together with autonomy. Neither autonomy, participation, nor their
interaction is significantly associated with language test scores. The positive effect of autonomy
and participation on schools is stronger among the poorest schools, and as strong for children of
poorer households.
12
Recently, Argentina participated in the Progress in International Reading Literacy Study
(PIRLS), along with 34 other countries. With 150,000 students at the fourth grade (9- and 10year-olds) tested, PIRLS 2001 is the first in a planned 5-year cycle of international trend studies
in reading literacy. PIRLS shows that Argentine fourth-grade students read poorly, coming in
31st out of 35 countries in reading, trailing several developing countries, including Colombia,
Turkey and Slovenia.
V. ANALYTIC DESIGN AND METHODOLOGY
It is hypothesized that there is no single but several different factors that affect basic
education quality in Argentina. Using a thorough econometric and statistical investigation all
factors related to institutions, schools, students, parents and teachers will be investigated.
We use a mixed-method approach to analyze the achievement of Argentina in this
international examination. The approach included distributional analysis of performance, factors
affecting achievement using ordinary and generalized least squares, and conditional relationships
among subgroups using quantile regressions. In the distributional analysis, we used descriptive
measures of achievement – such as analysis of means and variances (ANOVA) within Argentina
and with comparisons across countries. We also looked at measures of dispersion such as the
coefficient of variations and the range between top and low achievers. The statistical significance
of these measures was assessed and confirmed. We also used similar methods to analyze and
compare performance among schools in Argentina and for international comparisons. Overall
performance of Argentina students will be compared using standardized scores against the
international averages. Performances in reading were categorized further into six standardized
levels that cover reading comprehension, critical thinking and making inferences. Performance
of schools at different geographic locations was also assessed and compared. Argentina
performance will be compared to other Latin American countries that participated in the PISA
including Argentina, Brazil, Chile and Peru. Different level of comparisons will be conducted
and factors affecting achievement will be contrasted.
Factors affecting achievement are also analyzed and compared. In this regard Ordinary
Least Squares (OLS) methods are used to analyze the determinants of learning. The following
linear regression model is estimated:
Y = β1 X1+ β2 X2+ ε
(1)
Where Y is the test score and X1 is a vector of student variables that include household
characteristics such as socioeconomic indicators, X2 is a vector of school indicators such as
school resources, school and institutional features. However, because of the non-spherical error
term, the OLS estimation is not supposed to be highly dependable. In order to accommodate for
schools fixed effects we use the Generalized Least Squares (GLS) estimation methodology. To
accommodate the school factors and cover for the between schools and within schools
dimensions we estimate a combined model:
Y=βX+uS+r
13
(2)
Where X is the predictors’ matrix that also includes the school variables - which are fixed for
each student at the same school. S is the predictors matrix that includes student variables only.
Where u is a random element associated with school disturbances (as a second level random
variables), which we assume to have covariance matrix T. We use the GLS estimate for β as
β*=(X’V-1X)-1X’V-1Y. Where V is the variance matrix and is equal to ZTdZ’ + σ2 I, and Td is the
diagonal matrix for the variance of u. Since T and σ2 are most likely to be unknown we estimate
their values to fit the parameters by GLS. For the estimation iterative generalized least squares
will be used. The GLS variables then used in a quantile regression model as a segmentation
strategy to estimate the conditional relationships among subgroups especially between best and
worst performing groups. The basic quantile regression model specifies the conditional quantile
as a linear function of covariates. For the qth quantile, a common way to write the model (see,
e.g. Buchinsky, 1998) is
yi = xi' β q + uqi , Quantq (yi | x i ) = xi' β q , q ε (0, 1)
(3)
where Quantq(yi|xi) denotes the quantile of yi, conditional on the regressor vector xi. The
distribution of the error term is left unspecified. It is only assumed that uqi satisfies the quantile
restriction Quantq(yi | xi)= 0. The qth regression quantile (0 < q < 1) of y is the solution to the
minimization of the sum of absolute deviations residuals
1 ⎧⎪
,
,
⎨ ∑ q | yi − x i β | + ∑ (1 − q ) | yi − x i β
min
,
n ⎪⎩i:yi≥ x ,i β
β
i:yi≥ x i β
⎫⎪
|⎬
⎪⎭
( 4)
Based on quantile regressions, we interpreted the national performance scores conditional on
observable characteristics, as the reflection of different education systems.
Data
The student population chosen for PISA are 15 year-olds, who are thus assessed as they
approach the end of their compulsory schooling. For more information about the design,
development and implementation of PISA, see http://www.pisa.oecd.org. In Argentina a national
random sample of 3,983 (weighted to represent 505,404) 15-year-old students are chosen from
about 200 schools participated in the assessment. Thirty-two countries altogether took part in the
first PISA survey, which took place in 2000. Twenty-eight of these were OECD member
countries and four were non-OECD countries. In 2002, thirteen more countries participated in
PISA (called PISA+). Altogether, more than a quarter of a million students were involved in
PISA 2000. The learning domains of reading, mathematical and scientific literacy, together with
some other areas such as students’ familiarity with computers, learning strategies students use,
and students’ attitudes towards their schools, have been chosen to be the foci of PISA. The
variables used in the analysis are listed in Table 1.
Table 1: Variables used in the analyses and there descriptive statistics
Variable
Number of Siblings
Mother’s education above Secondary
Mother is working
Mean
2.57
44%
49%
14
S.D.
1.57
0.50
0.50
Home educational resources (index variable)
This was created based on the availability, in the home, of dictionary,
quiet place to study, desk for study, books, and calculators.
School is in a large city
Number of students in the school
Proportion of teachers fully certified by designated authorities
Percentage of girls at the school
Time spent on homework (index variable)
Derived from time student devote to homework per week
Enjoyment of reading (index variable), derived from students’ level of
agreement with following statements: I read only if I have to; reading
is one of my favorite hobbies; I like talking about books with other
people; I find it hard to finish books; I feel happy if I receive a book
as a present; for me, reading is a waste of time; I enjoy going to a
bookstore or a library; I read only to get information that I need; and,
I cannot sit still and read for more than a few minutes
Sense of belonging (index variable), derived from students’ reports on
their level of agreement with the following statements concerning
their school: I feel like an outsider (or left out of things); I make
friends easily; I feel like I belong; I feel awkward and out of place;
other students seem to like me; and, I feel lonely
Teacher behaviors (index variable), derived from principals’ reports on
the extent to which the learning by 15-year-olds was hindered by:
low expectations of teachers; poor student-teacher relations; teachers
not meeting individual students’ needs; teacher absenteeism; staff
resisting change; teachers being too strict with students; and students
not being encouraged to achieve their full potential
Teacher morale (index variable), derived from the extent to which school
principals agreed with the following statements: the morale of the
teachers in this school is high; teachers work with enthusiasm;
teachers take pride in this school; and, teachers value academic
achievement
Student uses computers at school, dummy that measures whether a
student uses computer at school several time a week or several times
a month
Student uses calculator at school, dummy that measures whether a
student uses the calculator at school several time a week or several
times a month
Student uses the Internet at school, dummy that measures whether a
student uses Internet several time a week or several times a month
School is public
Student uses labs at school (dummy)
15
-0.8
13%
672
85%
53%
1.29
0.33
446
0.23
0.18
0.09
0.97
0.26
0.75
0.08
0.97
0.10
1.01
-0.11
1.11
33%
0.47
68%
0.47
9%
85%
38%
0.28
0.35
0.49
VI. RESULTS AND FINDINGS
General overall performance of Argentina
Overall, Argentine students outperformed the average of students from other Latin
American (LAC) countries in reading, math and science and was closest to Mexico than others.
Figures 1-3 show the performance across participating countries in the three subjects. In
comparison to other participating countries, Argentina outperformed only three countries:
Indonesia, Macedonia and Albania. Although students in high-income countries in general
performed better than low and middle-income countries, there exist a wide variation. For
Argentina’s overall performance (see Annex I):
• Reading: 37th out of 43 countries and 2nd among 5 Latin American countries (in the
order of Mexico, Argentina, Chile, Brazil, and Peru)
• Math: 36th out of 41 and 1st in LAC countries (in the order of Argentina, Mexico, Chile,
Brazil, Peru)
• Science: 38th out of 41 and 3rd in LAC countries (in the order of Mexico, Chile,
Argentina, Brazil, Peru)
In reading, Argentina has 22 percent of students below level 1 (see Table 2 in the
Appendix for definitions), which is 16% higher than the OECD average. Students at level 1,
Figure 1: Average Reading Literacy Scores, PISA 2000
550
Average score
500
450
400
350
300
Peru
Albania
Indonesia
FYR
Brazil
Chile
Argentina
Mexico
Bulgaria
Thailand
Luxembourg
Israel
Latvia
RussianFeder
Portugal
Greece
Poland
Hungary
Liechtenstein
Germany
Italy
Czech
Spain
Switzerland
Denmark
United States
Norway
France
Iceland
Belgium
Austria
Sweden
Japan
United
Korea
Hong KongIreland
Australia
New Zealand
Canada
Finland
Country
according to the OECD, may be able to read, but have not acquired the skills to use reading for
learning. Only 2 percent of students in Argentina (higher than other LAC countries but low in
comparison to the OECD average at 10 percent) are at level 5, at which they are able to evaluate
information, build hypotheses, draw on specialized knowledge, and accommodate concepts
contrary to expectations (Table 3, Figure 1 and Figure 4).
16
Figure 4 (a): Reading Performance of (b) Percentage of students in the different
Argentina by Reading Level
levels of Reading, PISA 2000
2%
9%
Reading Literacy Level 5
Evaluating information and building
hypotheses, Drawing on specialized
knowledge, Accommodating
Concepts contrary to expectations
Figure 4 (b): Pe rcentage of stude nts in the
diffe re nt levels of Reading, PISA 2000
100%
90%
26%
21%
70%
60%
50%
40%
30%
20%
10%
0%
Country
OECD average
Korea
Spain
Italy
US
Mexico
Chile
Argentina
Below Level 1
These students may be able to read,
but have not acquired the skills to use
reading for learning
Brazil
22%
Reading Literacy Level 1
Recognize main theme on a familiar topic,
make simple connections
Average score
80%
20%
Only 31 percent of Argentine students are either at or above Level 3 proficiency, as compared to
61 percent of students in OECD countries. OECD has identified level 3 proficiency as least
acceptable. In contrast, Finland and Korea - that are among the top performers- has 79 percent
and 75 percent respectively are at or above level 3. However, among the Latin American
countries, Argentina and Mexico has the smallest percentage of students below Level 3. This
seems to indicate that Argentina and other Latin American countries need to focus on decreasing
this percentage by improving quality of reading education.
Table 3: Students in each of six levels of reading, selected countries (percent)
Country
Brazil
Argentina
Chile
Mexico
US
Italy
Spain
Korea
OECD average
Below Level 1 Level 1
23.3
22.6
19.9
16.1
6.4
5.4
4.1
0.9
6.0
32.5
21.3
28.3
28.1
11.5
13.5
12.2
4.8
11.9
Level 2
Level 3
27.7
25.5
30.0
30.3
21.0
25.6
25.7
18.6
21.7
12.9
20.3
16.6
18.8
27.4
30.6
32.8
38.8
28.7
Level 4
Level 5
3.1
8.6
4.8
6.0
21.5
19.5
21.1
31.1
22.3
0.6
1.7
0.5
0.9
12.2
5.3
4.2
5.7
9.5
Comparisons based on GDP, enrollment and expenditures on education
Argentina is below the trend-line in math, reading and science performances when
controlling for net enrollment in secondary. Argentina under-performed in comparison to other
17
countries (other than those in Latin America) when controlling for GDP per capita as a proxy for
wealth, or public expenditure on education per student (see Figures 5-7). Countries such as
Thailand, Russia, Latvia and Bulgaria, with GDP per capita lower than Argentina, performed
significantly better. In comparison Mexico has a lower net enrollment ratio in secondary than the
other LAC countries (which participated in PISA), performance is still better.
Performance and Equity
The overall variation in performances among Argentine students is large compared to the
rest of participating countries and is the opposite, for example, of Mexico. The gap between the
top 5 percent and the bottom 5 percent in Argentina is one of the highest and only smaller than
Israel. Top achievers such as Korea and Finland had the lowest. When comparing countries on
performance controlling for level of dispersion, one sees that Argentina is close to Peru but
different from Mexico, Chile and Brazil. It is similar to other LAC countries as it belongs to the
under-performing group in terms of test scores; but Mexico, on the other hand, is in the same
quadrant as Portugal, Italy and Thailand (see Figure 8).
Figure 8: Math score and dispersion across countries
600
NLD
JPN
HKG
KOR
FIN
NZL
CHE
AUSGBR
CAN
ISL DNK FRA
IRL
AUT
SWE
NO
500
CZE
USA
HUN
DEU
RUS
ESP
POL
LVA
ITAPRT
LUX
S
C
O
R
E
BEL
LIE
GRC
THA
400
BUL
MEX
CHL
MKD
ISR
ROM
ARG
ALB
IDN
BRA
300
200
240
PER
260
280
300
320
340
DISPERSION
18
360
380
400
420
In terms of socioeconomic status: Table 4 shows the results of the analysis of means and
variance tests. In addition to testing the equality of mean scores among the three socioeconomic
levels in each country, it also shows the results of the F-value, from Levine’s test of homogeneity
of variance, to test the hypothesis of equal variances among the three socioeconomic levels. The
equality of variation in scores among the three levels is a sign of high equity in scores for that
country. In other words, a high equity country will have no significant difference in the amount
of variance among individuals, regardless of socioeconomic background.
Table 4: Analysis of means and variances in math performance
among countries
F-value
Difference in Means
F-value
for means (Between SES groups) for variance
Argentina
99.0a
between all groups
1.5
Mexico
108.6a
between all groups
1.1
a
Peru
58.0
not between 2&3
22.7a
Chile
116.0a
not between 2&1
0.5
a
Brazil
214.1
between all groups
23.6a
Italy
17.7a
not between 2&3, 3&4
1.4
Korea
32.2a
only between 1&others
3.0 b
Spain
33.7a
not between 2&3
2.3
U.S.A
108.1a
between all groups
4.8a
Portugal
95.3a
between all groups
1.5
Notes:
a
b
statistically significant at the 0.001 level
statistically significant at the 0.05 level
The less the difference in means and variation in scores among and between different
groups, the higher equity is for that country. If there is a difference, then at least the amount of
variance should be similar among individuals, regardless of socioeconomic background. As in
almost all other countries there is a significant difference in mean scores between each pair of the
four socioeconomic groups. Argentina has a complicated inequity pattern as it shows high but
although uniform variation among the four socioeconomic groups. This is different for example
from the pattern in, for example, Mexico, Portugal, Spain and Italy. In Brazil, on the other hand,
there are significant differences between means and variances between each pair of the
socioeconomic groups, and this is also the case for the United States. One country among the
top achievers with high equity is Korea, with a difference in means between the lowest
socioeconomic group and the rest, and small variation in scores and no significant difference
among SES groups.
Another level of analysis is to compare performance of each socioeconomic group (SES)
in Argentina to a similar group in other countries. This relative comparison of performance of
peers across countries consists of constructing z-scores for each socioeconomic group and
comparing these standardized scores across countries (Figure 9). The z-score is a normalization
of values by subtracting the overall mean and dividing by the standard deviation. Hence it is a
measure of how far is the actual value relative to the average score in standard deviation units.
The z score for SES level l in country c is:
19
Z
where x
( x
=
lc
cl
−
σ
x
l
)
l
is the mean score for students in SES level l in country c; x
cl
students in SES level l in all countries; and σ
l
l
is the mean for all
is the standard deviation for all students in SES
level l in all countries.
In other words, we compare Argentina’s standardized scores at the socioeconomic
categorization to other countries for each socioeconomic group. We find the following:
• Students from all four SES groups scored below the international average in their
group
• Students who come from highest SES group in Argentina performed better than their
peers from LAC countries but worse than peers in majority of other countries.
• Relative to their SES groups, the richest two groups performed better than their peers
in LAC countries but poorest two groups performed lower than their peers in Mexico.
• There was a significant difference between the tops two and the lowest two SES
groups in Argentina but no difference in Mexico, for example.
In this area, later in the report we show that mother’s education and home educational resources
as proxies of wealth and awareness to learning were among the very significant factors in
performance.
Figure 9: Performance Relative to Socioeconomic Level Using Standardized Scores at Each
level
1.5
1
0.5
0
POOREST
-0.5
LEVEL 2
LEVEL 3
-1
RICHEST
-1.5
-2
-2.5
-3
JAPAN
FINLAND
KOREA
NORWAY
CZECH REPUBLIC
GERMANY
HUNGARY
SPAIN
UNITED STATES
ITALY
THAILAND
PORTUGAL
MEXICO
CHILE
MACEDONIA
ARGENTINA
BRAZIL
PERU
20
Focusing on the education production function we started by an ordinary least squares
regression analysis then tuned it up by accommodating for schools fixed effects using the GLS
model (1) above we assessed the determinates of learning. School factors were investigated then
the classrooms and student characteristics.
Schools comparison
Type and location of schools affect performance (Figures 10a, 10b, 10c, 10d, 10e, 10f,
and 10g). There is a significant difference in performance between private and public schools
(Figure 10a). On average, private schools achieved better and had lower dispersion than public
schools; at the same time, some public schools’ performance is similar, sometimes better, to
some private schools. There is a significant difference in school average score and school
dispersion based on location of the school; schools in large cities had higher average scores than
the rest of schools; schools in villages and small towns had higher dispersion than schools in
cities; top achieving schools were from large and medium sized cities. Figure 10h shows the
performance in details among schools in different locations by type. The significance of school
type and location has been also confirmed by multivariate and generalized least squares methods
for the disciplines (Table 3).
Table 10h: Distribution of test scores by school type and geographic location
700
SUBJECT= Math
600
500
500
Average score
Average Score
600
400
300
299
335
337
200
418
356
143
200
400
300
Private
N=
200
Public
100
41
105
Math
41
105
Reading
41
105
PRIVATE
Science
100
SUBJECT
PUBLIC
1
2
3
4
5
School Location (small to large )
21
6
SUBJECT= Reading
SUBJECT= Science
700
600
600
500
Average score
Average score
500
400
300
200
PRIVATE
400
300
200
PRIVATE
100
PUBLIC
1
2
3
4
5
100
6
PUBLIC
1
2
3
4
5
6
School Location (small to large )
School Location (small to large )
School educational resources were carefully investigated and only those that have direct
connection to the curriculum are found important. In science only laboratory equipments were
found to be significantly associated with performance and the value increased by level of
performance as shown by the quantile regression estimates (Table 4, Figure 11).
Between and within school variance
In most of PISA countries a considerable proportion of the variation in student
performance lies between schools. In 25 countries, differences between schools exceed one third
of the overall variation in student performance of a typical OECD country. These are the
countries depicted in the upper part of (see Figures in Annex E2). Independently of whether
overall variation is large or small, between-school variation amounts to more than half of total
variation in 13 countries and accounts for about 60 per cent or more in Austria, Belgium,
Germany, Hungary and Poland. While non-OECD countries do not belong to the five most
extreme cases, with the exception of Latvia and Thailand, they all show larger performance
differences between schools than at the OECD average where between-school variance
represents 35 per cent of the total variation in student performance. This means that in all of
these countries students encounter a quite different learning environment in terms of average
academic standards. In LAC countries and Argentina, specifically, performance is closely related
to the learning environment offered by the respective schools and learning climate.
School educational resources
Overall, the marginal effect of an increase in the quality of educational resources tends to
be highest in countries where deficiencies reported by principals are particularly pronounced.
This negative relationship may suggest diminishing returns to investment in educational
resources. However, the value of coefficients varies widely across countries. In Argentina,
Mexico and Peru, but also in Germany, a one unit change of the index is associated with
2
Source: OECD. The analyses were based on an OLS model used in each of the countries
22
difference in scores by 25 points or more, corresponding to an improvement of more than a third
of a proficiency level on the combined reading literacy scale.
Quality of Teachers
In an effort to assess the qualities of teachers that are associated with high performance of
students, we looked at teacher’s morale, behavior and attitude, and certification and
qualifications. High teacher morale, as perceived by the school principal, is associated with
better performance. Confirming with the substantial evidence that teacher’s quality are highly
correlated with student test scores (Rockoff 2004; Murnane 1975; Armor et al. 1976). For
schools where teachers work with enthusiasm, the math and reading scores seem to be higher.
When teachers take pride, it is observed that students in that school achieve more. When
teachers value academic achievement, then students significantly achieve higher in math and
reading. Moreover, teacher morale is the most significant factor associated with math
performance for top achievers as shown by results from the quantile regression (Figure 12).
Teacher behavior and teacher-related factors affecting school climate were associated
with performance. In schools where teachers had high expectations, students are observed to
perform better; and when principals feel that there is a strong relationship between students and
teachers, students perform better. Moreover, students going to schools with high levels of
teacher absenteeism, and when students are not encouraged to achieve to their full potential, they
achieve less, especially in mathematics and reading.
Teacher certification did not show a significant positive impact, although about 85
percent of students were taught by teachers that are fully certified by the designated authorities, it
does not a significant positive impact on performance. Figure 12b shows that magnitude of this
negative effect among the different levels of achievements. Math and reading are affected, most
especially among low achievers.
Technology: Quality and not quantity
The international experience of the effects of computers at schools on students’
achievement and learning shows mixed results (see Krueger & Rouse 2003; Angrist and Lavy
2002; Broozer, Krueger and Wolkon, 1992; Goolsbee and Guryan, 2002; Kirkpatrick and Cuban
1998; Wenglisky 1998). The research of Rouse and Krueger (2003) in the United States suggests
that while the use of computers may improve some aspects of students’ language skills, it does
not appear that these gains translate into a broader measure of language acquisition or into actual
readings skills. On teachers readiness or preparedness to use technology, Rowand (2000)
reported that only one-third of teachers felt well or very well prepared to use computers an the
Internet. For the Netherlands, Leuven, Lindahl, Oosterbeek and Webbink (2003) showed that
extra funding for computers and for language materials did not improve test scores in languages,
arithmetic or information processing. All point estimates are negative. For the computer subsidy
there is more evidence of negative effects, especially on the arithmetic test. Angrist and Lavy
(2001) examined the effect of computer funding on performance in Israel. The study examined
the installation of computers in many elementary and middle schools. This program provides an
opportunity to estimate the impact of computerization on both the instructional use of computers
23
and pupil achievement. Results from a survey of Israeli school-teachers show that the influx of
new computers increased teachers' use of computer-aided instruction (CAI) in the 4th grade, with
a smaller effect on CAI in 8th grade. Although many of the estimates are imprecise, on balance,
CAI does not appear to have had educational benefits that translated into higher test scores. OLS
estimates show no evidence of a relationship between CAI and test scores, except for a negative
effect on 8th grade Math scores in models with town effects. Estimates for 4th graders show
lower Math scores in the group that was awarded computers, with smaller (insignificant)
negative effects on language scores.
In Argentina mixed results have also been observed. The existence of computers and the
computer to student ratio at the school did not make that much of a difference. This was tested
using computer-to-student ratio, computers available to students only or teachers only, as
reported by school principle, but in schools where the use of computers is significant, in
Argentina, students achieved significantly higher than other students in reading, math and
science (while controlling for other factors). Moreover, this is confirmed using quantile
regression analysis (Figure 13). The analysis highlights two major findings for Argentina in this
regard:
a. Although using computers at school was associated with positive achievement,
we notice that in mathematics using calculators at school, as reported by students,
also played an important factor. Hence low threshold technology can also be
helpful. Table 5 shows the performance and calculator effects. When controlling
for achievement levels as revealed by the estimates from the quantile regressions
we see that calculators showed higher contribution to achievement than computers
among low achieving students but computers played this role for high achievers
(Figure 14).
b. Moreover, giving the opportunity to students to use computers at school has
contributed most to achievement in reading than to science and mathematics. This
may have been due to availability of software and learning modules related to
reading than in science and mathematics (Figure 13).
In order to put the findings into perspective, one must not forget the issue of costefficiency. While the use of computers and calculators were found significantly associated with
performance especially in math, calculator are very cost effective as compared with computers.
The price of a computer is many times higher than a capable calculator (or $1000 to $10)
Student-related factors
Instrumental motivation as measured by students perception and understanding of
education as a mean to increase their job opportunities; to insure that their future will be
financially secure; and to get a good job was seen to be significant in the GLS models for science
and mathematics. When students are aware of the importance of studying mathematics and
science in the labor market and their future careers, they tend to achieve better in these areas.
The level of association also varies between the different achievement quantiles. This was more
24
significant for high achieving students than low achievers in science and was only significant for
the top two achieving quantiles (75th and 90th) in mathematics as shown in Figure 15.
The awareness and enjoyment levels of the subject matter were important in reading
performance and among the different achievement groups as shown in Figure 17. Also,
allocating time and working out class assignments and homework was associated with better
performance. Noticeable importance of this factor was observed among low achievers in
mathematics and top achievers in science as shown in Figure 18.
Family factors
Most important were mother’s education level and availability of educational
resources in the home of the child. Mother’s education level make the most significant effect in
science specially among the top achievers. Mother’s work negatively affected the performance in
mathematics but not in the other fields. This may be due to the fact that reasonable amount of
practice, assistance and guidance at the home may help in understanding and mastering
mathematical concepts (Figure 19).
Educational resources at home affect reading performance the most and its effect
increases by the achievement level; its effect on the performance of top achievers was twice as
much as for bottom achievers. Educational resources include books and computer in the house
(Figure 20). Table 6 summarizes the results.
25
Table 6: Summary of performance factors
School Factors
Positive Factors
• Private school over public in all
subjects
• More girls in school is associated
with better performance
Mixed Effects
Negative factors
High student-teacher ratio
School educational resources:
• Students who used computers (effectively) at associated with low score
school achieved better in all subjects
• Using calculators at school has positive
effect on math achievement
High percentage of certified
• Total number of computers available to
teachers has negative effect
teachers does not have significant positive
impact
• Location of school (cities specially • Computer-Student ratio at school does not
large ones) has positive influence
have a clear impact
(villages and small towns
disadvantaged)
Existence of the Internet at
school was not associated with
good performance
• Ηigher school hours was associated • Availability of science equipments and
with better performance.
laboratories
Teacher related factors
Positive Factors
• High level of relationship between teachers and students (perceived by school
principal)
Negative factors
Certification didn’t seems to
be associated with
performance.
• High teacher morale is associated with higher scores (perceived by principal)
• Teacher behavior and related factors affecting school climate associated with high
scores
Degree in the subject matter, especially in science, was associated with positive
performance
Student and learning factors
Positive Factors
• Interest in subject has positive effect
• Student perception of relationship with teacher (get along, interest in student, listen, extra help, treat fairly)
• Improving students sense of belonging at school was associated with better performance
• Time on homework associated with better performance
General Characteristics
Positive Factors
Negative factors
• Boys achieved better than girls in math and science but girls perform better (but big Mother’s employment
associated with low
difference) in reading
performance
Number of siblings
• Mother’s education (above secondary) associated with better performance
• Home educational resources associated with high performance in math and reading
26
VII. CONCLUSION AND POLICY IMPLICATION
Although Argentine students performed similar to their peers in most other Latin
American countries, more could be done to bring them to a comparable level with other OECD
and PISA participating countries. An immediate intervention and amendments to policy are
recommended to be taken to: further master the reading, math and science skills that are needed
for a knowledge economy; and overcome the abnormal and high dispersion in performance
among students. Comprehensive modeling and analysis of the education production functions in
which: both the fixed and random effects, and the robustness of the factors among the different
performance groups and levels were used to analyze the performance. Student’s performance in
Argentina is associated mostly with: school and learning climate; quality of teachers, their
instructions and their ability to use resources; and whether students are encouraged, guided and
oriented to be effective learners. As mentioned earlier that the analysis conducted with this type
of data indicate association and does not show cause and effects. The findings show evidence of
significant relationships that need future follow-ups to investigate how each indicator is affecting
performance.
a. Performance issues related to schools and teachers
What is working?
School, teaching and learning climate: When some or all of the following are available
students tend to perform higher:
• High level of teacher morale: This has been detected and confirmed by analyzing
variables such as: overall morale of teachers at school as reported by the school principal;
level of enthusiasm of teachers; teachers take pride in their schools; and teachers value
academic achievement.
• High level of relationship between students and teachers: This was revealed by
examining: level of expectations of teachers; student-teacher relationship; teachers
meeting individual student’s needs; level of teacher absenteeism; staff resisting change;
whether teachers are strict with students; and whether students are being encouraged to
achieve their full potential.
• Providing an environment in which students feel a high sense of belonging. Examining of
students’ reports on their level of agreement with issues concerning their school, reveal
that the following were important: students feel that they are liked, involved and feel like
insiders and not outsiders; making friends easily; the student feels like he/she belong at
school; student don’t not feel awkward, out of place, or lonely.
• Students are guided and encouraged to use calculators, computers, science labs and
technology frequently and effectively.
• School type and coordinates: private schools; proportion of girls at school; location of
school in the advantage of cities.
In terms of policy around the issue of school climate, the following could be introduced or
improved:
• Initiate awareness campaigns at schools to highlight the importance of improving school
climate and curricula to accommodate that
27
•
•
•
•
•
•
•
Train teachers and school authorities to become customer/student oriented and improve
their level of confidence and sense of belonging
Provide incentives and activities for teacher to improve their morale. For example:
students can nominate best teachers for monetary a ward. Provide an opportunity for
teachers to be innovative by offering “grants” to do that. Provide raises based on student
achievement.
Help build a school environment in which high level of satisfaction and interest in
teaching and learning are present. Giving incentives to teachers (for example with extra
pay) to work more with students and guide them.
In order not to affect teacher’s morale by having a teacher assigned to weaker student
population make sure to apply a mechanism to diversify the student population in terms
of student’s competencies.
Provide scholarships or opportunities to needy groups to take advantage of the success of
private schools. More private (-like) schools in rural and small cities.
Learn from best practices at successful public and private schools and apply them to low
performing schools.
Rotate teachers from private/successful public schools to go to low performing schools (if
possible)
What is not clear?
Existence of computers, calculators and labs for science. The effective use of such resources
was associated with performance and not its existence and its numbers. In this regard serious
consideration of the following may help:
• Rethink the resources issues. Do not focus on computer-student ratio or number of
computers or Internet access at school (because they were found not to have a significant
association with performance).
• Think of cost-efficient spending. Calculators in math courses are more cost-effective and
associated with learning better than the expensive computers.
• Build the curriculum to accommodate the usage of computers with proper training of
teachers and guidance to students. Computers if not used carefully in the curriculum may
not pay off as seen in other countries (like Israel).
• Train teachers and students to use computers effectively in relation to their course work.
• Research is needed to identify best practices in using technology or material at schools.
• Make sure that science students see experiments and use labs
What is not working?
Teacher certification, whether a degree is obtained in the subject matter, and student-teacher
ratio. The following policy suggestions could be implemented:
• Conduct a comprehensive evaluation study to assess the type, method, and
procedure/process of certification that teachers receive.
• Study why having a degree in the subject matter is not associated with high performance.
It could be that those teachers know the subject matter but are not trained on teaching it.
Or may be good quality graduates don’t choose the teaching track. This needs to be
investigated in a focused study.
b. Issues related to students and learning
28
In addition to the relationship between students and their teachers and the sense of their
belonging at school, the analyses shows that the interest in the subject matter was an important
factor. Surprisingly, working on homework was not a significant factor in performance. In order
to reduce the gap and the variation in performance between students the following policy issues
may help:
• Initiate tutoring function for poor students and students in need (with arrangement to
compensate participating teachers).
• To overcome the dispersion in scores, functions to mix students in the learning process
could be productive. Team or group work could be a good approach. During these
collaborative educational opportunities, weak students may learn from better students.
• Introduce programs or enhancement to the curricula to make subject matter more
interesting to students and make it related to applications and environments of high
interest to them. For example, case studies could be introduced in the curriculum based
on current stories of interest to children. Also books could be written to use
conversational approach that connect students to theory and not only the theory in a “dry”
fashion..
• Make homework as part of the course and guide students while working on homework.
By making part of the class time for working or starting the work on homework. Also this
can be done by assigning student in groups during class to solve selected homework
problems.
Finally, determinants of effective teaching and learning of the skills and competencies needed for
the knowledge economy in Argentina, as measured from the PISA 2000 data, are related
significantly to the school climate in which:
• teachers have the instructional qualities and enthusiasm; teachers are well prepared to
teach and not only certified.
• students are encouraged to learn, not left alone, and assisted to become effective learners
• instructional resources are used effectively
• teachers have positive attitudes and their relationship with students is constructive and
intends to encourage, guide and help students achieve better
The gap in students’ performances is alarming as it is bigger than in most countries. While
this phenomenon could delay the country’s efforts to adapt to rapid change associated with a
knowledge economy, at the same time it shows that some students and schools have done
reasonably well. So the efforts should be focused on bringing everyone up and focus on the
poorly achieved schools and students and reduce the gap. Within and between school
differences were observed; therefore follow up studies need to take place to identify how low
and high performing schools function and may help shape a moderate reform to meet the
demanding needs of knowledge economy.
The fact that only 2 percent of Argentine 15-years olds “are able to locate and possibly
sequence or combine multiple pieces of deeply embedded information; infer which information in
the text is relevant to the task; and deal with highly plausible and/or extensive competing
information” and the other 98 percent cannot is of main concern. Having a large percentage of
students that cannot critically evaluate, hypothesize, drawing on specialized knowledge or deal
with concepts that are contrary to expectations and draw on a deep understanding of long or
29
complex texts. Adding to that the low percentage of student that is able to solve real application
problems –that have mathematical or science concepts – may lead to a suspicion that a large
percentage of students in Argentina are not able to analyze, reason and communicate their ideas
– essential for the knowledge economy – effectively.
In this regard we can say that extensive efforts need to be taken to ensure that students in
Argentina have the capabilities to continue learning throughout their life.
In conclusion, Argentina’s overall performance is below the OECD level. Given that the
PISA tests on basic knowledge needed for a knowledge economy we can say that Argentina
needs to start implementation of strategic plans to identify gaps and implement remedial
interventions to catch up and improve their overall performance in reading, math and science.
The data indicates that there are relatively successful institutions and students and seem to be
capable to do better. The analyses also seem to provide tips on where the investigation needs to
start:
¾ How could the school climate be improved and what are the components of the
successful teaching environment?
¾ What instructional material needs to be added and how to be used effectively to
improve teaching and learning in the core skills?
¾ How can the family and community get involved to help their children achieve better
and how to reduce the influence of socioeconomic disadvantages of some groups,
students and communities?
In terms of inputs it has been confirmed in the education production function that the use
of calculators, computers and science labs are significantly associated with better performance.
The efficiency in using resources needs to be investigated further to assess its effectiveness and
benefit to cost value. There was an observed benefit of low threshold instructional resources
(such as calculators and necessary science equipments) but the association between achievement
and presence and use of high threshold resources (such as computers and Internet) needs to be
carefully studied. It is clear that significant attention needs to be given and serious steps need to
be taken to improve the overall performance. In this regard, looking at the education production
function under several simulated scenarios we find that changes in the above three areas have the
potential to increase the overall score of the bottom achievers by about 100 percent. Figures in
Annex F show the effect of improving the school climate on improving the average math score
for the bottom 10 percent achievement group by just providing them with what is available to the
top achievers in Argentina. It also shows more scenarios of the effect of the several education
inputs on science performance.
Clearly teaching for the knowledge economy requires a dynamic environment and
teachers and staff should not resist change and need to have a reasonable sense of flexibility.
Teachers cannot be rigid and too strict with students and should be able to encourage students to
achieve their full potential. More importantly – than the physical and the resources– Argentina
needs to be looking at improving the learning pedagogical environment and climate. These
factors were robust and confirmed by the quantile regressions with the three test subjects. School
climate has been found to be of measurable importance for the different groups of achievement.
In other words, the climate factors played a significant role in the performance of low and high
30
achievers indiscriminately. For example, when controlling for all other variables between 11 to
15 science (similar for math) points were associated with a unit increase in the index of teacher
morale3 and a 10 to 20 reading points to an increase in one unit of student sense of belonging at
school.
Given the fact that some significant number of students and schools are doing fine make
us think of more active collaboration between schools and students within schools. Enhancing
the style of learning to include sharing/teaming may help in improving the teaching and learning
environment. Introducing group and team tasks to the curriculum may enhance studying and
learning from peers. Interventions to help weak and unmotivated students during school hours or
after school may reduce the dispersion in performance between students. Providing programs
that involve tutoring or subsidized or sponsored services have been found to be effective in other
countries such as Korea. In Korea, the government has been helping in encouraging tutoring,
reducing its cost, and providing alternatives to help in this function. Parents also are active to
initiate private tutoring to their children. There have been many forms of private tutoring:
inviting tutors at home study, attending private cramming institutes or even participating in webbased cyber teaching programs. The government has also been helping in providing e-learning
to schools. Schools provide supplementary instruction program to students after school hours,
subject to the decision of school council. Also in Korea, despite having high ratio of students to
teacher the persistence of students to get into higher education encourages them to participate in
tutoring programs. In Korea it is known that they support the best treatment of teachers.
We recommend that more research needs to be initiated to focus on evaluating teachers
conditions and school climates. Research is also needed to identify best practices especially to
understand how and why some schools (in the data we identified them as outliers) have better
climate than others and how to use educational and instructional material effectively at schools.
Evaluating current policies and processes in certifying teachers and qualifying teachers to teach
is needed. Research to study practices that improve students and faculty sense of belonging and
their interaction and relationship. Study to research the association between performance in
basic in secondary education and admission policies in higher education is also needed. In most
countries with lose admission policies in higher education students don’t not have the interest
and the parents don’t put the pressure on teachers and schools to help their children achieve
higher. Incentives to reward teachers need to be investigated and evaluated. One incentive to be
introduced could be based on performance of their children in basic education and enrollment in
higher education.
Given that the associations in this report were drawn based on perceptions of students and
the schools principals, the findings need to confirmed by designed evaluation studies and by
controlled or quasi experiments. Most of our suggested initiatives or recommendations could be
piloted and evaluated. In conclusions based on PISA 2000 data, Argentina needs to quickly
initiate efforts to seek instructional innovations (to motivate teachers, increase effectiveness of
learners, make curriculum more interesting and effective) that would build the new standards and
the core of an education reform.
3
See Table 1 for definitions
31
REFERENCES
Abdul-Hamid, Husein. 2003. “What Jordan needs to do to prepare for the knowledge economy:
Lessons learned from TIMSS-R”. Working Paper.
Armor, D., et al. 1976, Analysis of the school preferred reading program in selected Los Angeles
minority schools. Rand Corporation Report, No.R-2007-LAUSD
Bishop, John H. 1989. “Is the Test Score Decline Responsible for the Productivity Growth
Decline?” American Economic Review 79(1): 178-197.
Bishop, John H. 1992. “The Impact of Academic Competencies on Wages, Unemployment, and
Job Performance.” Carnegie-Rochester Conference Series on Public Policy 37: 127-194.
Boissiere, M., Knight J.B. and Sabot R.H. 1985. "Earnings, Schooling, Ability and ognitive
Skills." American Economic Review 75:1016-30.
Boozer, Michael A.; Alan B. Krueger and Shari Wolkon. "Race and School Quality Since Brown
vs. Board of Education.". Brookings Papers on Economic Activity: Microeconomics,
1992.
Fuller, B. 1987. “What school factors raise achievement in the third world?” Review of
Educational Research 57(3): 255-292.
Goolsbee, A., & Guryan, J. (2002) The impact of Internet subsidies in public schools (National
Bureau of Economic Research Working Paper No. (9090).
Hanushek, Eric A. 1996. “School Resources and Student Performance.” In: Gary Burtless (ed.),
Does Money Matter?
Hanushek, Eric A. and Dennis D. Kimko. 2000. “Schooling, Labor Force Quality, and the…?
Kirkpatrick, H. & Cuban, L. (1998). Should we be worried? What the research says about gender
differences in access, use, attitudes, and achievement with computers. Educational
Technology, July-August.
Lee, Jong-Wha and Robert J. Barro. 1997. “Schooling Quality in a Cross Section of Countries.”
NBER Working Paper 6198, September.
Martin M. et. al. (2000). TIMSS 1999 International Science Report: Findings from IEA's Repeat
of the Third International Science and Science Study at the Eighth Grade. The
International Study Center at Boston College, The International Association for the
Evaluation of Educational Achievement.
Mullis, V. et. al. (2000) TIMSS 1999 International Mathematics Report: Findings from IEA's
Repeat of the Third International Mathematics and Science Study at the Eighth Grade.
Ina V.S. Mullis, et. al. The International Study Center at Boston College, The
International Association for the Evaluation of Educational Achievement.
Murnane, Richard J. 1975. Impact of school resources on the learning of inner city children.
Cambridge, MA: Ballinger.
Murnane, Richard J., John B. Willett, Frank Levy (1995). The Growing Importance of Cognitive
Skills in Wage Determination. Review of Economics and Statistics 77: 251-266.
32
OECD (2001), Knowledge and Skills for Life: First Results from the OECD Programme for
International Student Assessment, Paris.
OECD/UNESCO Institute of Statistics (2003). Literacy skills for the world of tomorrow-further
results from PISA 2000.
Plewis, I. (1997). Statistics in Education. London: Arnold.
Raudenbush, S.W., & Bryk, A.S., 2002: Hierarchical Linear Models: Applications and Data
Analysis Methods, 2nd edition. Newbury Park, CA: Sage Publications.
Rockoff, Jonah E. 2004. "The Impact of Individual Teachers on Student Achievement: Evidence
from Panel Data." American Economic Review 94, no.2 (May): 247-252.
Rowand, C. (2000). Teacher use of computers and the Internet in public schools (NCES #2000090). Washington, DC: U.S. Department of Education Office of Research and
Improvement.
Wenglisky, H. (1998). Does it Compute: The Relationship Between Educational Technology and
Student Achievement in Mathematics. Princeton, NJ: Educational Testing Service.
Woessmann, Ludger. 2001. “Schooling Resources, Educational Institutions, and Student
Performance: The International Evidence.” Royal Economic Society Conference, the
University of Durham. UK.
33