Learning Counts:
International Seminar on Assessing
and Improving Quality Learning for All
UNESCO, Paris, 28-30 octobre 2008
Published in 2009 by the United Nations Educational,
Scientific and Cultural Organization
7 P i a c e de F o n t e n o y - 75007 P a r i s
(France)
UJUJUI.unesco.org/education
A u t h o r : M a r t a ENCINAS-MARTÍN
C o n t r i b u t o r s : A n a L u i z a M A C H A 0 0 , ] e a n BERNARD a n d A a r o n BENAVOT
Table of Contents
Acronyms and Abbreviation
5
I
Background and Introduction
II
Opening Statement
III
Panels
7
8
Panel 1: Quality education for all - visions, frameworks and practices.
Where and how do they converge?
12
Panel 1: Presentations
12
Evaluation of learning to live together programmes,
with special reference to peace education
(Anna Obura, International Consultant)
12
Quality frameworks, with a focus on the last and hardest to reach
(Margaret Meagher, CARE)
14
Multiple expectations and one practical framework?
(Kazuhiro Yoshida, Hiroshima University, Japan)
15
Panel 1: Discussion highlights
16
Panel 2: Learning assessments. How can they become more effective
tools for improving quality learning?
20
Panel 2: Presentations
20
Assessment of and assessment/or learning: challenges to enhance
education quality
(Anil Kanjee, HSRC)
21
International learning assessments
(Andreas Schleicher, OECD)
22
Do the results of national assessments contribute towards
improving public schools?
(Maria Castro, Säo Paulo State Department of Education, Brazil)
Panel 2: Discussion highlights
25
24
Panel 3: Indicators of quality learning. What is still missing in the
EFA-FTI Indicative Framework? Should EFA partners adopt common
core indicators?
30
Panel 3: Presentations
30
Indicators of educational achievement
(HansWagemaker, IEA)
30
Measuring and analyzing education quality at national/regional level
with system of indicators
(MarkAgranovitch.FIEL)
31
Information needs and the EFA-FTI Indicative Framework
(Cesar Guadalupe, UIS)
33
Panel 3: Discussion highlights
34
Synthesis and reflection on the panel presentations
(Aaron Benavot, UNESCO, General Rapporteur]
37
IV
Working Groups
Organization and objectives
Group recommendations
4
40
Proposals for concrete actions
4
Conclusion and follow-up strategies
V
4 7
Annexes
Annex 1 Working Group mandate
Annex 2 Agenda
51
Annex 3 List of participants
55
47
Acronyms and Abbreviations
APPEAL
Asia-Pacific Programme of Education for All
CONFEMEN
Conférence des ministres de l'éducation nationale ayant le français en
partage
[Conference of Ministers of Education of French Speaking Countries]
EFA
Education for All
FIEL
Federal Institute for Education Level
FTI
Fast Track Initiative
GER
Gross Enrollment Ratio
GMR
Global Monitoring Report
GTZ
Deutsche Gesellschaft für Technische Zusammenarbeit
HSRC
Human Sciences Research Council (South Africa)
[German Agency for Technical Cooperation]
IEA
International Association for the Evaluation of Educational Achievement
HEP
International Institute for Educational Planning
INGO
International Non-governmental Organization
LSAS
Large-Scale Assessments Surveys
LLECE
Laboratorio Latinoamericano de Evaluación de la Calidad de la Educación
[Latin American Laboratory for Assessment of the Quality of Education]
LSAS
Large Scale Assessment Survey
LTLT
Learning to Live Together
NER
Net Enrolment Ratio
OECD
Organisation for Economic Co-operation and Development
OEI
Organización de Estados Iberoamericanos [Organization of
Iberoamerican States]
OREALC
Regional Office for Education in Latin America and the Caribbean
PASEC
Le programme d'analyse des systèmes éducatifs de la CONFEMEN
PIRLS
Progress in International Reading Literacy Study
PISA
Programme for International Student Assessment
RTI
Research Triangle Institute
[Programme for the Analysis of the Educational Systems of CONFEMEN]
SACMEQ
Southern African Consortium for Monitoring Education Quality
TIMSS
Trends in Mathematics and Science Study
UIS
UNESCO Institute for Statistics
UNESCO
United Nations Educational Scientific and Cultural Organization
USAID
United States Agency for International Development
WAEC
West African Examination Council
Background and Introduction
The "Learning Counts" International Seminar on Assessing
and Improving Quality Learning for All (UNESCO, Paris, 28-30
October 2008) was organized by the UNESCO Education Sector
in the context of the ongoing work of the Quality Task Team
of the Education for All Fast Track Initiative (EFA-FTI). The
meeting brought together education specialists, policy makers
and practitioners from every region in the world to address the
continuing challenge posed by EFA Goal G by encouraging an
open exchange of ideas and recent experiences in assessing
learning outcomes and improving learning processes. In
view of the widely acknowledged need for urgent action on
the quality issue, the seminar was also intended as a forum
for drafting specific recommendations that could be used
to inform and encourage EFA-FTI partners in their efforts to
develop more effective strategies for assessing and improving
"recognized and measurable" learning outcomes. Thus one of
the desired outcomes of the meeting was to provide inputs to
the EFA-FTI Appraisal Guidelines and Indicative Framework,
with special attention throughout the discussions to the
problems faced by educational systems most at risk of not
meeting the objectives set by the EFA and education-related
Millennium Development Goals.
Approaches to defining the quality of education as a global
concept have been the topic of much debate; however, most
current definitions contain common elements that refer to
the goals, processes and conditions for effective learning.
Because of the diversity of approaches and applications of
operational definitions, there is a continuing need for dialogue
for the purpose of reaching a common understanding of
the fundamental elements of quality and to develop sound
approaches to measuring, monitoring and improving quality
across diverse educational contexts. The seminar provided
an open forum for discussion of these issues, drawing upon
expertise at the global, regional and local levels. The ultimate
7
EFA Coal 6.
Improving all
aspects of
the quality
of education
and ensuring
excellence of all
so that recognized
and measurable
learning outcomes
are achieved by
all, especially in
literacy, numeracy
and essential life
skills.
aim of this dialogue is to build consensus on the feasibility of
developing common core indicators, assessment tools, and
strategies which focus on improving the quality of teaching
and learning.
In addition, the seminar provided an important opportunity
for participants to provide constructive feedback on the
preliminary version of a comprehensive desk review (Learning
Counts: Desk review of Approaches to Understanding,
Assessing and Improving the Quality of Learning for All)
recently undertaken by UNESCO on the invitation of the EFAFTI Quality Task Team and with the support of the Russian
Federation. This initial draft served as a background paper
for the seminar and, in its final form, is intended to become
a key document to guide educational policies and strategies
worldwide, with special focus on countries endorsed by EFAFTI. Comments and suggestions received from seminar
participants and other interested stakeholders were taken
into account to produce the final version.
Following introductions and opening remarks delivered by
the UNESCO Assistant Director-General for Education Nicholas
Burnett, an update was given on UNESCO's progress on the
preliminary desk review. The panel presentations, which formed
the central feature of the agenda (see Annex 2), focused broadly
on three key themes and posed specific, related questions:
1. Quality education for all - visions, frameworks and
practices:
Where and how do they converge?
2. Learning assessments:
How can they become more effective tools for improving
quality learning?
3. Indicators of quality learning:
What is still missing in the EFA-FTI Indicative Framework?
Should EFA partners adopt common core indicators?
In addition, a special session was devoted to short presentations
on a range of projects related to quality learning assessment
and improvement. The working group sessions which followed
8
allowed for ¡n-depth discussion of critical emergent issues as
well as proposals for specific recommendations and followup strategies. The groups were also tasked with producing
actionable statements on (1) common core indicators of
quality in education and (2) follow-up strategies on assessing
and improving quality learning for all.
This report briefly summarizes the panel presentations and
the main issues discussed in relation to each of the three
aforementioned themes. The general principles identified by
the working groups are presented at the end of the report,
together with their combined statements and proposed followup strategies, including the establishment of a Working Group
with a clear mandate (Annex 1] to carry on the work initiated
by the "Learning Counts" participants.
Throughout
the
seminar, participants
underscored
UNESCO's important leadership role in promoting better
communication and collaboration among the organizations
involved in international and national assessment and quality
improvement programs, and in facilitating agreement among
EFA-FTI partners on the next steps to be taken.
"LEARNING COUNTS"
DESIRED SEMINAR OUTCOMES
1. Constructive comments to improve the
preliminary draft Desk Review An Overview of
Approaches to Understanding, Assessing and
Improving the Quality of Learning for All
2. Inputs for improvement of the EFA-FTI
Appraisal
Guidelines and Indicative
Framework
3. Recommendation as to the feasibility of proposing
common core indictors of quality education
4. Design of follow-up strategies
9
Opening Statement
In his opening remarks, Mr. Nicholas Burnett, UNESCO Assistant
Director-General for Education, emphasized the importance of
addressingthe quality issue, particularly at this moment of global
financial crisis. Increasingly, countries will be under enormous
pressure to cut their expenses, making it necessary to protect
expenditures for education. Any cut in spending will result in a
reduction in quality, and fundingtargeted forquality improvement
is at risk of being deemed inessential. However, the impressive
gains in access made in recent years cannot be sustained without
a parallel improvement in quality; nor can sustainable economic
and social development be achieved without equitable access to
quality learning opportunities throughout life.
Mr. Burnett also stressed that although quality is now the
overriding concern in EFA, there is no common agreement on its
definition; nor is there a globally shared approach to how it should
be measured. Still, there are common elements of quality learning
that are obvious to everyone, and little argument on the basics
of what is needed to make it happen, namely decent facilities,
sufficiently trained teachers, and efficient use of time on task.
He reminded participants of the main purpose of the seminar,
and encouraged them to work collaboratively to recommend
follow-up strategies to help guide progress toward the quality
goal. Among other things, this would mean devising new methods
and approaches to assessment and linking these assessments
more closely with actual teaching and learning processes. It
also means helping countries and the international community
to further develop the capacity to measure, assess and monitor
quality through international, regional or national assessments
In closing, Mr. Burnett observed that all three subjects of the
seminar had the same purpose: to better serve every child. He
expressed the hope that UNESCO could play a key role in this
effort by promoting better communication and collaboration
among the organizations and playing a leadership role in
facilitating agreement on follow-up actions.
10
Panels
Panel
1
Quality e d u c a t i o n for all—visions,
f r a m e w o r k s and p r a c t i c e s :
Uhere and how do they converge?
There are many different visions, frameworks and approaches
to understanding, measuring and assessing quality, as
described in the EFA 2004 Global Monitoring Report and
reviewed in the UNESCO Desk Review which served as a
background paper for the Seminar. Still, there is considerable
agreement as to the basic elements of quality learning as
well as the steps needed to ensure that such elements are
realized. At the very least quality learning involves a set of
minimal conditions and decent facilities that enable students
to learn such as a safe, healthy learning environment, trained
and motivated teachers, organized programs of learning/
curricular contents, sufficient and efficient use of time on
task, and appropriate and engaging learning materials.
Panel I presenters were invited to address the general theme
of conceptualizing quality learning by sharing their visions
and experiences. In doing so, they helped identify innovative
practices and set the stage for considering possible operational
definitions and strategies for improving quality learning across
varied contexts and systems.
Panel 1
3. UNESCO (2005).
Education for All 2004
Global Monitoring Report:
The Quality Imperative.
Paris, France
Presentations
Moderator: Robin Horn, Education sector manager,
World Bank
/////////////////////'/////••Y/.y/.\-
.'
Evaluation of Learning to Live Together Programmes, with
Special Reference to Peace Education
4. UNESCO (2009).
Learning Counts: A desk
Anna Obura, International Consultant
Review of Approaches to
Understanding, Assessing
and Improving the Quality
of Learning for All. Paris,
France
Ms. Obura's presentation opened with the statement that
quality education for all guarantees the right of every child
to learn effectively and learn something useful. She stressed
that effective learning can only be achieved in a conducive
12
learning environment, with motivated, qualified teachers and
with children staying in school long enough to ensure learning.
She then focused her presentation on Learningto Live Together
(ULI) programmes which range across a continuum from
peace education, conflict management education through
human rights and citizenship education. Ms. Obura noted that
global interest in including such elements in national curricula
is matched by a concern over the feasibility and quality of
assessment of the programmes focused on peace education
evaluation and assessment. Many researchers maintain that
current assessments of LTLT programmes lack scientific rigour,
fail to produce hard data and tangible or convincing findings.
While some consider quantitative assessment more appropriate
than qualitative assessment, there is some evidence that there
are advantages to using classical qualitative techniques. As an
example, Ms. Obura described the experience of assessing the
eight-country Peace Education Programme across Africa. She
noted that peace education continues to be the poor orphan of
the LTLT cluster in terms of both programme development and
assessment and is least present where it is most needed.
Peace education programmes are rarely included in school
curricula for several reasons: First, usable assessment
instruments for specific national contexts are not widely
available. Second, donor skepticism and lack of interest limit
or eliminate external assistance for peace education. Third,
the continuing debate on whether and how to test peace
education in public examinations continues to intrude on
periodic national assessment of the programmes themselves.
Fourth, accumulating evidence on the negative, active role
played by education systems in fuelling conflict and the
absence or marginalization of peace building programmes in
many post-conflict education systems can, at every level, be
considered the most critical curriculum gap.
In terms of assessment of school curricula, LTLT assessment
practices - particularly peace education assessment - is new
and presents challenges. This does not mean that progress
13
5. For further details
on the LTLT cluster and
continuum of indicators relating to LTLT
programmes, see Obura
(200?). Facing the Gathering Storm: Framework for
the Evaluation of Learning
to Live Together Programmes sponsored by
the Deutsche Gesellschaft
für Technische Zusammenarbeit (GTZ].
cannot be made, nor that work in this domain should be
dismissed orabandoned. Ms. Obura encouraged more research
in the area since this would be the only way forward.
Quality Frameworks with a Focus on the Last and Hardest
to Reach
Margaret Meagher, Senior Technical Advisor, Basic and Girls'
Education, CARE
It is possible
to develop
indicators
and effective
instruments
to measure
progress at the
learner level
dimensions
6. UNICEF (2002].Quality
education for all: from a
girl's point of view. New
York, US
?. Pigozzi, M.J. (2004)
What is the 'quality of
education' (A UNESCO perspective) page 39 in Ross,
K. and I. Genevois (2006)
Cross-National Studies of
the Quality of Education:
Planning their Design and
Managing their Impact,
IIEP/UNESCO Paris, France
Ms. Meagher began her presentation by discussing the utility
of leading quality education frameworks as they relate to
a focus on reaching the most marginalized children - the
last and hardest to reach on the pathway to achieving EFA
goals. This target population, as is well known, includes
significantly more girls than boys. Moreover, the barriers that
prevent them from learning are multiple, complex, and heavily
contextual. The presentation incorporated an overview of two
leading conceptual models, UNICEF (2002) and UNESCO
Framework for Quality Education (Pigozzi 2004), as effective
frameworks at the practitioner level across contexts toward
conceptualizing, measuring and maximizing improvement on
dimensions of educational quality.
The logic for focusing on the most challenging target populations
in terms of quality education, EFA and the FTI at this time is
simple: successful inclusion of these children across contexts
will benefit all other children within their contexts along the
way. While traditional (summative) approaches to reaching
the first 80-90% may have proven successful, at least on
the surface, in terms of access or parity through increasing
inputs at a broader scale, approaches to reaching the last 10%
are more analogous to an algebraic formula than they are to
simple addition.
Ms. Meagher ended her presentation by noting that the factors
that impede the attainment of quality education for the most
marginalized children are represented in dimensions of
educational quality of the UNESCO and UNICEF frameworks as
14
well as in the four pillars of learning (Delors 1996). Moreover,
early work at International Non-Governmental Organization
levels indicates that it is possible to develop indicators and
effective instruments to measure progress at the learner level.
Further, she recommended that on the basis of preliminary
findings, the frameworks be revised to bring the principle of
equality to the fore.
//'/////////////.
'/.//A
Multiple Expectations and one Practical Framework?
Kazuhiro Yoshida, Associate Professor, Hiroshima
University, Japan
Mr. Yoshida introduced quality as an issue that is receiving
more attention than ever before; its definitions are becoming
more comprehensive, and multiple dimensions have been
identified. When it comes to trying to agree on a common set
of indicators, the task becomes more problematic and more
impractical. Since we all have different expectations about the
quality of education, it becomes a question of whose views
matter most. Common views about learning outcomes usually
include knowledge, skills, attitudes and values. The Delors
Report makes references to learning to be, to do, to learn, to
live together, but the question of whether these can be seen
to correspond to common views on learning outcomes or not
has to be looked at more carefully. The rights-based approach
to quality education, according to the literature, holds that the
aims of education are defined in terms of the potential of each
child and the scope of the curriculum. According to this view,
the curriculum should be a preparatory process for promoting
and respecting human rights.
In the 1960s, economists looked into the role of education and
other human aspects of capital, and applied the notion of rate
of return to education. This theoretical background helped aid
agencies to justify investment in education. More recently, in
the late 1990s, economists used the term human capability.
Educational planners talk about inputs, process, outcomes
and context or environment, whereas education research puts
15
8. Delors, J.; Al Mufti, I.;
Amagi, I.; Carneiro, R.;
Chung, F; Geremek, B.; Gorham, W.; Kornhauser, A.;
Manley, M.; Padrón, Quero,
M.; Savane, M.-A.; Singh,
K.; Stavenhagen, R.; Myong
Won Suhr; Zhou Nanzhao.
(1996). Learning: The
Treasure Within: Report
to UNESCO of the International Commission on Education for the Twenty-first
Century. Paris, UNESCO.
(www.unesco.org/delors/)
9. See footnote 8, page 15
more emphasis on what is happening inside the classroom
and school environment.
Mr. Yoshida asked two questions: Can the conventional
knowledge of learning outcomes be assessed and if so how
and by whom? Can it be assessed objectively? He indicated
that knowledge could be assessed by tests, exams or other
methods and could be assessed by teachers or international
or national institutions. To a logical extent assessment of
knowledge and skills acquisition can thus be done objectively,
but when it comes to measuring attitudes and values, we do
not know how they can be assessed, whether they should be
assessed and if it can be done objectively or not.
If we establish
quality indicators,
whose view
should we take?
Mr. Yoshida concluded by offering some key issues for
discussion: on whose views are we basing our discussions and
how can they be assessed? Which quality indicators should
we use for EFA? If we establish quality indicators, whose view
should we take? What kind of efforts should we put into having
a shared vision translated into more measurable outcomes
in terms of education quality? Should we attempt to put
together all the views from stakeholders, teachers, students,
parents, educational planners and economists? Can these be
converged into a common agreed framework?
Panel 1
////////////////////.
D i s c u s s i o n Highlights
•-'/.:.
The three panelists, drawing upon their different
experiences, presented the points of view of different
actors - economists, planners, and other stakeholders who seek to improve the conditions for quality learning.
While diverse perspectives were noted on what quality
education entails and how it is best implemented, there
was more agreement than might have been expected
on the steps needed to ensure the provision of quality
16
education. The main issues arising from the discussion
included:
Defining quality: Within the array of visions,
frameworks and approaches to understanding,
measuring and assessing quality, it is possible to
identify several common elements. In principle,
such elements can be transformed into universally
accepted indicators, consistent with the needs of
different stakeholders at different education levels
and in different countries. The importance of a human
rights-based approach to quality was emphasized, as
was the need to add "learning for change" to the four
pillars of learning described in the Delors Report .
Measuring learning outcomes: The focus on cognitive
development and a small set of core indicators of
quality learning is both useful and important. Having
said that, several speakers emphasized the importance
of going beyond conventional cognitive measures
to consider non-cognitive learning outcomes, which
could be examined using a variety of measurement
strategies.
What to assess? The discussion distinguished
between what can feasibly be assessed versus what
should ideally be assessed. Given time pressure and
other constraints, there is merit in movi ngforward with
an initial, straightforward measurement strategy of
quality learning, while keeping in mind additional and
alternative measures to be examined in the future.
Overall, there is a need for deeper analytical work, in
orderto knowwhat can be reliably measured and how.
Quality learning is too complex to be summarized
in a few simple measures. While there is a need to
10. See footnote 8, page 15
17
be pragmatic in the short term, it is important to
continue to explore indicators and measures of the
other aspects of quality learning.
Social skills: Considerable expertise in measuring
cognitive skills has been accumulated; but much
less is known about measuring social skills such
as learning to live together. Opinions differed as
to the importance of these issues, in part due to a
dearth of accumulated evidence and well-designed
research. By and large, these skills go unnoticed by
agencies, researchers and donors in their accounts
of the outcomes of schooling. Several participants
maintained that such subjects and skills should be
examined periodically, so as to provide a basis for
improving measures and fostering further work in
the future. While it is difficult to define core indicators
of student values, attitudes and behaviours, solely
encouragingthe measurement of cognitive outcomes
can undermine the value and importance of broader
student learning experiences.
The learning environment: The impact of the
learning environment on learning outcomes is
critical, especially in post-conflict situations or under
conditions of economic instability. Attention should
be focused on elements in the school environment
that are amenable to policy intervention, so that
policy makers can use existing knowledge and tools
for producing positive effects, even with limited
resources. Education systems are often called upon
to support reconstruction efforts in conflict-affected
societies and to reduce the salience of past conflicts
and to prevent their recurrence.
18
Fast Track Initiative: While there are notable
constraints as to what can be accomplished in FTI
countries, it is important to develop clear action plans
to improve conditions for quality learning. At the
very least, specific activities can be recommended
for further research/action in order to clarify what
different individuals and organizations can do to
improve education quality.
Donors: Donors and other stakeholders need clear
benchmarks and indicators of quality learning to
support progress for the goal of quality education.
As such, it is important to focus on measurable
and translatable learning outcomes, which are
communicable to policy makers, teachers, parents,
and partners in the international community. Moving
in this direction should not preclude discussions and
explorations of other aspects of quality education.
Further research on all aspects of quality learning including attitudinal, behavioral, and values-related
dimensions - is needed.
19
Panel 2
Learning A s s e s s m e n t s
Hou can they become more e f f e c t i v e
t o o l s for improving quality learning?
The assessment of educational outcomes and the development
of generally acceptable measures of educational quality are
persistent challenges facing educators and policymakers
alike. International and regional assessments of student
curricular knowledge or skill acquisition have become widely
used in evaluating learning gaps, identifying accountability
issues and developing criteria for policy formation. They are
extensively mentioned in debates on educational reform.
Learning assessments have increased awareness of how
education systems differ in structure and performance, both
with respect to the level and equity distribution of outcomes.
Growing comparative research also suggests that policy
interventions can improve student achievement and reduce
learning gaps and inequalities.
While learning assessments are important and necessary
tools to monitor selected aspects of education systems, they
rarely provide a comprehensive picture of the conditions for
quality learning. Such limitations need to be understood and
acknowledged.
The three presenters on this panel addressed the challenge of
how to make assessments more effective tools for improving
the quality of learning locally, nationally and globally.
They discussed devising new methods and approaches to
assessment and linkingthese assessments more closely with
classroom-based teaching and learning processes. They also
considered how countries and the international community
can further develop the capacity to measure, assess and
monitor quality.
20
Panel 2
Presentations
Moderator: Kenneth Ross, Coordinator technical
project management, International Institute for
Educational Planning-(HEP)
////////////////////.•/'.•.•.'y.'
.'-'
'
•.
- -
Assessment of and assessment for learning: Supporting
teachers to improve education quality
Anil Kanjee, Executive Director, Human Sciences Research
Council
As a prelude to his presentation, Mr. Kanjee noted that
within the current context of monitoring and evaluating the
performance of education systems, the use of cognitive
measures of learning has become one of the most critical
indicators of education quality. This shift is evident in the
significant increase of large-scale assessment surveys (LSAS)
in the last decade. While recognizing that education quality is
more than just learning outcomes, current measures of noncognitive outcomes at the systems level are still difficult to
obtain.
Across many developing countries, especially in Sub-Saharan
Africa, the primary emphasis of the assessment systems has
been on LSAS (assessment of learning) with limited focus
on effective classroom assessment practices for improving
learning (assessment for learning). Even in countries that
have revised their classroom assessment practices, the focus
has primarily been on improving assessment for supporting
final year examinations, rather than assessment for learning.
Many developing countries have established robust systems
for conducting reliable and valid LSAS, often with the support
of regional or international organizations. For these countries,
now is the time for extending assessment of learning systems
to include assessment for learning.
Within this context, Mr. Kanjee stated that actors in education
need to consider the following key questions:
•
What are the practical assessments strategies that we
know will result in improving learning?
21
•
•
Assessment for
learning is one
of the most
significant and
cost- effective
investments that
any country can
make to ensure
that the learning
needs of children
are addressed.
•
How do we introduce these strategies to teachers, and
ensure their effective application?
How feasible and cost effective are these new
strategies?
What will the impact be on teaching practice, learner
performance and - most importantly - improvement in
learning?
In addressing these questions, the key challenge is the
implementation of assessment for improving learning
practices (that is, making learning goals explicit, effective
questioning, providing appropriate feedback, peer and selfassessment), even at its very basic level, to enhance the
learning experiences of all children.
Mr. Kanjee concluded that assessment for learning is one of
the most significant and cost-effective investments that any
country can make to ensure that the learning needs of children
are addressed. In developing effective systems for improving
the quality of education, the alignment and integration of
assessment for and assessment of learning is one of the
most pressing challenges that developing countries need to
address.
/////////////////////
'///// v
International Learning Assessments
Andreas Schleicher, Head of the Indicators and Analysis
Division, Organization for Economic Co-operation and
Development-(OECD)
In a globalised world, the yardsticks for public policy in
education are no longer national goals or standards alone,
but increasingly the performance of the most successful
education systems internationally. Mr. Schleicher posited
that international assessments can be powerful instruments
for educational research, policy and practice by allowing
education systems to look at themselves in the light of
intended, implemented and achieved policies elsewhere. They
22
can show what is possible in education, in terms of quality,
equity and efficiency in educational services, and they
can foster better understanding of how different education
systems address similar problems. Most importantly, by
providing an opportunity for policy makers and practitioners
to look beyond the experiences evident in their own systems
(and thus to reflect on some of the paradigms and beliefs
underlying these) they hold out the promise to facilitate
educational improvement. Designing and implementing
valid and reliable international assessments poses major
challenges, including defining the criteria for success in ways
that are both comparable across countries while remaining
meaningful at national levels, establishing comparable target
populations and carrying out the surveys under strictly
standardized conditions.
In recent years, international assessments have made significant
strides towards this end. Some contend that international
benchmarking encourages an undesirable process of degrading
cultural and educational diversity among institutions and
education systems, but the opposite can be argued as well: in the
dark, all institutions and education systems look the same and it
is comparative benchmarking that can shed light on differences
on which reform efforts can then capitalize.
Mr. Schleicher acknowledged that international assessments
have their pitfalls too: policy makers tend to use them
selectively, often in support of existing policies rather than as
an instrument to challenge them and to explore alternatives.
Moreover, highlighting specific features of educational
performance may detract attention from other features that
are equally important, thus potentially influencing individual,
institutional or systemic behaviour in ineffective or even
undesirable ways. This risk of undesirable consequences of
inadequately defined performance benchmarks is very real,
as teachers and policy makers are led to focus their work on
those issues which performance benchmarks value and put
into the spotlight of the public debate.
23
International
assessments
can show what
is possible
in education,
in terms of
quality, equity
and efficiency
in educational
services
In conclusion, Mr. Schleicher argued that while the development
of international assessments is fraught with difficulties and
their comparability remains open to challenges, cultural
differences among individuals, institutions and systems should
not suffice as a justification to reject their use, given that the
success of individuals and nations increasingly depends on
their global competitiveness. The task for governments will be
to ensure that their citizens, institutions and education systems
rise to this challenge and international benchmarks can provide
useful instruments to this end.
Do the results of national assessments contribute towards
improving public schools?
Maria Helena Castro, Secretaria, Sao Paulo State
Department of Education, Brazil
Ms. Castro's presentation began with a brief historical overview
of the process of evaluating educational quality in Brazil.
During the 1990s, the Ministry of Education established a
vast information system to manage statistics and evaluations
of the quality of public and private education. The aim of the
system was to provide information to support the educational
policies of Brazil, in accordance with the needs of its states
and municipalities. In 1995, no reliable source of information
existed as yet regardingthe educational system in Brazil. Data
on school census were constantly out of date and schools
lacked information tools. As a result, it was impossible to plan
the organization of a rapidly expanding educational system.
A consensus was reached on the need to establish reliable
statistical data as well as national assessments of the
educational system as a whole, from primary to higher
education. During the 1990s the following assessment tools
were created: the national elementary education evaluation
system (SAEB), the national secondary school examination
(ENEM) and finally, the higher education evaluation system.
In 2005, the Ministry of Education initiated the Brazil test and
in 200?, the basic education development index.
24
Brazil now possesses solid data providing a concrete outline
of its educational system. It can measure the impact of data
and variables that influence scholastic performance, such
as the educational level of parents, the socio-economic
level of teachers and directors, the structure of educational
establishments orthe pedagogical project of each school. More
than a decade after the establishment of national educational
assessment systems, notable progress has been made. A
new culture of evaluation now exists; data are disseminated
to civil society and the media and in particular, these data
have resulted in new and more efficient public policies in
education.
Ms. Castro concluded her presentation with the reflection that
even though all these measures are in place, the system still
has its limitations and that all possibilities of intervention
have not yet been explored. One of the major problems is the
lack of qualifications and preparations of Brazilian teachers;
this makes it difficult for them to be able to integrate the
assessment results with pedagogical needs.
Panel 2
////////////////////•'//y,y//.y//
Discussion Highlights
•/.'•/.
••
The presentations in this panel revealed the diversity
of approaches to assessing, monitoring and measuring
quality learning. These approaches vary in terms of
conception, complexity, methodological rigour, cost,
accessibility, frequency, learning scales, reliability and
relevance. The debate focused on the following issues:
Type of assessments: International, regional and
national assessments can be either content-based
(examining what students actually learn in relation
to the curriculum) or competency-based (assessing
relevant
skills
and
competencies
students
should have acquired by the end of compulsory
25
education). Both perspectives are important and
complementary.
•
There is a continuing need for curriculum-based
assessments since most stakeholders, in
particular ministers of education, understand that
such assessments reflect the major objectives
and purposes of the educational system as well
as specific curricular policies. Ideally, these
learning outcomes are also applicable and useful
for the roles children eventually enter as adults.
•
Knowledge is limited about the full range of
skills that young people acquire, and even less
about how they utilize such skills in adult life.
Curriculum policies presume that certain kinds of
learningare particularly important, but evidence
for this assertion is weak. There is a growing
mismatch between what schools provide and
what society demands. Existing skills perceived
to be in demand (or not in demand) influence
current curriculum development, but the ability
to determine with any certainty the types of
skills that will be in demand in the future is
limited, especially in developing countries.
Purpose and consequences: All stakeholders
should raise questions about the purpose and
consequences of learning assessments. How
can assessment results contribute to improved
learning? Does strengthening the links between
assessment results and teacher incentives or
system policies improve the quality of learning? To
what extent do assessments stimulate real change
in the classroom?
26
Relevance and usefulness: Students constantly
need to relate what they have learned to their own
lives so that their educational experiences are
meaningful. Assessments should provide a basis
for examining whether this occurs and, if so, how.
Furthermore, assessments can be used to set policy
targets and to determine what (and how much)
progress has occurred.
Impact: To date, evidence about how assessment
results are actually used by different educational
stakeholders, and their impact on classroom
learning, is limited. Among the reasons that
assessments appear to have a limited impact on
education systems:
•
Many assessment reports are often not read and,
even when they are, the rich body of information
compiled is under-utilized.
•
The teaching-learning community has few real
opportunities to constructively use the results
of learning assessments.
•
The public dissemination of assessment results
often lacks sufficient transparency. More
often than not, representatives of the media
present assessments in problematic ways, oversimplifying results.
•
While the direct impact of assessments on
classroom life may be muted, the indirect
impact through policy change can be significant.
The challenge is how to distribute assessment
information, how to analyze it, and how to
manage its impact.
27
Incentives: Presenters highlighted the positive, and
at times problematic, ways that learning outcomes
are linked to teachers and teaching practices. They
cautioned against the broad use of teacher incentives
to improve learning outcomes for several reasons:
first, teachers who do not receive an incentive
may feel punished or unappreciated; second, some
assessments may focus on relatively unimportant
aspects of quality learning, thus promoting an
inappropriate link to teacher incentives; and third,
incentives sometimes pressure teachers to ignore
the least able and hardest to reach students. Public
recognition of teacher performance and student
achievement is a preferable substitute for incentives,
since it helps to create a culture for positive and
lasting change.
Equity and equality: The distinction between
equity and equality is important - in most contexts
advancing one facilitates progress in the other.
Equality is a key human rights focus with important
implications for education. Equitable policies and
practices in education are meant to lead to social
and economic equality. Improving equity and
equality should be viewed as dimensions of quality
education, though these linkages need further
exploration.
Capacity-building: There is an urgent and
recognized need to strengthen the capacity of
FTI countries to undertake national and local
assessment exercises and to make meaningful
use of the assessment results. There are serious
reliability and methodological challenges in
carrying out formal assessments in many African
28
and other FTI contexts. Countries and communities
differ enormously in their familiarity with testing, as
well as in the skills needed to carry out the tasks
involved in a responsible, equitable and effective
manner.
29
Panel
3
I n d i c a t o r s of Quality Learning
What is s t i l l m i s s i n g in the EFA-FTI
I n d i c a t i v e Framework?
Should EFA p a r t n e r s adopt c o m m o n core
indicators?
While a common global definition of "quality education"
remains elusive, many researchers and assessment
specialists believe that discrete, observable indicators of core
quality dimensions can capture its nature and provide a basis
for assessing progress. One aim of this panel was to explore
whether it is feasible and advisable to adopt a common set
of core indicators of quality learning, which can be compared
across learning contexts and educational systems. The
advantages and disadvantages of developing such indicators,
andtheirusefulnessforEFAstakeholdersand EFA-FTI partners
as a basis for measuring progress and improving the quality
of teaching and learning, was the central challenge addressed
by the presenters.
Panel 3
Presentations
Moderator: Andreas Schleicher, Head of the
Indicators and Analysis Division, Organization
for Economic Co-operation and Development
[OECD]
Indicators of Educational Achievement
HansWagemaker, Executive Director, International
Association for the Evaluation of Educational Achievement
(IEA)
The measurement of educational outcomes and the
development of generally acceptable measures of educational
quality remain some of the more persistent challenges
facing educators and policy makers alike. Mr. Wagemaker's
presentation outlined lEA's approach to developing
indicators of educational quality and provided the theoretical
30
underpinning for the approach taken. A number of examples
of the types of indicators that have been developed were
illustrated and in addition to describing how these might be
used to inform policy, they were used to highlight why the set
of indicators included in the indicative FTI framework are not
sufficient as measures of educational quality.
It was argued that despite an appeal to improve educational
quality in FTI countries, there is little in the FTI framework
document that goes beyond the measurement of educational
inputs.
In outlining some of the indicators that have been developed
by IEA, an argument was made that the international
perspective is important if FTI countries are to make
progress toward achieving the goals related to educational
improvement and poverty reduction. To achieve this objective
it was argued that outcome indicators need to be developed
which are internationally comparable and at the same time
locally relevant, that process is as important as product, that
the investment in the development of indicators needs to be
strategic rather than tactical and that indicators should be
dynamic, capable of reflecting change towards desired end
states as well as being capable of being changed to reflect
changing societal and economic demands.
Finally, to address concerns related to local relevance
particularly for FTI countries, Mr. Wagemaker gave a brief
introduction to lEA's new initiative in the area of reading (prePIRLS).
//////////////////////////¿'///M///////
Measuring and Analyzing Education Quality at National/
Regional Level with Systems of Indicators
Mark Agranovitch - Federal Institute for Education Level,
Russian Federation
Mr. Agranovitch's presentation described a research study
devoted to developing a system of indicators for measuring
31
Outcome
indicators need
to be developed
which are
internationally
comparable and
at the same time
locally relevant.
education quality at the national/regional levels in the
Russian Federation. Research results have been more or
less successful, as about one fifth of the Russian regions
have implemented the system developed in the practice
of education analysis and policy decision-making. At the
beginning of the research, there was an agreement on the
following definition:
The quality of education at national or sub-national level
includes:
1.
Education results:
a. Learning results (level of educational standards
performance)
b. Social skills for future life
2.
Meeting external requirements:
a. Officially established norms and standards
b. Parent-student needs
The system of indicators for measuring educational quality,
in addition to groups of indicators appropriated for the above
definition, should consist of contextual indicators characterizing
the socio-economic, information and cultural environment.
The analysis of indicators is based on comparable research of
changes during a specific period and/or comparison of different
regions and countries. This is why it is crucial for processes
for developing indicators to use standard statistical data that
are collected on a permanent basis. Otherwise, results will be
incomparable, unreliable or extremely expensive.
Mr. Agranovitch ended his presentation by stating that using
comparative indicators, one could identify the weak points of
the educational system and suggest practical measures for
educational quality improvement.
32
Information Needs and the EFA-FTI Indicative Framework
Cesar Guadalupe, Head of the Literacy Assessment and
Monitoring Programme, UNESCO Institute for Statistics
Regarding the question "What is missing from the EFAFTI Indicative Framework? Mr. Guadalupe stated that the
framework lacked a proper measure of completion, evidence
on learning outcomes and a linkage to a broader view of
quality. If we were to link support to performance in this way,
standard policy actions would not help in this regard.
In relation to the purpose of the Indicative Framework, the
presentation summarized criticisms about its exclusive
focus on primary completion, leaving aside other EFA goals.
Mr. Guadalupe questioned the extent to which the Indicative
Framework was actually indicative, adaptable to local
circumstances, and limited in scope. Since its main purpose is
to link support to performance, its "indicative" character may
disappear.
Concerning the structure of the Indicative Framework, it
combines instrumental measures and goals that Mr. Guadalupe
believed was questionable. In relation to the goals, the
framework relies on what he considered a false indicator that
is not correlated with completion. This is not only technically
flawed, he said, but could create substantive problems while
obfuscating actual exclusion-related situations.
Concerning the question on whether EFA partners should
adopt a common set of core indicators, Mr. Guadalupe stated
that it would depend on the purpose of having them. If they
were for monitoring progress, there are already monitoring
mechanisms at global, regional and national level. If they were
for linkingsupport to performance then we should: (i) consider
a broader set of goals, (ii) use robust indicators for the goals,
(iii) be very careful about suggesting standardized "recipes",
and (iv) be informed by current ideas on humanitarian action
principles.
33
Panel 3
Discussion Highlights
Participants underscored the broad agreement among
EFA-FTI partners and the learning assessment community
that simple yet robust measures of quality learning are
needed. The FTI Indicative Framework would be missing
basic measures of student learning. In fact, many view
this as a very significant deficit in the current system
of endorsing country plans and monitoring progress.
Several examples of common core indicators, as well as
options for assessing basic skills, were presented and
discussed.
Literacy and numeracy: As a first step, literacy
and numeracy skills should be assessed towards
the end of the primary school cycle. Turning
this objective into reality is neither easy nor
quick, especially if the assessment results are
to be compared across countries, cultures and
languages. Testing students using alternative
measurement strategies should be done with care,
particularly in some FTI countries where previous
assessment experiences are limited in number
and scope.
•
Defining literacy: Children should be able to read
and write by the end of primary school as this
reflects a core indicator of literacy. However,
several issues demand attention; for example,
should countries be encouraged to adopt a
universal definition of literacy, or can varying
definitions be employed? Should universal
reading standards be promoted, or should
countries develop their own benchmarks?
34
•
Age versus grade: Should literacy and numeracy
proficiency be linked to a child's maturation
according to age, or rather to grade-specific
standards? In addition, since in many FTI
countries large numbers of children do not attend
school or begin school at non-normative ages,
should assessments of literacy and numeracy
skills be school-based, or should they also be
household-based to reflect learning in informal
and/or non-formal settings? Answers to these
questions have implications for design and
measurement as well as for the specification of
core indicators.
•
Primary education: The duration of primary
education can vary from 4 to 8 years; so
too can the number of actual (or intended)
hours of instruction. Given the multiplicity of
primary education systems in the world, clear
definitions of early grades and end of primary
education are necessary.
Policy: Participants emphasized the need for
learning policies to be less prescriptive and more
enabling. As such, evidence should be compiled
concerning the extent to which policies are
changing and also bringing about real change in
the school and classroom. For example, do policies
enable teachers to better understand their students'
backgrounds, address their needs, and receive
pedagogical support? Do existing incentives push
teachers to focus solely on learning levels and not
on learning conditions? To what extent do policies
create an enabling framework, which improves the
learning environment?
35
Indicators: Measures should be sufficiently
transparent so that teachers can understand and
apply the results. Care should be taken to translate
complex information into a more user-friendly
knowledge base. Different views on the need to
have comparable core indicators were voiced.
Core indicators should certainly be of value to
stakeholders both within and outside of schools,
regardless of a student's grade level or mother
tongue. The measurement of quality should also
be applicable across multiple contexts (i.e., formal
schooling, informal settings, alternative schools),
and contribute to broad-based policy discussions
and formation. Ultimately, these indicators should
feed into the E FA and Millennium Development Goal
monitoring systems.
Cost of assessment: There was general agreement
that the cost of conducting assessments should
not be a determining issue, especially in light of
what countries can gain from having an educated
population. Indeed, given that education is the
second or third largest area of public expenditure
in most developing countries, it is critical that
countries have a clear understanding of what
children actually learn by the end of the primary
cycle.
36
General Synthesis and Reflection
on the Panel Presentations by the
General Rapporteur
Aaron Benavot, Senior Policy Analyst,
EFA Global Monitoring Report, UNESCO
The panel presentations, as well as the draft Desk
Review, amply demonstrated the range and multiplicity
of contemporary visions, frameworks and approaches
to understanding, measuring and assessing quality.
By drawing upon a wealth of expertise and experience,
speakers underscored, once again, the centrality of
quality learning and how it can be advanced through
careful reflection, measurement and evaluation.
While a succinct global definition of quality learning
may be difficult (if not impossible) to develop, some
common elements did emerge. To begin, there is
considerable consensus that quality learning is
multidimensional in nature. At the very minimum, it
involves two crucial dimensions: first, a set of basic
conditions that enable learning to take place—for
example, a safe and healthy learning environment,
trained and motivated teachers who are present in
the classroom, an organized programme of learning/
curricular contents, sufficient instructional time,
adequate learning materials; and second, a variety of
learning outcomes, including a student's mastery of
school-transmitted knowledge and basic skills like
literacy and numeracy, but also additional outcomes
such as advanced analytical skills, socially and
culturally relevant attitudes/dispositions, values and
patterns of behaviour, which are malleable to change.
37
Quality learning was also viewed as a multi-level
phenomenon: it embodies different things at different
levels: for the individual, in classrooms and schools,
across regions or sub-regions, and in the aggregate
for a country (or educational system) as a whole. For
some colleagues, the individual and system levels
represented the key levels for measuring, monitoring
and assessing quality; for others emphasis was placed
on intermediate levels like classrooms, schools and
regions.
The presentations, and the discussions they sparked,
brought out the complexity of existing assessment
approaches and designs. Some might view this
overwhelming complexity as a sign of weakness, of
a field in disarray. But the opposite could be true as
well. Professionals involved in assessing, monitoring
and measuring quality necessarily address multiple
end-users and stakeholders, many of whom have
different purposes and interests in mind. Thus, it
is both necessary and valuable that the purveyors
of quality assessment carry conceptual and
methodological toolkits that are rich in variety and
contents.
Beyond this, speakers emphasized the importance
of a pragmatic and flexible approach to quality
assessment. They reminded us of the deep challenges
and harsh conditions affecting the provision of
education in many EFA-FTI countries. Examples in
Africa described by Anil Kanjee and Anna Obura, and
in Central and South America by Maria Helena Castro
and Margarete Meagher, provided a much-needed
reality check. One clear point emerged: when learning
assessments are conducted in developing contexts,
38
issues of suitability, feasibility, reliability and validity
demand special attention. Countries and communities
differ enormously in their familiarity with testing, and
the skills needed to carry out the tasks involved in the
manner requested.
During the discussion several important questions
emerged: what do children actually gain - socially
and collectively - from having attended school,
especially from their educational encounters with
teachers and other caring adults? Beyond their
cognitive development, do students learn important
social skills like the rights and responsibilities of
community membership; the ways of dealing with
individuals who differ from those of your immediate
family; or the health skills and family competencies
that impact the quality of one's life? The inculcation of
these social and life skills find curricular expression
in subjects like peace education, social studies or
history, but achievements in these areas typically
go unnoticed by agencies, researchers and donors
in their accounts of the outcomes of schooling. While
rarely measured orassessed, they are of considerable
relevance to young people's lives, as well as to notions
of civil-mindedness, cultural tolerance and political
participation.
Finally, the presentations and discussions continually
affirmed the central importance of feedback loops: in
other words, of linking assessments to actual teaching
practices, to classroom life, to teacher motivation and
eventually to improvements in learningoutcomes. How
assessments are utilized (or not), by whom, and for
what purposes are issues that deserve serious study
in the future.
39
Working Groups
On the second day of the seminar, three parallel working
groups were organized to allow for more in-depth deliberations
on the major themes of the seminar and to note key issues,
challenges and conclusions. The three working groups were
divided along the same lines as the previous day's panels for
further discussion of the following themes:
1. Quality education forali
2. Learning assessment
3. Indicators of quality learning
Each group was also asked to focus additional discussion on:
1. The feasibility of proposing common core indicators of
quality: advantages, problems
2. Follow-up strategies on assessing and improving quality
learning
Finally, the groups were requested to propose concrete
actions on the above.
Compiled Group Recommendations
•
•
11. See footnote 8, page 15
It is feasible to have core indicators of quality that
countries should feel free to adopt. These indicators of
quality learning could be measured differently in different
countries depending on their mechanisms and standards.
There is a need for an operational definition of quality
education which should include (among other things)
the four pillars of learning11 and mechanisms on how to
assess them. This vision of quality should recognize and
further develop the multi-dimensional nature of quality
learning assessment. However, it is also necessary to
be pragmatic by prioritizing simple, cost-effective and
feasible steps toward more complex, longer term goals.
40
•
Shared understanding is needed among EFA partners
on the differentiation between indicators, standards and
measures of quality learning.
•
Educational systems should be encouraged to establish
minimal standards for: 1) learning outcomes in literacy,
numeracy and essential life/social skills; and 2] the
enabling conditions for achieving these outcomes.
•
Literacy and numeracy would be the starting point, with
emphasis on literacy, at the end of the primary cycle but
there should also be earlier milestones.
•
A common operational definition of primary education
should be reached, includingduration, hours of instruction,
etc.
•
Countries should have mechanisms for disseminating
information on assessment of learning outcomes
and share actively with the public the results of those
assessments.
•
The equity dimension should be part of these indicators,
so that not only a certain percentage of the population
is concerned. There is a need to clarify the relationship
between equity and quality, and to advise countries on
how both can be included in educational assessments.
•
The relationship between equity and equality needs to be
clarified. Equality relates to inclusion (participation], justice
(distribution of materials and resources], and integration
(having children from different social backgrounds], while
equity is often expressed in terms of results. Strategies for
inclusion need to look at the equality of opportunities to
learn.
•
Greater collaboration among organizations is needed in
order to facilitate countries' participation in numerous
assessment exercises, to help them use the results
to benefit the different stakeholders (policy-makers,
teachers, parents, etc], and to alleviate the high cost of
41
human and financial resources. It is also important to
establish a mechanism for quality control of the different
assessment systems around the world.
•
Since not all children are in schools, and not all schools
can be called "formal schooling" there is a need to use
household level surveys as well as school level measures
to track quality, and, of course, there is a need to continue
to track access to more formal schooling.
•
Concerning assessment of learning outcomes, a bottom-up
approach is recommended. Over half of the developing
countries in the world have carried out national assessments.
There is a way to look from the bottom up and begin to identify
commonalities, the conception of literacy, the way literacy is
defined and being assessed, and the kind of measures that
have been used. This could lead to gradually converging on
common definitions and a common scale overtime.
•
UNESCO should encourage EFA-FTI countries to develop
their own core indicators. Where necessary, UNESCO
in association with other institutions should help build
capacity for national assessment systems.
•
Caution is urged on the "cost of assessment" issue.
Sometimes assessment appears to be expensive in an
absolute sense, or even on a per-learner sense, but it is
very inexpensive relative to its utility and power, and as a
percentage of total expenditure.
Conclusion and Follow-up Strategies
Based on the compiled recommendations drafted by the
three working groups, the following concrete actions were
proposed to follow-up the work initiated at the Learning Counts
Seminar:
1. Set up a Working Group for the purpose of putting together
a set of indicators that will respond to learning outcomes
42
and learning environments. The Working Group would also
address the broader dimensions of the quality issue by
focusing on indicators of the conditions for learning.
2. Revisit the FTI Indicative Framework
a. Some indicators like Net Enrolment Ratio (NER), Gross
Enrolment Ratio (GER) or survival to grade 5 ratios could
be misleading
b. Need for literacy and numeracy indicators at the end of
primary school as well as in the early grades
c. Improve rigour of measurement of some indicators
such as completion
3. Convene an international workshop of experts on enabling
learning conditions and measuring non-cognitive skills.
4. Organize a follow-up forum for sharing information on
educational assessments and related issues.
5. Agree to be pragmatic in order to act quickly and efectively.
6. UNESCO should encourage FTI countries to come up with
theirown core indicators (via regional off ices and partners,
coordinated by UNESCO HO) and help build capacity for
assessment systems.
7. Support research on the best ways to measure learning
outcomes, the learning environment and non-cognitive
skills.
The meeting concluded with the adoption of a technical
Working Group as its main outcome. This Working Group will
carry out the work proposed by the participants in the final
discussion and summarized in the Meeting Statement in next
page. It will operate under the leadership of UNESCO in order
to 1) reach consensus on common core indicators of quality
and 2) address the broader dimensions of the quality issue,
including identification of enabling conditions for quality
learning. (See Annex 1)
43
This Working Group includes members of the key leading
international organizations involved in assessment of
education quality. Participation in the Working Group
offers them a unique opportunity to pool their considerable
expertise and to work collaboratively with institutes, regional
associations and other partners to develop concrete proposals
for assessing and accelerating progress toward the quality
goal. The objectives of the Working Group are expected to be
achieved within 2 years, beginning in January 2009.
UNESCO will ensure that cooperation and coordination among
organisations working on the area of educational assessment
takes place in order to facilitate the efforts of the countries to
monitor and improve the quality of their education systems.
Meeting Statement
Summary of general recommendations
12. See footnote 8, page 15
•
The underlying principle for action on improving quality
learning is clearly articulated in the quality goal itself —
namely, the need for "recognized and measurable learning
outcomes [to be] achieved by all, especially in literacy,
numeracy and essential life skills."
•
Equality and inclusion — the "for all" aspect of the quality
goal— should be integral elements in the formulation of
visions, indicators and assessment strategies. Quality
education for all should take into account the four pillars
of learning12as well as the need to address life skills,
includingcitizenship and conflict management, in a rapidly
changing world. Responses to the needs of excluded and
neglected learners - namely, low income groups, rural
populations and girls- must be prioritised as a matter of
urgency.
•
There is a need for a vision that recognizes the
multidimensional, multipurpose nature of quality learning
assessment that couples systems level measures with
continuous assessment at the learner level. However, it
44
is also necessary to be pragmatic by prioritizing simple,
cost-effective and feasible steps with a commitment to
more complex, longer term goals which better reflect the
diversity of learning needs and outcomes.
•
As an initial priority, ongoing work on quality assessment
and improvement should focus on schoolingatthe primary
level or the basic education cycle. This would include
learning outcomes nurtured by diverse forms of schooling
as long as they possess a clear structure, course of study
and level progression.
•
Learning steps, stages and outcomes should be measured
early and often, as well at the end of certain learning cycles,
with initial focus on assessment of literacy and numeracy
as the foundational skills. Assessment of social and life
skills should not be neglected, but the question of how
and when requires more focused attention based on the
growing body of research and sharing of good practices in
this area.
•
To effectively improve the quality of learning, more
emphasis needs to be placed on "closing the feedback
loop"—in other words, ensuring that results of both
formative and summative assessments are integrated
into actual teaching and learning practices as well as
decision-making processes at the system level.
•
Global indicators of learning outcomes and the conditions
of learning are both possible and desirable as catalysts
for improving quality education worldwide, but specific
targets for enhancing learning achievement, intermediary
benchmarks and the means to measure progress toward
them are the rightful domain of national, local and
community systems.
Statement on common core indicators of quality
•
It is indeed possible to reach consensus on common core
indicators of quality defined in terms of learning outcomes
45
for literacy and numeracy, together with evidence-based
recommendations on the most conducive and achievable
processes and conditions of learning in a variety of
learning contexts.
•
Education systems should be encouraged to establish
their own minimal standards for: 1) learning outcomes
in literacy, numeracy and essential life/social skills and
2) the enabling conditions for achieving these outcomes.
Statement on follow-up strategies
•
A Working Group was established to explore and discuss
points of convergence among multiple approaches to
conceptualizing, assessing and improvingquality learning
at both the learner and systems levels. Specifically,
the central task of the Working Group would be to reach
consensus on an operational definition of common core
indicators of quality, with specific recommendations for
improving the EFA-FTI Indicative Framework and Appraisal
Guidelines.
The Working Group would also address the broader dimensions
of the quality issue by focusing on indicators of the conditions
for learning, including the acquisition of knowledge, values
and skills in both the cognitive and social domains, as well as
actual teacher practices, classroom effectiveness and system
support.
46
Annexes
An n e x 1 : Draft Working Group Mandate
A Working Group was established to explore and discuss
points of convergence among multiple approaches to
conceptualizing, assessing and improving quality learning at
both the learner and systems level. Specifically, the central
task of the Working Group would be to reach consensus on an
operational definition of common core indicators of quality,
with specific recommendations for improving the EFA-FTI
Indicative Framework.
The following broad mandates and specific issues are intended
to shape, stimulate and enlighten the deliberations of the
proposed Working Group. In some cases, specific studies will
be required by the Group to feed the discussions and guide its
conclusions.
a.
First mandate: reaching consensus on a set of
common core indicators of quality. Major issues
include:
i.
Definition of primary education. Measurable learning
outcomes in literacy, numeracy and essential life skills
should be achieved by the end of the primary level of an
education system. Therefore, a clear definition of what
primary education is and what learners are expected
to know and do upon completion is required.13
¡i.
Agreement on what is going to be measured and
what the priority should be. Discussions of a global
definition of quality education are never-ending, but
it may be possible to reach a basic consensus on
common elements emerging from the literature review
contained in the background document prepared for
the Seminar.14 In response to the clear and repeated
call for a pragmatic approach, it is necessary to plan
47
13. The ISCED definition
is suggested as a starting
point: "Programmes at level
1 are normally designed
on a unit or project basis
to give students a sound
basic education in reading,
writing and mathematics
along with an elementary
understanding of other
subjects such as history,
geography, natural science,
social science, art and
music. In some cases,
religious instruction is
featured."
14. See Part l o f UNESCO
[2009] Desk Review
Learning Counts: An
overview of Approaches to
Understanding, Assessing
and Improving the Quality
of Learning for All. Paris,
France.
for action at all levels. However, the first step should
be to establish frameworks, tools and indicators for
measuring and improving literacy and numeracy
during the primary cycle of education.
iii.
Recommendation on when and how often to
measure. Assessment of progress in literacy and
numeracy during the early stages of the primary
cycle is essential to improving the quality of learning
processes and practices and to achieving end-ofcycle learning outcomes. Therefore, a clear indication
of measuring points (or benchmarks) needs to be
established.
¡v.
Recommendation on approaches to different forms of
summative and formative assessment for systemlevel monitoring and measuring for learner-level
action. It might be advisable to combine two sorts of
measures: those that are mainly intended to monitor
progress at system level; and those that can be seen
more as tools for community, school and classroomlevel action.
v.
Consideration of whether and how these previous
elements (i-iv) can be connected to existing initiatives
at international, regional and national levels.
b.
Second mandate: addressing the broader
dimensions of the quality issue. The four pillars
of learning (Delors15 are considered by many as
a key element for defining quality education.
"Holistic" (and rights-based) approaches (for
example: UNICEF 16 2002, Pigozzi17 2004] can be
seen as responding directly to the four pillars
(Delors 18 1996) and can, therefore, be taken as a
good starting point to promote consensus on the
following:
i.
Consideration of how learning outcome measures
are linked to a broader view of quality education.
The challenge here is how to encourage a more
15and 18 See footnote 8,
page 15
IB. UNICEF (2002).
Quality education forali:
from a girl's point of view.
New York, US
1?. Pigozzi, M.J. [2004]
What is the'quality of
education' (A UNESCO perspective) page 39 in Ross,
K. and I. Genevois (2006)
Cross-National Studies of
the Quality of Education:
Planning Their Design and
Managing Their Impact,
HEP/UNESCO Paris, France
48
constructive use of assessments as tools to improve
quality at all levels.
ii.
Identification of enablingconditions for quality learning.
This should be based on existing empirical evidence
and any other elements that are considered valuable
to human and social development. Having a core set
of factors that are understood to enhance learning
can lead to the creation of tools for assessment and
prioritize teaching-learning practices for classroom
effectiveness as well as promote desired values,
social and life skills.
iii.
The need to address equality and inclusion. These
two interrelated issues, which have been identified as
critical to any effort to improve the quality of learning,
can be approached at least at three levels: ( 1 ) analysis
of the distribution of skills among the population
vis-à-vis disparities based on gender, ethnicity
or socioeconomic status, [2] policies intended to
promote inclusion, and (3) actual practices at school
level (for instance, admission policies, availability
of reading materials in local languages, inclusive
learning environments, etc].
49
Annex 2 Agenda
DAY l Tuesday 28 October
09h00 - 9h30 Registration
09h30 - lOhOO Opening of the Meeting
Chair: Ana Luiza Machado
•
Welcome address by Nicholas Burnett, Assistant
Director-General for Education
•
Orientation of the meeting and adoption of the
agenda, chairpersons, rapporteurs
•
Brief introduction of participants
10h00-10h30
•
Background and update on the EFA-FTI Quality
Learning Desk Review by Lene Buchert and Jean
Bernard
10h30 - 10h45 Coffee/tea
10h45-12h30
•
Panel 1: Quality education for all—visions,
frameworks and practices
Where and how do they converge?
Moderator:
Robin Horn
Presenters:
Anna Obura
Margaret Meagher
Kazuhiro Yoshida
Co-reporter:
Ichiro Miyazawa
50
12h30-13h45 Lunch
13h45 - 15h45
•
Panel 2: Learning assessments
How can they become more effective tools
for improving quality learning?
Moderator:
Kenneth Ross
Presenters:
Maria Helena Gimaraes Castro
Anil Kanjee
Andreas Schleicher
Co-reporter:
Marta Encinas Martín
15h45 -16h00 Coffee/tea
16h00 - l?h45
•
Panel 3: Indicators of quality learning
What is still missing in the EFA-FTI
Indicative Framework?
Should EFA partners adopt common core
indicators?
Moderator:
Andreas Schleicher
Presenters:
Hans Wagemaker
MarkAgranovitch
César Guadalupe
Co-reporter:
Daniel Taccari
I?h45-18h30
Moderator:
•
Thomas Gaie
Special Session Presentations
18h45 Welcome cocktail - Miollis Bar
51
DAY 2 Wednesday 29 October
09h30-10h00
•
Summary of Day 1: Aaron Benavot, General Rapporteur
•
Working group orientation: Marta Encinas-Martín
Working Group
Leader
Reporter
1. Quality education for all
Marguerite Clarke
Lilia Torranzos
2. Learning assessment
Hector Valdes
Demus Makuwa
3. Indicators of quality learning j Luis Crouch
Natalie Schwendy
lOhOO-llhOO
•
Working group discussions
I l h 0 0 - l l h 2 0 Coffee/tea
N o t e on the working
Each of the three
groups
parallel
groups will be asked to:
Ilh20-13h00
•
Working group discussions (cont.)
13h00- 14h30 Lunch
14h30- 16hl0
Chair: Hameed Hakeem
•
Group presentations
lGhlO- 16h30 Coffee/tea
16h30- -18h00
Chair: Lene Buchert
•
<"» discuss one of the Day I
themes in more
<"< focus additional
depth
discussion
on:
- the feasibility of
proposing common core
indicators of quality
- follow-up strategies on
assessing and improving
quality learning
- propose concrete
recommendations for
action on the above
items
General discussion:
— Revision of UNESCO preliminary draft desk review
— Group recommendations
Wrap-up and announcements
52
DAY 3 Thursday 30 October
9h30-10h45
Chair: Steven Obeegadoo
•
General discussion: improvement of EFA-FTI
Appraisal Guidelines and Indicative Framework
1 0 h 4 5 - l l h 3 0 Coffee/tea
Ilh00-13h00
Chair: Ana Luiza Machado
•
General Rapporteur's preliminary summary:
Aaron Benavot
•
Follow-up strategies
•
Final comments from participants
•
Closing remarks
53
Annex 3 Participants List
Participants List
•
MarkAGRANOVITCH
Head of the Centre for Monitoring &
Statistics of Education
Federal Institute for Ed Devel
(Russian Federation)
Email: magrantgfiro.ru
• Koli BANIK
Education Planner and Task Team
Liaison EFA-FTI Secretariat
•
Marguerite CLARKE
Senior Education Specialist
World Bank
Email: [email protected]
•
Luis CROUCH
Senior Economist and Research VP
RTI International
Email: [email protected]
Email: kbamk§Kdücatioi-itesttidck.oiy
•
•
Aaron BENAVOT
Senior Policy Analyst
EFA Global Monitoring Report
UNESCO
Email: [email protected]
•
Sandra BERTOLI
Education Specialist
USAID
Thomas GAIE
Head of National Office
WAEC (Liberia)
Email: [email protected]
• César GUADALUPE
Head of the Literacy Assessment and
Monitoring Programme (LAMP)
UIS
Email: [email protected]
Email: [email protected]
•
Mark BRAY
Director
HEP-UNESCO
• Abdul HAKEEM
Education Adviser and Coordinator
"APPEAL"
Email: m.bray § iiep.unesco.org
UNESCO Bangkok
Email: [email protected]
•
•
Maria Helena CASTRO
Secretaria
Sào Paulo State Department of
Education (Brazil)
Email: [email protected]
Robin HORN
Education Sector Manager and
(Acting) Education Sector Director
World Bank
Email: [email protected]
54
•
Anil KANJEE
• Kenneth ROSS
Coordinator Technical Project
Executive Director
Human Sciences Research Council
Email: [email protected]
•
Demus MAKUWA
Acting Director of SACMEO
HEP-UNESCO
Email: [email protected]
•
Margaret MEAGHER
Senior Technical Advisor - Basic &
Girls' Education
CARE
Email: [email protected]
• Valeria SAKHAROVA
Deputy Director
Center for International Cooperation
in Education Development (Russian
Federation)
Email: [email protected]
•
Andreas SCHLEICHER
Head of the Indicators and Analysis
Division
•
Ichiro MIYAZAWA
Programme Specialist
UNESCO Islamabad
Email: ¡[email protected]
•
Steven OBEEGADOO
Director, EFA Country Relations
UNESCO
Email: [email protected]
•
Management
HEP-UNESCO
Email: [email protected]
OECD
Email: [email protected]
•
Natalie SCHWENOY
Education Advisor
GTZ
Email: [email protected]
• Daniel TACCARI
UIS Regional Advisor for LAC
OREALC - UNESCO
Anna OBURA
Email: [email protected]
International Consultant
Kenya
Email: [email protected]
• Abdel Moneim OSMAN
Director
UNESCO Beirut
Email: [email protected]
55
• Lilia TORRANZOS
Education Specialist
Oraganization of
Iberoamerican states (OEI)
Email: lilia.toranzosiggmail.com
• Hector VALDES
Coordinator of LLECE
UNESCO/OREALC
Email: [email protected]
• IgorVALDMAN
Director
Center for International Cooperation
in Education Development
(Russian Federation)
Email: iavaldmantggmail.com
• HansWAGEMAKER
Executive Director
International Association for the
Evaluation of Educational
Achievement
Email: hanswagemaker@
compuserve.com
Organizing Committee
• Ana Luiza MACHADO
Deputy Assistant Director-General for
Educational Programme Management
UNESCO
Email: [email protected]
• LeneBUCHERT
Chief, Section of Inclusion and Quality
Learning Enhancement
UNESCO
Email: [email protected]
• Jean BERNARD
Senior Programme Specialist
UNESCO
Email: [email protected]
• Marta ENCINAS-MARTÍN
Education Programme Specialist
office of the Deputy Assistant
Director-General for Education
Programme Management
UNESCO
Email: [email protected]
• Kazuhiro YOSHIDA
Associate Professor
Hiroshima University
Japan
Email: [email protected]
56