1
1
Organisation for Economic Cooperation and Development (OECD)
Quality and equity in
educational outcomes
Seeing school systems through the prism of PISA
Campbell What Works seminar
9 November 2006
Dr. Karin Zimmer
OECD / Directorate for Education
2
2
3
3
In the dark…
…all students, schools and education systems look the same…
But with a little light….
4
4
In the dark…
…all students, schools and education systems look the same…
But with a little light….
…important differences become apparent….
Portugal
Mexico
Turkey
1
Spain
1980's
Italy
9
Greece
90
Korea
Ireland
Poland
Belgium
Australia
France
1970's
Iceland
Luxembourg
Hungary
Netherlands
Finland
United Kingdom
Switzerland
New Zealand
1960's
Japan
Austria
Sweden
Slovak Republic
Canada
Denmark
10
Norway
100
Czech Republic
20
Germany
80
United States
5
5
Approx. by % of persons with upper secondary qualfications in age groups 55-64, 45-55, 45-44 und 25-34 years
Baseline qualifications
A world of change
1990's
1
70
60
50
40
30
24
0
6
6
Overview
1. The PISA approach

Objectives and methods underlying OECD’s
Programme for International Student
Assessment (PISA)
2. Where we are today - and where we can be

What PISA shows students in different
countries can do with what they have learned
3. How we can get there

Some policy levers that emerge from
international comparisons
7
The PISA approach
Measuring the quality of learning outcomes
PISA country participation
Key features of PISA 2003
8

Information collected
 volume of the tests
– 3½ hours of mathematics assessment, less than half in
multiple-choice format
– 1 hour for each of reading, science and problem solving

each student
– 2 hours on paper-and-pencil tasks (subset of all questions)
– ½ hour for questionnaire on background,
learning environment, engagement and motivation

school principals
– questionnaire (school demography, learning environment
quality)

Coverage


OECD countries participating from PISA 2000
countries participating from PISA from 2003
PISA coversOECD
roughly
nine tens of the world economy
OECD partner countries participating from PISA 2000
Representative
samples
of between
and 50,000
OECD partner
countries participating
from PISA 3,500
2003
students OECD partner countries participating from PISA 2006
9
9
Deciding whom to assess...
grade-based sample
OR
age-based sample
For PISA, the OECD countries chose the latter, selecting
15-year-olds in school as the population.
10
10
Deciding what to assess...
looking back at what students were
expected to have learned
…or…
looking ahead to what they can do with
what they have learned.
For PISA, the OECD countries chose the latter.
11
11
Three broad categories of
key competencies
Using “tools”
interactively to
engage with the world
To analyse, compare, contrast, ande.g.
evaluate
Using language, symbols and texts
Toinformation
think imaginatively
Interacting with
Capitalising on the potential
PISA concept
of literacy
of technologies
Acting
Interacting
in
Accessing,
managing,
integrating
autonomously
diverse groups
and evaluating
written information
e.g.
e.g.
in order
to develop
andwithin
potential,
Acting
the bigger picture
Relating
wellones
to knowledge
others
and to participate in, and contribute to, society
Co-operating, working in teams Learning strategies
Taking responsibility and
Managing and
resolving situations
conflicts
To apply knowledge
in real-life
understanding rights and limits
To communicate thoughts and ideas effectively
12
12
Using “tools”
interactively to
engage with the world
To analyse, compare, contrast, ande.g.
evaluate
Using language, symbols and texts
Toinformation
think imaginatively
Interacting with
Capitalising on the potential
Reading
literacy
of technologies
Acting
Interacting in
Using, diverse
interpreting
autonomously
groups and reflecting
e.g.
on written
material
e.g.
Acting within the bigger picture
Relating well to others
Co-operating, working in Forming
teams and conducting life plans
Taking responsibility and
Managing and
resolving situations
conflicts
To apply knowledge
in real-life
understanding rights and limits
To communicate thoughts and ideas effectively
13
13
Using “tools”
interactively to
engage with the world
To analyse, compare, contrast, ande.g.
evaluate
Using language, symbols and texts
Toinformation
think imaginatively
Interacting with
Capitalising on the potential
Scientific
literacy
of technologies
Acting
Interacting in
Using scientific knowledge, identifying scientific
autonomously
diverse groups
questions, and drawinge.g.evidence-based conclusions
to
e.g.
Acting the
within
the bigger
picture
understand and
make well
decisions
about
natural
world
Relating
to others
Co-operating, working in Forming
teams and conducting life plans
Taking responsibility and
Managing and
resolving situations
conflicts
To apply knowledge
in real-life
understanding rights and limits
To communicate thoughts and ideas effectively
14
14
Using “tools”
interactively to
engage with the world
To analyse, compare, contrast, ande.g.
evaluate
Using language, symbols and texts
Toinformation
think imaginatively
Interacting with
Capitalising on the potential
Mathematical
literacy
of technologies
Acting
Interacting
in
Emphasis is on
mathematical
knowledge put into
autonomously
diverse groups
functional use in a multitude
of different e.g.
situations
e.g.
Acting within the
bigger picture
well to
others
in varied,Relating
reflective
and
insight-based
ways
Co-operating, working in Forming
teams and conducting life plans
Taking responsibility and
Managing and
resolving situations
conflicts
To apply knowledge
in real-life
understanding rights and limits
To communicate thoughts and ideas effectively
15
Where we are - and where we can be
What PISA shows students can do
Examples of the best performing countries
16
16
High mathematics performance
Hong Kong-China
Liechtenstein
Macao-China
Iceland
Ireland
Poland
Latvia
Russian Federation
Italy
Average performance
Finland
Korea of 15-year-olds in
540
Netherlands
Japan mathematics
Canada
Belgium
Switzerland
Australia
New Zealand
520
Czech Republic
Denmark
France
Sweden
Austria
Germany
500
Slovak Republic
Norway
Luxembourg
Hungary
Spain
United States
480
Portugal
460
Low mathematicsGreece
performance
17
17
Mathematical literacy in PISA
The real world
The mathematical World
Making the problem amenable
to mathematical treatment
A model of reality
Understanding,
structuring and
simplifying the
situation
A mathematical
model
Using relevant
mathematical
tools to solve
the problem
A real situation
Validating
the results
Mathematical
results
Real results
Interpreting
the mathematical results
18
18
High mathematics performance
Hong Kong-China
High average performance
Large socio-economic disparities
Liechtenstein
Macao-China
Iceland
Strong socioeconomic impact on
student performance
Ireland
Poland
Latvia
Russian Federation
Italy
Low average performance
Large socio-economic disparities
Average performance
Finland High average performance
Korea of 15-year-olds in
540
Netherlands High social equity
Japan mathematics
Canada
Belgium
Switzerland
Australia
New Zealand
520
Czech Republic
Denmark
France
Sweden
Austria
Germany
500
Slovak Republic
Norway
Luxembourg
Hungary
Socially equitable
distribution of
learning opportunities
Spain
United States
480
Portugal
460
Low average performance
High social equity
Low mathematicsGreece
performance
19
19
High mathematics performance
High average performance
540
Large socio-economic disparities Netherlands
Liechtenstein
Hong Kong-China
Durchschnittliche
High average performance
Finland
Schülerleistungen
im
Korea
High social equity
Bereich Mathematik
Japan
Canada
Belgium
Switzerland
Australia
New
Zealand
520
Czech Republic
France
Denmark
Sweden
Austria
Strong socioIreland
economic impact on Germany
500
Slovak Republic
student performance
Hungary
Poland
Luxembourg
United States
480
Portugal
Low average performance
Large socio-economic disparities
460
Norway
Iceland
Socially equitable
distribution of
learning opportunities
Spain
Latvia
Russian Federation
Italy
Low average performance
High social equity
Low mathematics performance
Greece
School performance and schools’ socioeconomic background - Germany
20
20
800
Student performance and student SES
within schools
Student performance
School performance and school SES
School proportional to size
500
200
-3
Disadvantage
-2
-1
0
1
PISA Index of social background
2
Advantage
3
School performance and schools’ socioeconomic background - Denmark
21
21
700
OECD
OECD
Student performance
OECD
Student performance and student SES
within schools
School performance and school SES
Student performance and student SES
School proportional to size
500
300
-3
Disadvantage
-2
-1
0
1
PISA Index of social background
2
Advantage
3
School performance and schools’ socioeconomic background - Finland
22
22
800
Student performance and student SES
Student performance
Student performance and student SES
within schools
School performance and school SES
School proportional to size
500
200
-3
Disadvantage
-2
-1
0
1
PISA Index of social background
2
Advantage
3
Iceland
Finland
Norway
Sweden
Poland
Denmark
Ireland
Canada
Spain
New Zealand
Australia
United States
Mexico
Portugal
Luxembourg
Switzerland
Greece
Slovak Republic
Korea
Czech Republic
Netherlands
Austria
Germany
Italy
Belgium
Japan
Hungary
Turkey
23
23
Is it all innate ability?
Variation in student performance
140
120
100
80
60
40
20
0
OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 4.1a, p.383.
In other
large performance
Is countries,
it all innate
ability?
differences
amongperformance
schools persist
Variation
in student
in mathematics
Austria,
Belgium, Germany,
Hungary,
Italy,
In In
some
countries,
parents
can rely
onJapan,
high
Netherlandsstandards
and Turkey, most
of the
performance
andthe
consistent
across
schools
Variation
of
24
24
100

80
among schools lies between schools…
performancevariation
within

In Canada, Denmark, Finland, Iceland and Sweden
schools
… and in some of these countries, most notably those
60
average student performance is high…
that are highly stratified, a large part of that variation
… isand
largely unrelated
to the individual
schools
in which
explained
by socio-economic
inequalities
in learning
students are enrolled.
opportunities
40
20
0
-20
-40
Variation of
performance between
schools
-60
Iceland
Finland
Norway
Sweden
Poland
Denmark
Ireland
Canada
Spain
New Zealand
Australia
United States
Mexico
Portugal
Luxembourg
Switzerland
Greece
Slovak Republic
Korea
Czech Republic
Netherlands
Austria
Germany
Italy
Belgium
Japan
Hungary
Turkey
-80
OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 4.1a, p.383.
25
How can we get there?
Levers for policy that emerge from
international comparisons…
…and what countries have done with the findings
26
26
Money matters but other things do too
600
Performance in mathematics
550
Korea
Czech republic
Ireland
500
Slovak republic
Poland
Finland
Japan
Netherlands
Belgium
Canada
Australia
Iceland
Sweden
Germany
France
Hungary
Spain
Portugal
Switzerland
Denmark
Austria
Norway
United States
Italy
450
Spending
per student is positively associated
with average student performance…
… but
400 not a guarantee for high outcomes

Greece
Mexico
Australia, Belgium,
Canada, the Czech Republic,
Finland, Japan, Korea and the Netherlands do well R2 = 0.28
350in terms of “value for money”…
0
10,000
20,000
30,000
40,000
50,000
60,000
70,000
80,000
90,000
100,000
… while some of the big spenders perform
Cumulative expenditure (US$)
below-average

27
27
 Sympathy
doesn’t
raise standards –
High ambitions
and clear standards
aspiration does

PISA suggests that students and schools
perform better in a climate characterised
by high expectations and the readiness to
invest effort, the enjoyment of learning, a
strong disciplinary climate, and good
teacher-student relations
– Among these aspects, students’ perception of
teacher-student relations and classroom
Access to best practice
disciplinary climate display the strongest
and quality professional
relationships
development
28
28
Challenge and support
Strong support
Poor performance
Strong performance
Improvements idiosyncratic
Systemic improvement
Low
challenge
High
challenge
Poor performance
Conflict
Stagnation
Demoralisation
Weak support
29
29
Governance of the school system


In many of the best performing countries
Monitoring
and equity-related goals
 School-based decision-making is combined with
Standard
setting
and
equity-related
goals

Diverging
views
how
evaluation
and
assessment
devices to ensure a fair distribution ofcan and

Key should
objectives:
be used
substantive
educational
opportunities
– Raise
educational
aspirations,
establish
– Some
see them
primarily as
tools to reveal best practices and
over educational
reference
transparency
The
provision
of standards
curricula
at and
identify
shared
problems
inobjectives,
order and
to encourage
teachers
framework
teachersand develop more supportive and productive
schoolsfor
to improve
national/subnational
levels is combined with
learning
environments

Approaches
range
from definition of broad
advanced evaluation
and support systems



– Others
extend
their
purpose toof
support
contestability of
educational
goals
up to
formulation
concise
–public
Thatservices
are implemented
by professional
or market-mechanisms
in theagencies
allocation of
performance
expectations
resources
 Process-oriented
and/or
Some
countries
go beyond assessments
establishing educational
– e.g. by making comparative results of schools publicly available to
standards
as
mere yardsticks
and use
centralised
final examinations
are following
complimented
facilitate parental
choice or by having funds
students
performance
benchmarks
that students
at
with individual
and feed-back

Differences
in typereports
of performance
benchmarks being used
particular age or grade levels should reach
mechanisms
student
learning progress
and
reported foron
the
various stakeholders
involved,
Instruments
including parents, teachers and schools
– Minimum standards, targets defining excellence,
normative performance benchmarks
30
30
High ambitions
Devolved
responsibility,
the school as the
centre of action
Accountability
and intervention in
inverse proportion to
success
Access to best practice
and quality professional
development
31
31
High mathematics performance
High average performance
540
Large socio-economic disparities Netherlands
Liechtenstein
Hong Kong-China
Durchschnittliche
High average performance
Finland
Schülerleistungen
im
Korea
High social equity
Bereich Mathematik
Japan
Canada
Belgium
Switzerland
Australia
New
Zealand
520
Czech Republic
Iceland
Denmark
France
Sweden
Austria
Socially equitable
Ireland
Strong socioeconomic impact on Germany
Slovak Republic
student performance
Hungary
500
Poland
Luxembourg
United States
480
Portugal
Low average performance
Large socio-economic disparities
460
Norway
distribution of
learning opportunities
Spain
Latvia
Russian Federation
Italy
Low average performance
High social equity
Low mathematics performance
Greece
High mathematics performance
32
32
540
Netherlands
Liechtenstein
Hong Kong-China
Durchschnittliche
Finland
Schülerleistungen
im
Korea
Bereich Mathematik
Japan
Canada
Belgium
Switzerland
Australia
New
Zealand
520
Czech Republic
Iceland
Denmark
France
Sweden
Austria
Socially equitable
Ireland
Strong socioeconomic impact on Germany
Slovak Republic
student performance
Hungary
500
Poland
Luxembourg
United States
480
School with responsibility for
deciding which courses are offered
High degree of autonomy
Portugal
Norway
distribution of
learning opportunities
Spain
Latvia
Russian Federation
Italy
460
Low degree of autonomyLow mathematics performance
Greece
High mathematics performance
33
33
540
Netherlands
Liechtenstein
Hong Kong-China
Durchschnittliche
Finland
Schülerleistungen
im
Korea
Bereich Mathematik
Japan
Canada
Belgium
Switzerland
Australia
New
Zealand
520
Czech Republic
Iceland
Denmark
France
Sweden
Austria
Socially equitable
Ireland
Strong socioeconomic impact on Germany
Slovak Republic
student performance
Hungary
500
Poland
Luxembourg
United States
480
Early selection and
institutional differentiation
Portugal
Norway
Spain
Latvia
Russian Federation
Italy
460
High degree of stratification
Low degree of stratification
distribution of
learning opportunities
Low mathematics performance
Greece
34
34
Strong ambitions
Integrated
educational
opportunities
Accountability
Devolved
responsibility,
the school as the
centre of action
Individualised
learning
Access to best practice
and quality professional
development
35
35
High ambitions
Integrated
educational
opportunities
Devolved
responsibility,
the school as the
centre of action
Accountability
Individualised
and intervention in
learning
inverse proportion to
success
Access to best practice
and quality professional
development
36
36



37
37
Creating a knowledge-rich profession in which schools and
teachers have the authority to act, the necessary knowledge
to do so wisely, and access to effective support systems
The future of education
systems needs to be
“knowledge rich”
Informed professional
judgement, the teacher as
a “knowledge worker”
Informed
prescription
National
prescription
Professional
judgement
Uninformed
prescription, teachers
implement curricula
Uninformed professional
judgement, teachers
working in isolation
The tradition of
education systems has
been “knowledge poor”
38
38

www.pisa.oecd.org
– All national and international publications
– The complete micro-level database

email: [email protected]
Further information