Main achievement outcomes
continued. . . .



Performance on mathematics and
reading (minor domains) in PISA 2006,
including performance by gender
Performance across domains in PISA
2006
Trends in performance in Ireland and
across OECD countries (2000-2006
1
PISA Mathematics




Major domain in PISA 2003 (85 items)
Minor domain in PISA 2006 (48 items)
Overall scale in 2006; overall
proficiency levels, but no subscales
No new items
2
3
4
Sample Question 15



Quantity – Closed constructed response
Full credit – both maximum (137 zeds) and
minimum (80 zeds) correct
Partial credit –maximum or minimum correct
PISA Item Difficulty
Item Stats
% OECD
% Irel
Scale Score:
464 (PC)
496 (FC)
Full correct
67
69
Partial corr.
11
8
Proficiency
Level:
2 (PC)
3 (FC)
Incorrect
18
21
Missing
5
2
5
Sample Question 16


Quantity – Traditional multiple choice
Correct answer: D (12)
PISA Item Difficulty
Scale Score
570
Proficiency Level
4
Item
Statistics
Correct
Incorrect
Missing
%
OECD
%
Ireland
46
30
50.0
67
4
3
6
Sample Question 17

Quantity – Short
constructed
response
PISA Item Difficulty
Scale Score
554
Proficiency Level
4
Item
Statistics
%
OECD
%
Ireland
Correct
50
50
Incorrect
45
48
Missing
5
2
7
Performance on PISA
Mathematics (2006)




Mean score for Ireland – 501.5
Not significantly different from OECD
average of 497.7
Rank: 16th among OECD countries
(range: 12th-17th)
Rank: 22nd among 57 participation
countries (range: 17th-23rd)
8
Chinese Taipei
Finland
Hong Kong-Ch.
Korea
Netherlands
Switzerland
Canada
Macao-China
Liechtenstein
Japan
New Zealand
Belgium
Australia
Estonia
Denmark
Czech Republic
Iceland
Austria
Slovenia
Germany
MATHEMATICS
IRL
Mean
549
548
548
548
531
530
SE
4.10
2.30
2.67
3.76
2.59
3.15
527
525
525
523
522
520
520
515
513
1.97
1.30
4.21
3.34
2.39
2.95
2.24
2.75
2.62
▲
▲
▲
▲
▲
▲
▲
▲
▲
510
506
506
505
504
3.55
1.81
3.74
1.04
3.87
0
0
0
0
0
▲
▲
▲
▲
▲
▲
Sweden
Ireland
OECD
France
United Kingdom
Poland
Slovak Republic
Hungary
Luxembourg
Norway
Lithuania
Latvia
Spain
Azerbaijan
Russian Fed.
United States
Croatia
Portugal
Italy
Greece
Mean
502
502
498
496
495
495
SE
2.41
2.79
0.54
3.17
2.14
2.44
IRL
0
492
491
490
490
486
486
480
476
476
2.82
2.89
1.07
2.64
2.93
3.03
2.33
2.26
3.87
▼
▼
▼
▼
▼
▼
▼
▼
▼
474
467
466
462
459
4.02
2.37
3.07
2.28
2.97
▼
▼
▼
▼
▼
0
0
0
0
Mathematics: Performance
at the 95th Percentile (2006)
720
707
700
692
678
680
Scale Score
694
672
660
640
645
647
OECD
Average
N. Ireland
634
620
600
Chinese
Taipei
Finland
Hong Kong
China
Korea
Netherlands
Ireland
10
Mathematics: Performance
at the 5th Percentile (2006)
420
411
400
392
Scale Score
386
380
382
373
366
360
346
341
340
320
300
Chinese
Taipei
Finland
Hong Kong
China
Korea
Netherlands
Ireland
OECD
Average
N. Ireland
11
Spread of Achievement in
Mathematics




Can be estimated by finding the
difference between scores at the 95th
and 5th percentile
Difference in Ireland: 268 points.
OECD average: 299
Only Finland (267) and Wales (270)
have differences close to Ireland’s.
12
Mathematics – Performance by
Proficiency Levels – Ireland and OECD
Average
35
30
29
Percent of Students
24
24
25
22
21
19
20
Ireland
OECD
14
15
12
10
10
5
9
8
4
3
2
0
Below
Level 1
Level 1
Level 2
Level 3
Level 4
Level 5
Level 6
13
Gender Differences in PISA
Mathematics (2006) – Ireland and
OECD Average
510
Scale Score
505
507
503
500
496
495
Ireland
492
OECD Average
490
485
480
Males
Females
14
Gender Differences in Ireland–
Mathematics Proficiency Levels
(2006)
35
30
30
27
Percent of Students
26
25
23
22
19
20
Males
Females
15
13
11
10
10
5
7
4 4
2
1
0
Below Level 1 Level 2 Level 3 Level 4 Level 5 Level 6
Level 1
15
Explaining Gender
Differences in Mathematics





Structure of test (e.g., impact of Space & Shape
items)
Item type (multiple-choice vs. constructed
response)
Nature of the PISA mathematics tasks
Contrast with Junior Certificate examination
where females achieve more A and B grades in
mathematics at higher and ordinary levels
Presentation by Seán Close this afternoon to
examine gender differences in detail.
16
Trends in Achievement
(2000-2006)




Complexity in establishing trends over time
as domains shift from major to minor
Mix of items (e.g., effect on reading score of
many reading vs. many science items)
Reconfiguration of items into new clusters to
meet framework specifications
Characteristics of the particular set of linking
items used (country interaction)
17
Trends in Mathematics
(2003-2006)




Major to minor domain from 2003 to 2006
48 ‘common items’
Neither Ireland’s overall mean scores, nor
the OECD country average scores, changed
between the two years.
No significant differences in Ireland at the
5th or 95th percentiles, though the score of
students in Ireland at the 95th was 7 points
lower in 2006.
18
Trends in Mathematics (2003-2006) –
Mean Scores, Ireland and OECD Average
510
505
502
500
2006
2003
2006
490
2003
Mean Scale Score
498
470
450
Ireland
OECD Average
19
Trends in Mathematics
(2003-2006)



Students in Mexico (+20) and Greece
(+14) registered significantly higher mean
scores in 2006.
Students in France (-15), Iceland (-10)
and Belgium (-9) had significantly lower
mean scores.
Students in Denmark and The Netherlands
scoring at the 95th percentile had
significantly lower scores in 2006.
20
PISA Reading Literacy





Major domain in PISA 2000 (141 items)
Minor domain in PISA 2003/2006 (28
items)
Overall scale in 2006; overall
proficiency levels, but no subscales
Items drawn from 3 reading processes:
retrieve (25%), interpret (50%), and
reflect/evaluate (25%)
No new items
21
Sample Reading Literacy Item –
Q. 1
PISA Item Difficulty
Scale Score
558
Proficiency Level
4
Item Stats
% OECD
% Irel
Correct
48
58
Incorrect
36
36
Missing
16
6
22
Sample Reading Literacy Item –
Q. 2



List two examples. . . Interpretation of text.
Answer – two of 3 possible answers
High level of difficulty
PISA Item Difficulty
Scale Score
669
Proficiency Level
5
Item Stats
% OECD % Irel
Correct
7
6
Incorrect
52
65
Missing
41
29
23
Performance on PISA
Reading Literacy (2006)




Mean score for Ireland – 517.3
Significantly higher than OECD
average of 491.8
Rank: 5th among OECD countries
(range: 4-6)
6th among 56 participating countries
(range: 5-8) (US missing for reading)
24
READING
IRL
Mean
SE
Korea
Finland
Hong Kong-Ch.
Canada
556
3.81
▲
547
2.15
▲
536
527
2.42
2.44
▲
▲
New Zealand
Ireland
Australia
Liechtenstein
521
2.99
0
517
3.54
513
2.06
510
3.91
0
0
508
2.79
▼
507
507
3.44
2.92
▼
▼
501
501
3.04
2.93
▼
▼
499
3.06
▼
498
496
3.65
3.38
▼
▼
495
495
2.26
4.41
▼
▼
495
494
3.18
0.99
▼
▼
Poland
Sweden
Netherlands
Belgium
Estonia
Switzerland
Japan
Chinese Taipei
United Kingdom
Germany
Denmark
Slovenia
Mean
SE
IRL
Macao-China
OECD
Austria
France
492
492
1.10
0.60
▼
490
488
4.08
4.06
▼
▼
Iceland
Norway
Czech Rep.
Hungary
484
484
1.95
3.18
▼
▼
483
4.18
▼
482
3.28
▼
Latvia
Luxembourg
Croatia
Portugal
Lithuania
Italy
Slovak Rep.
Spain
Greece
Turkey
Chile
Russian Fed.
480
3.73
▼
479
477
1.28
2.81
▼
▼
472
470
3.56
2.98
▼
▼
469
2.43
▼
466
461
3.06
2.23
▼
▼
460
447
4.04
4.21
▼
▼
442
440
4.99
4.32
▼
▼
▼
Reading: Performance at the
95th Percentile (2006)
700
Score at 95th Percentile
688
683
680
675
674
661
660
660
659
642
640
620
600
Korea
Finland
HongKong
China
Canada
New
Zealand
Ireland
N.
Ireland
OECD
Average
26
Reading Literacy: Performance
at the 5th Percentile (2006)
430
410
410
399
Score at 5th Percentile
390
390
370
358
357
350
339
330
317
311
310
290
270
250
Korea
Finland
HongKong
China
Canada
New
Zealand
Ireland
OECD
Average
N. Ireland
27
Spread of Achievement in
Reading Literacy




Based on difference between scores at
the 95th and 5th percentile
Difference in Ireland: 303 points.
OECD average: 324
Korea (289), Finland (265) and
Denmark (293) have smaller gaps,
indicating a more homogeneous
spread of achievement in those
countries.
28
Reading Literacy – Performance by
Proficiency Levels (2006) – Ireland and
OECD Average
35
30
Percent of Students
30
28
25
25
21
23
21
20
Ireland
15
12
9
9
10
5
OECD
13
7
3
0
Below
Level 1
Level 1
Level 2
Level 3
Level 4
Level 5
29
Gender Differences and
Reading Literacy (2006)
540
534
530
520
511
510
Mean Score
500
500
Ireland
OECD Average
490
480
473
470
460
450
440
Males
Females
30
Gender Differences in Ireland
on Reading Proficiency Levels
35
Percent of Students
30
25
20
Males
Females
15
10
5
0
Below
Level 1
Level 1
Level 2
Level 3
Level 4
Level 5
31
Explanation of Gender
Differences in Reading
Literacy




Emphasis on text type (continuous vs. noncontinuous)
Item type (multiple-choice vs. constructed
response)
Reading process (retrieve, interpret,
reflect/evaluate)
Presentation by Brian Murphy this afternoon
will examine differences in more detail
32
Trends in Reading Literacy (2000-2006) –
Mean Scores, Ireland and OECD Average
530
527
520
517
516
Mean Score
510
Ireland
500
500
OECD Average
494
492
490
480
470
2000
2003
2006
Year
33
Trends in Reading Literacy (2000-2006)
– Mean Scores



Korea’s mean score increased by 31 points
between 2000 and 2006
Poland’s mean score increased by 29 points
between the same years.
Significant declines in mean reading scores
in Australia (-15), France (-17), Greece
(-14), Iceland (-22), Italy (-19), Japan
(-24), Norway (-21), and Spain (-32).
34
Trends in Reading Literacy (2000-2006)
– Performance at Key Benchmarks



Significant (22 point) drop in performance of
students in Ireland at 95th percentile in 2003
(relative to 2000), though some of the
difference (14 points) was made up in 2006
No significant differences at 5th percentile.
More comprehensive trend data for reading
literacy in 2009, when it becomes a major
domain again - the first time a major
domain is repeated in PISA.
35
Trends in Science
Performance (2000-2006)



Difficulty in developing trend data going
from minor domain Decline in performance
on link items in Ireland between 2003 and
2006, but difference not significant.
Significant increases in Mexico (+22.7) and
Greece (20.5) and a drop in France (-16)
Ireland performed at about the same level
on all three assessments to date –
comfortably above the OECD average in
2000, and just above it in 2003 and 2006.
36
Science
Finland
Hong Kong-Ch.
Canada
Chinese Taipei
Estonia
Japan
New Zealand
Australia
Netherlands
Liechtenstein
Korea
Slovenia
Germany
United Kingdom
Czech Republic
Switzerland
Macao-China
Austria
Belgium
Ireland
Hungary
Sweden
OECD MEAN
Reading
Korea
Finland
Hong Kong-Ch.
Canada
New Zealand
Ireland
Australia
Liechtenstein
Poland
Sweden
Netherlands
Belgium
Estonia
Switzerland
Japan
Chinese Taipei
United
Kingdom
Germany
Denmark
Slovenia
Macao-China
OECD MEAN
Austria
Mathematics
Chinese Taipei
Finland
Hong Kong-Ch.
Korea
Netherlands
Switzerland
Canada
Macao-China
Liechtenstein
Japan
New Zealand
Belgium
Australia
Estonia
Denmark
Czech Republic
Iceland
Austria
Slovenia
Germany
Sweden
Ireland
OECD MEAN
37
Consistency in Performance
across Domains

Most countries tend to perform at a
similar level on each domain
– The top 4 spots in science, reading and
maths are shared between 5 countries
– The bottom 4 spots are shared between 6
countries

Ireland shows a more mixed
performance, with relatively lower
performance in mathematics
38