Use of indicators for
educational outcome by the
Dutch Inspectorate. Pitfalls
and new challenges in a
network society
SICI conference ‘Assessment of
Educational Results Prague 2830 march 2017
Bruno Vreeburg
Dutch Inspectorate of Education
Four questions to start
1. Is the use of indicators for educational outcome for accountability
of schools by an inspectorate always necessary? When not?
2. Do you deal with undesirable strategic behavior in schools when
the inspectorate uses the assessment of educational results for
accountability? How to prevent?
3. We think norms and strong benchmarks for the minimal level of
schools are necessary. But how about the better schools? Do the
inspectorates need a differentiated use of educational results for
accountability? How to design?
4. Can schools give their own calculation of educational results for
accountability? When it is valid? How to compare?
5. …..
6. …..
Contents of the presentation
1.
2.
3.
4.
5.
6.
7.
3
Educational information in the Netherlands
Indicators for schools about educational outcome
Assessment information for accountability and strategic behaviour
Information at the system level
Discussion about outcome indicators
Development of outcome indicators
Final remarks
THE EDUCATIONAL
SYSTEM IN THE
NETHERLANDS AND THE
ASSESSMENT
INFORMATION
4
Data
DATA
Compulsory
Standardized Final
Test 12 year
- Primary Education
- Special Needs
Education
Intermediary
test
Core subjects
Diagnostics
Final Examination
- Standardized
Central tests (all
Subjects)
- Schooltests (all
Subjects)
No central
examination in
senior vocational
education and
higher education
Senior Vocational
Education
PSE
Primary Education
Secondary Education
diploma
Higher Education
Content
and
remarks
55
Test about core
subjects
Different test
providers
Schooladvice for
secondary education
Not for
use by
inspectorate
Not
diploma
Examination scores
for all the students
5 tracks
diploma
Only information
about educational
succes: percentage
diplomas, drop-outs,
switch, levels and
study duration
INDICATORS FOR
SCHOOLS ABOUT
EDUCATIONAL OUTCOME
6
Three essential conditions for using educational
assessment for accountability
1. Indicators of educational outcome at a school level must
cover the full width of the results.
2. The indicators for educational outcomes must take into
account the relevant circumstances of the school.
- Gross and net effects: Correction for student characteristics and
value added.
3. The indicators have to be transparant, verifiable (you can
recalculate them) and you can influence them.
De Bruin (2001), Hargreaves and Braun (2014)
7
Functions of the educational system
Three functions:
• Qualification: diploma, ticket for labour market
• Socialization: citizenship, participation in the society
• Allocation: sufficient people in the relevant jobs. Talent
development for every child.
We can make indicators for the system and for schools.
8
Outcome indicators: school level and system level
School
System
Qualification
Learning performances (tests), diplomas, attained level,
no drop out
x
(gross and
net)
x
(gross)
Distribution of student performance in different groups
x
x
x
x
(male/female, ethnicity, social-economic background)
Socialization
Citizenship, social cohesion. Hard to observe (only
qualitative).
Allocation
Enough qualifications, labour shortage, distribution of
educational outcome by groups.
x
Placement in a succeeding trajectory
x
Conditional
Efficiency, no retardation
x
(gross and
net)
9
X
(gross)
Examples of indicators
Primary Education
• Final test results assessed over 3 years.
• The norm depends on the percentage of children with poorly
educated parents.
Secondary education
• Final test results.
• Attained level.
• Yearly succes and no retardation.
• Correction for poorly educated parents, sometimes entrance level.
• Balanced scoremodel.
10
Use of the outcome indicators
• Schools use it for quality management
• Risk detection
• Judgment
• Diagnostics (there are a lot of diagnostic indicators).
11
Challenges now
Not only the information and the outcome indicators of the
inspectorate are interesting.
More important:
• School also have information about outcome. Empowerment of
schools with instruments to analyse their outcome themselves.
• Encouraging schools to set their own ambitious targets.
12
ASSESSMENT INFORMATION
FOR ACCOUNTABILITY AND
STRATEGIC BEHAVIOUR
13
Strategic behaviour
Desired behaviour:
• More effort. Higher results.
Undesirable behaviour (schools and students)
• Teaching only for the test.
• Minimalisation of effort.
• No admission of weak students. Schools may exclude pupils whom
they fear they will bring down the mean test score.
• Slide between encouraging, a little help and cheating during test.
• Schools may use test questions as practice material.
Lazear (2006), Koretz, Hamilton in: Brennan (2006), Fischbacher, Föllmi-Heusi (2013), Borghans,
Schils (2015), Muskens, Tholen (2015)
https://www.onderwijsinspectie.nl/documenten/publicaties/2015/11/10/onbedoelde-effecteninspectietoezicht-onderzocht
14
Strategic behaviour: Distribution primary
education
Near the ceasura you see a
slight deviation from normality
15
Questionaire example. Admission of a student:
Suppose a new student is registered for the
school and you can expect he will be a weak
student. Do take into account the consequences
for the judgement of outcome by the inspectorate
when deciding to allow or not allow this student?
On my own
On other
school
schools
Not at all
79
20
A little
16.5
30
Strong
4
22
Don’t know/No answer
0.5
29
16
Mean scores for the school test (SE) and the
standardised central test (CE)
6.90
After 2011: No
compensation
6.80
6.70
Until 2011: Schooltest can
compensate standardized
central test
6.60
6.50
SE
6.40
CE
6.30
6.20
6.10
6.00
2006
17
2007
2008
2009
2010
2011
2012
2013
2014
2015
Conclusions from our research
• Strategic behaviour exists and you can detect it at the system
level but not always at the school level or student level.
• In most cases it’s not very problematic and heavy.
• But you have to organize some prevention.
18
OUTCOME INFORMATION
AT THE SYSTEM LEVEL
19
Information at the system level
Information about student assessment can also be used at the level
of the educational system:
• Cohort flow and efficiency
• Results by group
• Trends, Variability, Spread
Each year: ‘The State of Education’.
• Thematic enquiry
• Aggregation of surveillance results
Next page: Main message in 2016 about growing inequality.
20
21
Conclusion
Research at the system level is not only summing up results of the
inspections of schools.
Research in its own right.
Gives different views and other insights you cannot see at the
individual schools.
Trend towards:
• Surveillance at system level
• Thematic enquiries
22
DISCUSSION ABOUT
INDICATORS FOR
EDUCATIONAL OUTCOME
23
Political discussions and complaints of some
schools
There is a lot of discussion about the use of outcome indicators by
the inspectorate:
• Inspectorate incites too much testing.
• Anti testing movement. You shall not test children.
• Using indicators for accountability incites strategic behaviour.
• Inspectorate is too strict (but insufficient outcome according to
inspectorate is about 6 - 10%).
But also ….
24
Press and publicity
For good schools the indicators and the norms don’t give a challenge
Press uses outcome information in their own way.
Choice lists, rankings and league tables.
They say mostly: inspectorate is too soft.
25
Example: A sample of winners and losers according
to the press (Elseviers Magazine, jan 2017)
26
Value added score in vocational education 2015. You
can at most distinguish three groups
Schools with much value
added
Schools with low value
added
27
Difference between press and inspectorate
Same data but different view.
Inspectorate
• Correction for fair judgment
• 6.5% insufficient
• No lists
In the press
•
League tables from low to high.
They are always fond of lists.
•
Winners and losers
•
No correction
•
13.5% insufficient
Elseviers Magazine, jan 2017
28
Conclusion
The stakeholders have conflicting needs and the inspectorate has to
deal with them.
29
DEVELOPMENT OF
OUTCOME INDICATORS
30
Response to criticism
How can you respond to criticism and reduce risks for strategic
behaviour?
31
1. Nudging
Nudging is an answer to undesirable strategic behaviour.
Give people a little push (nudge) in the good direction.
General examples of nudging:
• Default value in electronical form of student loan is maximum or
zero. Last possibility gives a significant smaller loan (the same
with a creditcard).
• To sign your tax form in advance or after. The first possibily gives
a better and fairer completion of the tax form.
Oreopoulos, Behaviouoral economics of Education (2015; http://www.behaviouralinsights.co.uk/
32
Nudging and educational outcome
Balanced score model.
(With our data this is not easy in primary education).
A small change in an indicator can change behaviour.
- Possibilities for recalcultation.
- Give more weigth to a correction than statistically is necessary.
But ………
more important is the interaction with the stakeholders and
empowerment of schools.
33
2. Three design principles
1. Interaction
Design indicators together with the stakeholders who have interest
in them. Ownership of indicators.
2. Variety
Variation in the concepts of outcome and definitions.
3. Dynamics
Not only measuring the outcome but also the underlying process.
De Bruin (2001), Raad voor Maatschappelijke ontwikkeling (2011), Hargreaves en Braun (2014)
34
3. Empowerment of the schools
They have their own opinion about educational results.
• Focusing on results
• Data driven teaching
They analyse results systematically with regard to:.
• Individual children/students
• Groups
• The program
• Teacher behaviour.
Management has a focus on good results.
Visscher, Coe (2002); Hamilton, Halverson e.a. (2009); Visscher (2015)
35
FINAL REMARKS
36
Final remarks: evolution
From
To
•Small data sets.
•Only inspectorate has
information
•Inspectorate constructs
indicators.
•Emphasis on technical
refinement
•Inspectorate judges the
outcome of school.
•Inspection of schools
•School level.
•Large data sets.
•Sharing and other stakeholders
have their own information
•Joint construction with
stakeholders. Joint ownership.
•Outcome in a broader context.
37
•Empowerment of schools to set
their own standards.
•Inspection of school boards.
•System level gets more
emphasis.
Four questions to start
1. Is the use of indicators for educational outcome for accountability
of schools by an inspectorate always necessary? When not?
2. Do you deal with undesirable strategic behavior in schools when
the inspectorate uses the assessment of educational results for
accountability? How to prevent?
3. We think norms and strong benchmarks for the minimal level of
schools are necessary. But how about the better schools? Do the
inspectorates need a differentiated use of educational results for
accountability? How to design?
4. Can schools give their own calculation of educational results for
accountability? When it is valid? How to compare?
5. …..
6. …..
Thank you for your attention.
Do you have questions?
39
Information about Dutch schools (mostly in
Dutch)
Website of the Dutch Inspectorate:
www.onderwijsinspectie.nl
Information about schools by the Dutch inspectorate:
https://zoekscholen.onderwijsinspectie.nl/
Website of the organisations of schools in primary and secondary
education with school information:
www.scholenopdekaart.nl
The State of Education in the Netherlands (English)
https://www.destaatvanhetonderwijs.nl/downloaden/englishversion
40