New Survey Questionnaire
Indicators in PISA and NAEP
Jonas Bertling
Director, NAEP Survey Questionnaires, ETS
RWJF & ETS Workshop
Understanding the opportunities for collaboration between the health and
education sectors in a culture of health
Princeton, May 5, 2016
Main talking points today
• Survey questionnaires in educational large-scale assessments
started collecting broader types of contextual variables reflecting
the increased policy interest in “noncognitive” skills and
additional outcomes.
• PISA broadened the range of constructs captured with the
student questionnaires and introduced health and well-being
facets in 2015. PISA 2018 will include an optional well-being
questionnaire.
• NAEP started innovating their survey questionnaires and will
introduce a series of new contextual modules (including
noncognitive indices) in 2017.
J. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016
Copyright © 2016 by Educational Testing Service. All rights reserved. ETS, the ETS logo and MEASURING THE
POWER OF LEARNING are registered trademarks of Educational Testing Service (ETS). 36319
2
Main talking points today (continued)
• NAEP 2017 contextual modules can also be mapped to PISA 2018
draft well-being framework.
• PISA conceptualizes noncognitive variables explicitly as additional
outcomes; NAEP currently models them on the predictor
side only.
• Challenges fall especially into three categories: Measurement,
Perceived Intrusiveness and Respondent Burden.
J. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016
3
Changing role of large-scale assessments
How are students in the Nation
doing in terms of other outcomes
than achievement?
How do students approach
learning situations and utilize
learning opportunities?
How do learning opportunities
differ across the Nation?
How do subgroups perform?
J. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016
4
Shift in policy interest
Measure contextual variables
Measure constructs of own interest
• Describe test-taker populations
• Report additional outcomes
• Improve estimates of achievement
(conditioning model)
• Measure trend
• Compare subgroups
• Evaluate policies
• Accountability
Figure from Bertling, Borgonovi, & Almonte (2016)
J. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016
5
J. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016
6
Survey questionnaires data put achievement
results into context …
… and can add value to large-scale assessments by
capturing important additional variables.
35
30
Minutes
25
20
15
10
5
0
NAEP
PISA
TIMSS/PIRLS
J. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016
PISA
Programme for International Student Assessment
Organisation for Economic Co-operation and Development (OECD)
8
PISA broadened the range of constructs captured with survey
questionnaires since the first cycle in 2000
• Since its early cycles, PISA has included questions capturing
students’ attitudes and sense of belonging to school
• PISA 2012 introduced new domain-general student attitudes and
behaviors (e.g., perseverance, openness for problem solving)
• PISA 2015 added additional noncognitive outcomes also covering
health and well-being
• This trend continues with PISA 2018
Figure from Bertling, Borgonovi, & Almonte (2016)
J. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016
PISA 2015 Questionnaire Framework
Figure from OECD (2016)
J. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016
Well-being-related constructs in PISA 2015
• General life satisfaction
• Participation in physical education classes at school
• Engagement in moderate and vigorous physical exercise
• School-related well-being (sense of belonging, bullying, test anxiety)
A larger number of additional health- and well-being-related questions (e.g., BMI, dietary
habits, enjoyment of physical activity, etc.) were piloted but not taken to operational
largely due to timing constraints.
J. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016
11
Well-being questionnaire in PISA 2018
• Developed draft
framework for PISA
2018 Module
DRAFT*
QUALITY OF LIFE: INDICATORS
Subjective Indicators
Objective Indicators
Perceptions
Affect
Satisfaction
Life as a Whole
QUALITY OF LIFE: DIMENSIONS
• OECD is planning an
optional
questionnaire
devoted to student
well-being for PISA
2018 in addition to
other noncognitive
factors captured
with the regular
student
questionnaires.
Self
Health
Education and skills
Psychological functioning
School Environment
Social connections
Schoolwork
Social Well-being
Out-of-School Environment
Social connections
Material conditions
Leisure time
Overall Well-being
Possible
composite
indices
(Examples)
Subjective Well-being
Work/School-life Balance
Overall Life Satisfaction
*Note: The PISA 2018 well-being framework is not a public document and currently remains a DRAFT version, is subject
to all the normal OECD copyright provisions, and should clearly be acknowledged with the OECD as owner and author.
J. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016
12
NAEP
National Assessment of Educational Progress
National Center for Education Statistics (NCES)
13
Innovations underway for the NAEP Survey
Questionnaires
Expand constructs to include more
I. contextual factors critical for student
achievement
Improve reporting of contextual factors
II. for more robustness & policy relevance
Improve pre-testing and questionnaire
III. design for digital delivery to maximize
validity and keep student burden low
J. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016
NAEP Contextual Variables
(SURVEY QUESTIONNAIRES)
BUT: The term “noncognitive” is not
intended to be interpreted literally. All
aspects of human thought and
behavior are related to cognition!
“Noncognitive” Student Factors:
Skills, strategies, attitudes and
behaviors that are distinct from
content knowledge and academic
skills (e.g., motivational and selfregulatory variables)
Opportunity-to-Learn (OTL) Factors:
Students’ exposure to
opportunities to acquire relevant
knowledge and skills, both at
school and outside of school.
Domaingeneral
factors
(“Core”)
Subjectspecific
factors
“Noncognitive”
Student Factors
Cognitive Ability*
Achievement
OTL Outside of School
OTL at School
Basic Demographics
*not measured with the NAEP survey questionnaires
(COGNITIVE ASSESSMENT)
Balanced reporting
approach based on indices
and stand-alone questions
J. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016
Implementation for NAEP 2017 Core, Reading, Writing and Mathematics:
Item
Development
Pre-testing in
Cognitive Labs
Large-scale
Pilot Testing
Operational
Administration
2014 
2014-15
2016 
2017
Domaingeneral
(Core)
Noncogs
OTL
Outside of
School
OTL
at School
1
Subjectspecific
“Noncognitive”2
1
2
Student
Factors
3
4
3
OTL Outside of School
5
4
6
OTL At School
5
7
5 Core Modules
Additional Subjectspecific Modules
1. Grit/Perseverance
(including self-control)
2. Desire for Learning
1. Self-Efficacy/Confidence
2. Achievement Goals
3. Interest & Enjoyment
3. Components of SES
4. Technology Use
5. School Climate
4. Out-of-School Activities
5. Resources for Learning
and Instruction
6. Organization of
Instruction 16
7. Teacher Preparation
Basic Demographics
J. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016
Mapping NAEP to PISA 2018 WB Framework
DRAFT*
QUALITY OF LIFE: INDICATORS
Subjective Indicators
Objective Indicators
Perceptions
Affect
Satisfaction
Grit/Perseverance,
Desire for Learning,
Self-Efficacy
School Climate
SES
QUALITY OF LIFE: DIMENSIONS
Life as a Whole
Self
Health
Education and skills
Psychological functioning
School Environment
Social connections
Schoolwork
Social Well-being
Out-of-School Environment
Social connections
Material conditions
Leisure time
Overall Well-being
Possible
composite
indices
(Examples)
Subjective Well-being
Work/School-life Balance
Overall Life Satisfaction
*Note: The PISA 2018 well-being framework is not a public document and currently remains a DRAFT version, is subject
to all the normal OECD copyright provisions, should clearly be acknowledged with the OECD as owner and author.
J. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016
17
Difference between PISA and NAEP approaches
Additional
Outcomes
Achievement
Predictors
J. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016
18
Challenges
1. Measurement
2. Perceived Intrusiveness
3. Respondent Burden
Figure from Bertling, Borgonovi, & Almonte (2016)
J. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016
19
Summary
• Survey questionnaires in educational large-scale assessments
started collecting broader types of contextual variables reflecting
the increased policy interest in “noncognitive” skills and
additional outcomes.
• PISA broadened the range of constructs captured with the
student questionnaires and introduced health and well-being
facets in 2015. PISA 2018 will include an optional
well-being questionnaire.
• NAEP started innovating their survey questionnaires and will
introduce a series of new contextual modules (including
noncognitive indices) in 2017.
J. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016
20
Summary (continued)
• NAEP 2017 contextual modules can also be mapped to PISA 2018
draft well-being framework.
• PISA conceptualizes noncognitive variables explicitly as additional
outcomes; NAEP currently models them on the predictor
side only.
• Challenges fall especially into three categories: Measurement,
Perceived Intrusiveness and Respondent Burden.
J. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016
21
Thank you!
Questions? Comments?
[email protected]
Twitter: @JonasBertling
J. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016
Copyright © 2016 by Educational Testing Service. All rights reserved. ETS, the ETS logo and MEASURING THE
POWER OF LEARNING are registered trademarks of Educational Testing Service (ETS). 36319
22