Advances in Measuring Non-Cognitive Skills
Tim Kautz
Mathematica Policy Research
RISE Conference
June 17, 2015
This draft, June 13, 2016
Kautz
Inequality
1 / 32
Measuring Non-Cognitive Skills
We have come to focus on achievement test scores to assess
students, teachers, and schools (and even countries)
Achievement tests miss important non-cognitive skills that can
be shaped through interventions and required for success in
school and beyond (Kautz, Heckman, Diris, ter Weel, and
Borghans, 2014)
Key policy-makers and organizations are planning to measure
non-cognitive skills at a large-scale but there are challenges
Recent advances in measurement provide some solutions to
these challenges relevant for international use
Kautz
Inequality
2 / 32
Outline
1
Why measure non-cognitive skills?
2
Traditional approaches to measuring non-cognitive skills
3
Challenges and advances in measuring non-cognitive skills
Kautz
Inequality
3 / 32
Source(s)
James J. Heckman, John E. Humphries, and Tim Kautz (2014). The
Myth of Achievement Tests: The GED and the Role of Character in
American Life. Chicago, IL: University of Chicago Press.
Kautz
Inequality
4 / 32
Source(s)
Fosteringzandz
MeasuringzSkills:z
ImprovingzCognitivez
andzNon-CognitivezSkillsztoz
PromotezLifetimezSuccess
TimzKautz,zJameszJ.zHeckman,zRonzDiris,z
BaszterzWeel,zLexzBorghans
Tim Kautz, James J. Heckman, Ron Diris, Bas ter Weel, Lex Borghans
(2014). Fostering and Measuring Skills Improving Cognitive and
Non-cognitive Skills to Promote Lifetime Success. OECD Education
Report.
Kautz
Inequality
5 / 32
1. Why measure non-cognitive skills?
Kautz
Inequality
6 / 32
There are three main reasons to measure non-cognitive skills
1
IQ and achievement tests miss important skills
2
Non-cognitive skills predict important long-term outcomes
3
Non-cognitive skills are malleable and can be improved through
interventions
Kautz
Inequality
7 / 32
(1) IQ and Achievement tests miss non-cognitive skills
We have come to place great emphasis on cognitive
tests (Heckman and Kautz, 2014)
But these tests are not all that predictive of later life outcomes
Kautz
Inequality
8 / 32
Figure 1: Validities of Cognitive Measures in Age-35 Labor Market
Outcomes (Adjusted R-Squared)
.2
(a) Males
Hourly Wage
Hours Worked
0
.05
Adjusted R−Squared
.1
.15
Earnings
IQ
AFQT GPA
IQ
AFQT GPA
IQ
AFQT GPA
Source: Heckman and Kautz (2012) using the National Longitudinal Survey of Youth (NLSY79)
Kautz
Inequality
9 / 32
(2) Non-cognitive skills predict important life outcomes
A common measurement system is the “Big Five” (Openness
to Experience, Conscientiousness, Extraversion, Agreeableness,
Neuroticism)
Conscientiousness – the tendency to be organized and complete
tasks – is the most predictive across a wide variety of outcomes
Kautz
Inequality
10 / 32
Figure 2: Association of the Big Five and Intelligence with Years of
Schooling in GSOEP
(a) Males
Kautz
Inequality
11 / 32
95
Hanushek_2011 978-0-444-53444-6
Figure 3: Correlations of Mortality with Non-Cognitive Skills, IQ, and
Personality Psychology and Economics
Socioeconomic Status (SES)
113
0.30
0.25
Correlation
0.20
0.15
0.10
0.05
0
SES
IQ
C
E/PE
N
A
Figure 1.18 Correlations of Mortality with Personality, IQ, and Socioeconomic Status (SES).
Notes:
The figure
represents
results from
a meta-analysis
of 34(2007).
studies. Average effects (in the correlation
Source:
Roberts,
Kuncel,
Shiner,
Caspi,
and Goldberg
metric) of low socioeconomic status (SES), low IQ, low Conscientiousness (C), low Extraversion/Positive
Emotion (E/PE), Neuroticism (N), and low Agreeableness (A) on mortality. Error bars represent standard
Kautz
Inequality
12 / 32
(3) Non-cognitive skills are malleable and can be improved through
interventions
IQ becomes relatively rank stable by age 10 while non-cognitive
skills are more variable (Almlund, Duckworth, Heckman, and
Kautz, 2011)
Neuroscience shows that this malleability is associated with the
slow development of the prefrontal cortex (Walsh, 2005)
Consistent with review of the intervention literature (Kautz,
Heckman, Diris, ter Weel, and Borghans, 2014)
Only interventions that started before age 3 had a long-term
effect on IQ
Many interventions starting after age 3 have effectively
improved outcomes by improving non-cognitive skills
Adolescent interventions that teach personality skills in the
workplace (or specific context) are promising
Kautz
Inequality
13 / 32
2. Traditional approaches to measuring non-cognitive skills
Kautz
Inequality
14 / 32
Non-cognitive skills are typically measured using self reports
Self-reports are a primary method of measuring non-cognitive
skills
The “Big Five” is a relatively well-accepted taxonomy
Kautz
Inequality
15 / 32
Table 1: The Big Five Traits
OCEAN
II.
Trait
Openness to Experience
Conscientiousness
III.
Extraversion
IV.
Agreeableness
V.
Neuroticism
I.
Kautz
Definition of Trait
The tendency to be open to new aesthetic, cultural, or intellectual experiences.
The tendency to be organized, responsible, and
hardworking.
An orientation of one’s interests and energies toward the outer world of people and things rather
than the inner world of subjective experience;
characterized by positive affect and sociability.
The tendency to act in a cooperative, unselfish
manner.
Neuroticism is a chronic level of emotional instability and proneness to psychological distress.
Emotional stability is predictability and consistency in emotional reactions, with absence of
rapid mood changes.
Inequality
16 / 32
Example question used to measure Conscientiousness
I see myself as someone who tends to be lazy
Rating scale: 1 – “strongly agree”, 5 – “strongly disagree”
Kautz
Inequality
17 / 32
3. Challenges and advances in measuring non-cognitive skills
Kautz
Inequality
18 / 32
Basic measurement challenges
All psychological measurements are based on performance on a
task (Heckman and Kautz, 2012)
An interpretive problem lies at the heart of any psychological
measurement system for any particular trait
It is necessary to standardize for incentives and the effects of
other traits in performing a task
Kautz
Inequality
19 / 32
Figure 4: Determinants of Task Performance
Incentives
Effort
Task
Performance
Non-Cognitive
Character
Skills
Skills
Test Scores
Self-Reports
Other Behaviors
Cognitive
Skills
Kautz
Inequality
20 / 32
Figure 5: Decomposing Variance Explained for Achievement Tests and
Grades into IQ and Character: Stella Maris Secondary School,
Maastricht, Holland
Source: Borghans, Golsteyn, Heckman, and Humphries (2011). Note: Grit is a measure of persistence on tasks (Duckworth,
Peterson, Matthews, and Kelly, 2007).
Kautz
Inequality
21 / 32
Can boost IQ by 15 points by giving candies for correct answers
— the Black/White gap in IQ in U.S
Segal (2012) shows that introducing performance-based cash
incentives in a low-stakes administration of a measure of IQ
increases performance substantially among roughly one-third of
participants.
Kautz
Inequality
22 / 32
Reference bias is one challenge with self-reports
Respondents rate themselves relative to their peers rather than
the population at large
Reference bias can be especially problematic if comparing
across different contexts (e.g. between countries and schools)
A form of a situation (peer group) affecting measurement
Kautz
Inequality
23 / 32
Rank in Big Five Conscien7ousness (High to Low) Figure 6: National Rank in Big Five Conscientiousness and Average
Annual Hours Worked
Finland 1 United States 2 Chile 3 France 4 Slovenia 5 Canada 6 Turkey 7 Italy 8 Portugal 9 Greece 10 United Kingdom 11 Spain 12 Austria 13 Germany 14 Poland 15 Australia 16 Estonia 17 Mexico 18 Switzerland 19 New Zealand 20 Netherlands 21 Belgium 22 Czech Republic 23 Slovakia 24 South Korea 25 Japan 26 1300 1500 1700 1900 2100 Annual Hours Worked 2300 2500 Source: The Conscientiousness ranks come from Schmitt, Allik, McCrae, and Benet-Martı́nez (2007). These measures were
taken in 2001 (Schmitt, 2002). The hours worked estimates come from Organisation of Economic Cooperation and
Development (2001). Notes: Several countries are omitted due to lack of data.
Kautz
Inequality
24 / 32
Anchoring vignettes address reference bias
Anchoring vignettes are an additional question that helps to
standardize the situation
Describe a situation and ask the respondent to rate
performance
Kautz
Inequality
25 / 32
Example anchoring vignette from Primi, Zanon, Santos, De Fruyt, and
John (2016) used in Brazil
“Aline leaves her belongings in a mess, hates cleaning the house,
and usually doesn’t complete her homework.”
How organized do you think Aline is?
(1) Not at all (2) A little (3) Moderately (4) Very much (5)
Completely
Kautz
Inequality
26 / 32
Anchoring vignettes can be effective
Kyllonen and Bertling (2013) demonstrates that including
anchoring vignettes changes a cross-country relationship
between teacher support and achievement from -0.45 to 0.29 in
PISA 2012
Kautz
Inequality
27 / 32
Figure 7: Determinants of Task Performance
Incentives
Effort
Task
Performance
Non-Cognitive
Character
Skills
Skills
Test Scores
Self-Reports
Other Behaviors
Cognitive
Skills
Kautz
Inequality
28 / 32
Explore alternative measures available to schools
Measures traditionally viewed as “outcomes” contain
information on non-cognitive skills
“Real-world” non-cognitive skill measures: grades, absences,
credits earned, disciplinary infractions
Highly predictive of later behavior
Kautz
Inequality
29 / 32
Table 2: Predictive Validity (R 2 ) from Ninth-Grade Measures on Various
Outcomes
Ninth-Grade Measure
Outcome
Explore
Test GPA
ACT Score (Grade 11)
GPA (Grade 11)
Absences (Grade 11)
Arrested within 4 Years
Grad HS within 5 Years
Enroll College within 6 Years
Grad College within 10 Years
0.78
0.21
0.09
0.06
0.11
0.15
0.17
Kautz
0.22
0.49
0.22
0.14
0.35
0.20
0.17
Credits Absences Discipline
0.05
0.28
0.12
0.12
0.36
0.16
0.07
Inequality
0.10
0.20
0.35
0.10
0.23
0.12
0.09
0.02
0.05
0.03
0.10
0.06
0.03
0.01
All
0.79
0.52
0.39
0.20
0.41
0.25
0.23
30 / 32
Figure 8: Predictive Validity of Cognitive and Non-Cognitive Skill for
High School Graduation
0
.1
R−Squared
.2
.3
.4
Grad HS within 5 Years
Cognitive
Non−Cognitive
Total
Measure
Kautz
Inequality
31 / 32
Conclusions
Non-cognitive skills predict outcomes and are malleable
There are challenges with implementing traditional measures of
non-cognitive skills at scale
Recent advances suggest some promising methods to address
the challenges
Kautz
Inequality
32 / 32
Almlund, M., A. Duckworth, J. J. Heckman, and T. Kautz (2011): “Personality
Psychology and Economics,” in Handbook of the Economics of Education, ed. by E. A.
Hanushek, S. Machin, and L. Wößmann, vol. 4, pp. 1–181. Elsevier, Amsterdam.
Borghans, L., B. H. H. Golsteyn, J. Heckman, and J. E. Humphries (2011):
“Identification Problems in Personality Psychology,” Personality and Individual Differences,
51(3: Special Issue on Personality and Economics), 315–320.
Duckworth, A. L., C. Peterson, M. D. Matthews, and D. R. Kelly (2007): “Grit:
Perseverance and Passion for Long-Term Goals,” Journal of Personality and Social
Psychology, 92(6), 1087–1101.
Heckman, J. J., and T. Kautz (2012): “Hard Evidence on Soft Skills,” Labour
Economics, 19(4), 451–464, Adam Smith Lecture.
(2014): “Achievement Tests and the Role of Character in American Life,” in The
Myth of Achievement Tests: The GED and the Role of Character in American Life, ed. by
J. J. Heckman, J. E. Humphries, and T. Kautz, pp. 3–56. University of Chicago Press,
Chicago, IL.
Kautz, T., J. J. Heckman, R. Diris, B. ter Weel, and L. Borghans (2014):
“Fostering and Measuring Skills: Improving Cognitive and Non-Cognitive Skills to Promote
Lifetime Success,” OECD, Forthcoming.
Kyllonen, P. C., and J. P. Bertling (2013): “Innovative questionnaire assessment
methods to increase cross-country comparability,” Handbook of international large-scale
assessment: Background, technical issues, and methods of data analysis, 277.
Organisation of Economic Cooperation and Development (2001): “OECD
Employment and Labour Market Statistics,” Data available from
http://www.oecd-ilibrary.org/employment/data/
oecd-employment-and-labour-market-statistics_lfs-data-en.
Kautz
Inequality
32 / 32
Primi, R., C. Zanon, D. Santos, F. De Fruyt, and O. P. John (2016): “Anchoring
Vignettes,” European Journal of Psychological Assessment.
Roberts, B. W., N. R. Kuncel, R. L. Shiner, A. Caspi, and L. R. Goldberg (2007):
“The power of personality: The comparative validity of personality traits, socioeconomic
status, and cognitive ability for predicting important life outcomes,” Perspectives in
Psychological Science, 2(4), 313–345.
Schmitt, D. P. (2002): “Are Sexual Promiscuity and Relationship Infidelity Linked to
Different Personality Traits across Cultures? Findings from The International Sexuality
Description Project,” Online Readings in Psychology and Culture, 4(Unit 4), 1–22,
Retrieved from http://scholarworks.gvsu.edu/orpc/vol4/iss4/4.
Schmitt, D. P., J. Allik, R. R. McCrae, and V. Benet-Martı́nez (2007): “The
Geographic Distribution of Big Five Personality Traits: Patterns and Profiles of Human
Self-Description Across 56 Nations,” Journal of Cross-Cultural Psychology, 38(2), 173–212.
Segal, C. (2012): “Working When No One Is Watching: Motivation, Test Scores, and
Economic Success,” Management Science, 58(8), 1438–1457.
Walsh, D. A. (2005): Why Do They Act That Way? A Survival Guide to the Adolescent
Brain for You and Your Teen. Free Press, New York, 1 edn.
Kautz
Inequality
32 / 32