DRAFT – FOR COMMENT, NOT QUOTATION
Measures and determinants of non-cognitive ability across seven English-speaking
countries
Dr. Nicholas Biddle – Research School of Social Sciences, Australian National University,
Canberra, Australia
Sarah Ball – Research School of Social Sciences, Australian National University, Canberra,
Australia
Samuel Weldeegzie, – Crawford School of Public Policy and Research School of Social
Sciences, Australian National University, Canberra, Australia
[email protected]
Centre for Aboriginal Economic Policy Research
Copland Building #24
Australian National University
Canberra, ACT, 2601
Australia
Abstract
While non-cognitive skills such as self-control and perseverance are found to be strong
predictors of success, little is known about how these skill sets are distributed across
countries and what determines them in a given country. After developing an index for such
skills using principal component analysis from a range of questions, this paper uses the most
recent data from PISA 2012 to examine what factors are associated with cognitive and noncognitive abilities in seven English-speaking countries. However, we still need further
research on identifying exactly which skills are most important and improve measurement
thereof.
Keywords: Non-cognitive ability, Programme of International Student Assessment,
Education policy, Educational attainment
DRAFT – FOR COMMENT, NOT QUOTATION
Measures and determinants of non-cognitive ability across seven English-speaking
countries
1. Introduction
Education plays a key role in alleviating the negative outcomes associated with poverty and
disadvantage. There is an extensive literature on measuring returns to education with
experimental, quasi-experimental and observational data showing that those individuals,
communities and countries that have relatively high levels of education tend to have better
outcomes across a range of economic and other wellbeing measures (Psacharopoulos and
Patrinos, 2004, Carneiro, Heckman et al. 2010). Other research has shown that the education
levels of parents has a significant and substantial effect on the education outcomes of their
children (Black and Devereux 2011).
Policy in most developed countries reflects this with considerable public investment in early
childhood, school and post-school education. However, this investment has tended to focus
on school attainment and measures of cognitive ability such as test scores in Mathematics,
Literacy and Science. In the United States (US), one of the major policy focuses over the last
decade has been No Child Left Behind (NCLB), which has been based on, and assessed
against standardised student assessments of mathematics and literacy (Dee and Jacob 2011).
Similarly, a major policy driver in Australia is NAPLAN, or the National Assessment
Program – Literacy and Numeracy. These sets of tests, administered to students across
Australia in Years 3, 5, 7 and 9, make up a large focus of the My School1 website in which all
schools in Australia are ranked. This information has been used extensively in arguing for
changes to school funding formulas.
Similarly, when countries are ranked internationally in terms of education performance, a
similar set of tests are used, administered as part of the OECD’s Programme for International
Student Assessment (PISA) for 15 year olds. Like NAPLAN, the PISA tests focus on
mathematics and reading ability, but also extend to science scores.
While the focus of the PISA is on comparisons of literacy, numeracy and science ability,
significant personal and attitudinal data is also collected through a background questionnaire.
This paper argues that while cognitive ability is important, skills like self-control, persistence
and determination have also been shown to have a major impact on educational outcomes and
later measures of wellbeing and are therefore deserving of a much greater policy and research
focus. This set of skills is commonly referred to as non-cognitive ability and has been shown
to improve the likelihood of school completion (Heckman 2008, Barón and Cobb-Clark
2010), employment (Mischel, Shoda et al. 1989, Heckman and Masterov 2007), and even
reduce crime and single parenthood (Heckman 2006, Heckman and Kautz 2013).
While the PISA results are used to measure cognitive ability across countries, we know very
little about how non-cognitive ability is distributed internationally, and what the within-
1
http://www.myschool.edu.au/
DRAFT – FOR COMMENT, NOT QUOTATION
country determinants are. This paper aims to fill this gap. For reasons outlined in the
methodology section, we focus our analysis on seven English speaking countries – Australia,
Canada, Ireland, New Zealand, Scotland, UK (excluding Scotland), and the USA. We begin
with a discussion of previous research on non-cognitive ability (in Section 2), followed by a
discussion of issues related to measurement (in Section 3). We then discuss our methodology
(in Section 4), followed by a set of descriptive statistics (Section 5) and more detailed
modelling (Section 6). The final section in the paper provides a summary and some
concluding comments.
2. Previous research
The role of non-cognitive skills in early life and education is supported by a growing body of
research. Research on non-cognitive ability rose to prominence (at least within the field of
economics) through the work of James Heckman. With colleagues, Heckman has undertaken
a number of significant studies that highlight the role of non-cognitive ability on a variety of
socioeconomic outcomes. Heckman and Rubinstein (2001) for example, analysed students
who completed the General Education Development (GED) program, which certifies that
students have equivalent cognitive ability to a high school graduate. The study found that
those who complete the GED do not have better outcomes than their peers who do not
complete the GED. The authors argue that this is because GED students lack the noncognitive skills of their school-completing peers (Heckman and Rubinstein 2001).
Heckman’s work on the long-term benefits of investment in early childhood is also notable
(Heckman 2000, Heckman 2006, Heckman and Masterov 2007, Heckman, Moon et al. 2010).
A key example is his work on the Perry Pre-school Programme. The programme was carried
out from 1962-67 and provided high quality pre-school education to 128 African-American
children. Children who participated in the evaluation of the programme were followed up at
27 and 40 years of age. Considered somewhat of a failed experiment, by 8-9 years of age the
IQ scores of participants were no better than the control. However, in the later follow up
children who had participated showed higher rates of school completion, were more likely to
be employed and were less likely to have spent time on welfare. In searching for why this
might have occurred, Heckman and colleagues decided to look at data related to personal
behaviour which had been collected but never analysed. This research showed an
improvement in non-cognitive factors, like self-control, which could be attributed to as much
as 65% of the total benefit of the intervention (Heckman and Masterov 2007, Tough 2012).
The work of Walter Mischel is also significant when considering non-cognitive skills. The
famous Stanford Marshmallow test led by Mischel has contributed much to our
understanding of delayed gratification and self-control and its impact on life outcomes.
Mischel conducted the test in the late 1960s at Stanford University using a group of preschoolers. A child is asked to choose between one treat now (a marshmallow for example) or
to wait a short time for a two treats later. It is essentially a test of self-control, willpower and
delayed gratification (Mischel, Shoda et al. 1989). Mischel tracked many of these children
for decades, collecting data on each child’s education, health and other factors. He found that
those who had resisted temptation on average had higher academic scores, achieved higher
DRAFT – FOR COMMENT, NOT QUOTATION
educational degrees, earned more money and even had a lower body mass index (Mischel
2014) .
A recent paper released by the Brookings Institution titled The Character Factor: Measures
and Impact of Drive and Prudence evaluated existing datasets from the US in order to find
reliable measures of non-cognitive skills such as self-control, delayed gratification and
persistence (Reeves, Venator et al. 2014). The authors then used one of these measures (a
hyperactivity scale) to estimate its influence in the early and middle childhood years for
educational and other outcomes. Their findings show a significant relationship between drive
and prudence and higher educational attainment even when controlling for other factors.
They also show that children from disadvantaged backgrounds were more likely to perform
poorly on the early measures of non-cognitive skill (Reeves, Venator et al. 2014).
In the United Kingdom the Effective Pre-school, Primary and Secondary Education 3-16+
(EPPSE 3-16+) project is currently being used to explore the impact of pre-school on child
development. This research also highlights the role of non-cognitive skills on developmental
trajectories. The project is a longitudinal, mixed methods study, commissioned by the
Department for Education. It has tracked 3172 students since 1997 across three phases of
schooling (pre-school, primary and secondary school) and is now looking at their posteducational outcomes (Sammons, Sylva et al. 2014). In addition to academic and socialbehavioural outcomes from testing, grades and teacher ratings, the EPPSE 3-16+ included
questionnaires on student disposition (including non-cognitive markers such as selfregulation). These questionnaires used self-reported measures and while acknowledging the
issues of reliability the researchers state that, “there is a definite move towards both
acknowledging the importance of student well-being and non-cognitive outcomes, and the
development of robust measures to capture these aspects of young people's experiences.”
(Sammons, Sylva et al. 2014).
Research related to other comparable countries is somewhat more limited. In Australia, the
most recent, relevant work has been done by Professor Deborah Cobb-Clark (Barón and
Cobb-Clark 2010, Cobb-Clark and Tan 2011). In her 2010 paper with Juan D. Baron, CobbClark uses data collected for the Youth in Focus Project2, specifically focusing on the locus
of control. She refers to literature which shows that those with an internal sense of control are
likely to be more hardworking, motivated and persistent in problem solving (Gatz and Karel
1993, Coleman and DeLeire 2003). Cobb-Clark and Baron found that young people with an
internal locus of control were more likely to complete school and on completion meet the
requirements to enter university.
The Youth in Focus Survey is about the experiences of young people in Australia. It asks questions about
family background, living arrangements, education, work, relationships, income, health, spare time, and
aspirations and attitudes. It also uses administrative data collected from the social security system. Two waves
were conducted in 2006 and 2008. Young people aged 18 and one of their parents or carers were interviewed.
The first wave included a sample of 4000 young people and 3900 parents or carers.
2
DRAFT – FOR COMMENT, NOT QUOTATION
3. Measurement
Despite the research noted above, education policy debate appears to focus nearly exclusively
on the development of cognitive skills. A key example of this can be seen in the recent
review of the Australian Curriculum which stated that personal and social capability, critical
and creative thinking, and ethical and intercultural understanding, should no longer be treated
in a cross-curricular fashion and only be embedded in subjects and areas of learning where
relevant (Department of Education 2014).
One of the key reasons for government policy to focus on cognitive skills is the ability to
standardise measures across the country, and internationally. Measuring non-cognitive skills
is significantly more problematic and research shows that the complexity and breadth of noncognitive skills has impaired its use in research and policy (Reeves, Venator et al. 2014).
There are several key reasons why non-cognitive measures have proven problematic. Firstly,
when dealing with social and emotional traits it can be difficult to separate the findings from
situational and contextual influences. Secondly, much research relies on self-reported
measures which implicitly assume that respondents are capable of unbiased assessment of
their personality. Finally, it can also be a challenge to ensure consistency of answers to a
psychometric task over time or across observations (Hoover 2013, Reeves, Venator et al.
2014, West, Kraft et al. 2014). The overall risk of unreliable measures and issues of validity
continue to plague more standardised approaches of measuring non-cognitive skill (Kyllonen
2005).
Angela Duckworth however has done extensive work on the value of non-cognitive skills in
predicting academic performance and her use of the ‘grit’ scale to measure determination and
drive is a centrepiece in the current research (Duckworth and Seligman 2005, Duckworth,
Quinn et al. 2012). The grit scale uses twelve brief statements, such as ‘I am a hard worker’
or ‘My interests change from year to year’, on which respondents must rate themselves on a 5
point scale. While simple, it was shown to be surprisingly effective at predicting later success
(Duckworth, Peterson et al. 2007). A key example arose from work Duckworth and
colleagues did with West Point Military Academy. The grit scale was used to test over 1200
cadets as they entered summer training and proved more effective than the Academy’s own
extensive evaluation system at predicting which cadets would complete the training
(Duckworth, Peterson et al. 2007).
Duckworth and colleagues have done extensive work on the grit scale and significant
research to show its effectiveness (Duckworth and Seligman 2005, Duckworth, Peterson et al.
2007, Borghans, Duckworth et al. 2008, Duckworth, Quinn et al. 2012, West, Kraft et al.
2014). However, they also openly admit that no measure is perfect (Duckworth, Quinn et al.
2011). The very extensiveness of the traits that can be considered and the issue of validating
self-reported data, continue to make standardisation a major challenge (Reeves, Venator et al.
2014).
One possible approach that has been suggested is the use of resources that have been
developed by psychologists for evaluating non-cognitive skills. Some of these measures
include the Big Five taxonomy (Borghans, Duckworth et al. 2008, Cobb-Clark and Tan
DRAFT – FOR COMMENT, NOT QUOTATION
2011), the Mayer-Salovey-Caruso Emotional Intelligence Test (MSCEIT) (Kyllonen 2005)
and a recent study considered using survey item response rates (Hitt, Trivitt et al. 2014).
However these are yet to gain widespread acceptance for measuring the effectiveness of
education policy (Heckman and Rubinstein 2001).
4. Methodology
The most recent version of PISA3 provides an opportunity to address both the current gap in
data and the standardisation issue. PISA is an international set of surveys that aims to test the
skills and knowledge of students aged 15 years across a range of countries or regions within
countries. Surveys are conducted every three years, with the most recent survey being
undertaken in 2012 across 66 distinct countries or regions. A range of information is
collected on the students, with a particular focus on standardised tests across math, reading
and science. In addition though, a range of background information is collected. In addition
to these measures of academic ability and outcomes, information is collected in PISA on
other education-related measures, some of which fall under the general heading of noncognitive ability (Heckman and Mosso 2014) or what Reeves and colleagues (2014) have
labelled drive or prudence.
Non-cognitive ability is measured most explicitly by two sets of questions – perceived selfcontrol, and perseverance. While every student answered the questions on test scores, not
everyone answered the questions on perceived self-control or perseverance. However, there is
full information on 304,974 students for the former and 308,482 students for the later. This
accounts about 63% of the total sample of students. In Appendix 3, we show that these
students are significantly different in their cognitive ability (reflected in test scores) compared
to those who didn’t reply to non-cognitive ability questions, with higher scores on average.
In terms of perceived self-control, students were asked about the extent to which six specific
characteristics described them well. Specifically, they were asked about whether in their
view:
•
•
•
•
•
•
They can succeed with enough effort;
It is their choice they will be good at school;
There are problems that prevent them from putting effort into school;
Different teachers would make them try harder;
They could perform well if they wanted; and
They would perform poorly at school regardless of their effort.
These measures are broadly related to the concept of self-efficacy. According to Julio Garcia
and Geoffrey Cohen, the concept of self-efficacy has significant explanatory power with
regards to differences in education outcomes. Children of equal ability will respond
differently when challenged because;
‘…those who doubt their ability to succeed in school or students who believe that
their level of intelligence is a fixed quality, are more likely than their peers to give
3
http://pisa2012.acer.edu.au/downloads.php
DRAFT – FOR COMMENT, NOT QUOTATION
up, persevere in ineffective strategies, experience negative emotions, and fail to
return to their original performance level following failure’ (Garcia and Cohen
2012)
Feeling that you have control over your environment is one thing. But exercising that control
requires sustained effort. In terms of perseverance, students in PISA were asked whether:
•
•
•
•
•
They give up easily;
They put off difficult problems;
They remain interested in tasks that they start;
They continue to work on tasks until perfection; and
When confronted with a problem they do more than what is expected of them.
Questionnaires such as these are complicated by their use of measures of self-perception,
rather than actual character traits. They are also only able to capture a limited number of
aspects of non-cognitive ability. Also, as the PISA questionnaire is administered as part of a
cross-sectional survey it can only tell us what is associated with non-cognitive ability, not
what it predicts. Nonetheless, they are useful proxies for the broad set of skills discussed
earlier.
For our analysis, we calculate an index of perceived self-control and perseverance that across
the entire PISA sample had a mean of zero and standard deviation of one. We followed a
four-step process separately for the six questions on perceived self-control, and the five
questions on perseverance. That is, we:
•
•
•
•
Undertake Principal Components Analysis (PCA) on the student responses;
Calculate a preliminary index value for each individual based on the scoring
coefficients;
Calculate a mean and standard deviation for the whole sample using that preliminary
index value; and
For each individual, subtract the sample mean from each individual’s preliminary
index value and divide it by the sample standard deviation.
In order to make comparisons with standard measures of academic ability, we took an
average of each student’s maths, reading and science scores. We then scaled in a similar way
by subtracting the sample mean and dividing it by the standard deviation.
In the first section of results (Section 5), we compare the measures of non-cognitive ability by
country or region across the entire PISA sample. Given issues of language and culture, we
then focus our comparisons across seven English speaking, developed countries – Australia,
Canada, Ireland, New Zealand, Scotland, the UK (excluding Scotland) and the USA. The
second section of results in the paper looks at the factors associated with the indexes of test
scores and non-cognitive ability. Analysis is conducted first using a pooled sample across the
seven English-speaking countries mentioned above (with a dummy variable for six of the
seven countries). We then estimate the models separately for each of the seven countries.
5.
Non-cognitive ability across the PISA sample
There is significant and substantial variation across countries and regions for the two
measures of non-cognitive ability. Keeping in mind that across the sample the variables have
DRAFT – FOR COMMENT, NOT QUOTATION
a mean of zero and a standard deviation of one, average perceived self-control ranges from a
value of 0.48 in Costa Rica (highest non-cognitive ability on this measure) to -0.63 in MacaoChina (lowest ability). Perseverance, on the other hand, ranges from 0.59 in Kazakhstan to 0.81 in Japan.
The next few figures summarise the spread across the rest of the countries. The first figure for
each of the variables is for those countries with an average above the sample mean. The
second figure for each of the variables is for those countries with average values below the
sample mean. The ‘whiskers’ around the bars are for the 95% confidence interval.
DRAFT – FOR COMMENT, NOT QUOTATION
Figure 1
Countries with above average perceived self-control
Figure 2
Countries with below average perceived self-control
DRAFT – FOR COMMENT, NOT QUOTATION
Figure 3
Countries with above average perseverance
Figure 4
Countries with below average perseverance
In addition to the very large spread across countries, the figures show that those countries that
tend to do well on standardised test scores don’t necessarily do well on measures of noncognitive ability. Indeed, at the aggregate level, there is a negative correlation between test
scores and the two measures (-0.17 for test scores and self-control and -0.51 for test scores
and perseverance). This implies that countries with high average test scores have low average
values for non-cognitive ability. This is a troubling finding for the measurement of noncognitive ability using such questionnaires. Given the research linked above, we would
expect to find a positive correlation.
There could be three reasons why this might be the case, all of which point to caution in how
measures of non-cognitive ability should be used. The first issue is selection effect – those
students who are doing well in test scores may be from relatively rich countries but are not
answering the questions on non-cognitive items. Interestingly, the data allows us to test this.
DRAFT – FOR COMMENT, NOT QUOTATION
Comparing mean outcomes of test scores shows that those students with missing noncognitive data have significantly lower test scores than their counterparts (see Appendix 3).
Therefore, we can’t conclude that, from a simple correlation observed, countries with high
average test scores would have low average values for non-cognitive ability for the entire
population.
Another issue could be aggregation. The research summarised earlier looked at variation
within countries. It may not be that non-cognitive ability predicts how well a country is
doing, but how a person is positioned within a country. This is borne out by the data. When
we looked at correlations across individuals, test scores were positively correlated with
perceived self-control (with a value of 0.20) and to a lesser extent perseverance (0.05).
A third issue in comparing across countries relates to language and cultural norms. Given the
list of countries in the figures above, the questionnaires for PISA need to be translated into
dozens of languages, with the potential for these language differences to impact on measured
variation. There are also cultural norms that need to be taken into account, with the survey
respondents taking into account, at least in part, what they think the right answer should be.
We explore these issues further in the next section by looking at the individual level data
within the seven English-speaking countries.
6.
Variation in non-cognitive ability within countries
We begin our analysis of individual data by analysing a pooled sample of students across the
seven English-speaking, developed countries presented in Figure 5. Results are presented in
full in Appendix 1, but are summarised in Figure 5 based on a linear regression analysis
taking into account weights and sample design. Results are presented as the difference in
predicted index values whilst holding other observed characteristics constant. Appendix 2
replicates the analysis within each of the seven countries separately.
Most of the 16 explanatory variables summarised in Figure 5 are binary variables and are
reasonably self-explanatory. Of the other two, the variable for parental education is measured
as the number of years of education for the parent of the child that has the highest level of
schooling. We present the association with an additional year of schooling. The final variable
in the figure is an index of wealth of the student’s household created by the OECD, based on
a range of questions on household possessions. We standardised this index to have a mean of
zero and a standard deviation of one. Results presented in the figure for this variable
represent the association with a one standard deviation increase in the index value. In
essence, for all variables, bars to the right are those where a person with that particular
characteristic has better outcomes than those in brackets, bars to the left are where that person
has worse outcomes.
Summary statistics for the three analyses are given below. What they show, and this needs to
be kept in mind when interpreting results for individual coefficients, is that the standard
demographic, early childhood, and family background characteristics explain a lot less of the
variation in measures of non-cognitive ability compared to variation in test scores.
•
Test scores – Sample size 53,439, R-Squared 0.1423
•
Perceived self-control – Sample size 34,198, R-Squared 0.0212
•
Perseverance – Sample size 34,565, R-Squared 0.0337
DRAFT – FOR COMMENT, NOT QUOTATION
DRAFT – FOR COMMENT, NOT QUOTATION
Figure 5.Variation in index values within seven English-speaking developed countries.
female
born overseas
speaks other than English
attended preschool
parent's schooling
mother employed part-time
mother not emplyoed
father employed part-time
father not employed
wealth index
-.3
-.2
-.1
0
.1
.2
Associated change in index values
Test
Persevere
.3
Control
Canada
Ireland
New Zealand
UK
Scotland
USA
-.5
-.4
-.3
-.2
-.1
0
.1
Associated change in index values
Test
Persevere
.2
.3
Control
Looking at the first section of the table, there were a number of individual and family-level
variables that had a similar association with the two measures of non-cognitive ability as with
the standardised test scores. In the seven English-speaking, developed countries, being born
overseas, having relatively well-educated parents and living in a relatively wealthy household
was associated with higher standardised test scores and higher measured non-cognitive
DRAFT – FOR COMMENT, NOT QUOTATION
ability. Having a father who was employed part-time (compared to having a father who was
employed full-time) was associated with lower values.
What is more interesting from a policy point of view though was those variables where the
association differed, either in terms of direction or statistical significance. Females, for
example, had higher standardised test scores and perceived self-control, but lower
perseverance. This pattern was also found for those who attended preschool (early childhood
education before the first year of full-time schooling), with the reverse found for those who
speak a language other than English at home (lower test scores and self-control, but higher
perseverance). The size of the associations were not as large, but low maternal employment
(not employed or part-time employed) was found to be positively associated with test scores,
negatively associated with perceived self-control, and not associated with perseverance.
There were some differences in the country groupings once individual and family-level
variables had been controlled for. Australia was included as the base case for no other reason
than alphabetical and the fact that the authors were based at Australian universities. The more
detailed modelling (figure 5) showed that Canada has higher test scores and measured noncognitive ability than Australia, whereas the US has lower test scores alongside the highest
perceived control and perseverance. Interestingly though, the US was found to have higher
test scores than the United Kingdom (including and excluding Scotland) once individual and
other characteristics have been controlled for. The results for Ireland also changed somewhat,
with higher measured test scores and non-cognitive ability once observable characteristics
were held constant.
7.
Discussion and concluding comments
Angrist and Pischke (2010) have argued that ‘empirical microeconomics has experienced a
credibility revolution, with a consequent increase in policy relevance and scientific impact’
and that ‘a clear-eyed focus on research design is at the heart of’ this revolution. If it hadn’t
before, economics has clearly embraced experiments with a justifiable focus on causal
inference. We are in strong support of this statement. There is, however, still a need for
robust descriptive analysis that starts with associations in order to point the way towards
carefully designed evaluations and theoretical innovations.
Non-cognitive ability has the capacity to improve outcomes for all students over the lifespan
by improving school completion rates and labour market outcomes. This is particularly
important for those students who are already disadvantaged. These skills are also of
importance in regard to equality of opportunity as there appears to be an achievement gap
between wealthy and disadvantaged students. It is reasonably easy for policy makers to
appreciate the role that literacy, numeracy and science knowledge will have on future
outcomes and productivity. There is equally compelling evidence though that non-cognitive
ability (also often labelled as character skills) strongly predicts future labour market,
education and life success (Heckman and Kautz 2013).
The aim of this paper was twofold. First, we use a PCA to develop an index for measures of
non-cognitive abilities from a range of questions asked on how students perceive their
perseverance and self-control. Second, we explore the variation across countries, and across
individuals within selected countries in three measures – standardised test scores (averaged
across Mathematics, Reading and Science), perceived self-control, and perseverance
DRAFT – FOR COMMENT, NOT QUOTATION
A number of important, policy relevant findings emerge. First, there is non-random, nonresponse to the questions on self-control and perseverance with those who did not answer the
question having lower test scores than those who did. Cross-country analysis of noncognitive ability needs to reflect this. Second, for those who did answer the questions, there is
a negative cross-country correlation between test scores and non-cognitive ability. There are
potential cross-cultural and language reasons for this. However, the important point is that we
cannot assume that education policy frameworks directed at reading, Mathematics and
science will also optimise non-cognitive ability.
Within seven English-speaking countries where cross-cultural and language issues are
minimised (if not eliminated) there are important patterns in the variation of test scores and
non-cognitive ability. Sex had a complex association with females doing relatively well in
self-control and test scores, but relatively poorly in terms of perseverance. Many of the
variables that we know are associated with test scores or measures of cognitive ability are
also associated with measures of non-cognitive ability. In particular, both preschool
attendance and parental education have a strong association with at least one of the measures
of non-cognitive ability.
We control for these student level characteristics in our analysis of variation across countries
and find significant variation across the three measures that, importantly, do not necessarily
move in the same direction. Canada and Ireland are the two countries in our analysis that do
relatively well on all three variables. New Zealand does relatively poorly on all three
measures. Australia does relatively well on test scores, but relatively poorly on the two
measures of non-cognitive ability. The UK (excluding and including Scotland) do poorly on
test scores but moderately on non-cognitive ability. Finally, the US does quite poorly on test
scores but very well on measures of non-cognitive ability. There are no large-scale
evaluations that we know of that would point to specific policies that are driving these
differences. But, the descriptive analysis would suggest a need for such research.
Ultimately, as economists and other social scientists increasingly recognise the importance of
non-cognitive ability for future labour market and broader outcomes, surveys like PISA will
help form the evidence-base for within- and across-country comparisons. At the same time,
survey items included in PISA or similar will hopefully be used in a greater range of trials
and evaluations. The analysis in this paper has shown that these variables have important
information in them, but cannot be used unquestioningly.
DRAFT – FOR COMMENT, NOT QUOTATION
5. Bibliography
Barón, J. D. and D. A. Cobb-Clark (2010). "Are young people's educational outcomes linked
to their sense of control?" Available at SSRN 1594792.
Black, S. E. and P. J. Devereux (2011). "Recent developments in intergenerational mobility."
Handbook of labor economics 4: 1487-1541.
Borghans, L., A. L. Duckworth, J. J. Heckman and B. Ter Weel (2008). "The economics and
psychology of personality traits." Journal of Human Resources 43(4): 972-1059.
Carneiro, P., J. J. Heckman and E. J. Vytlacil (2010). Estimating marginal returns to
education, National Bureau of Economic Research.
Cobb-Clark, D. A. and M. Tan (2011). "Noncognitive skills, occupational attainment, and
relative wages." Labour Economics 18(1): 1-13.
Coleman, M. and T. DeLeire (2003). "An economic model of locus of control and the human
capital investment decision." Journal of Human Resources 38(3): 701-721.
Dee, T. S. and B. Jacob (2011). "The impact of No Child Left Behind on student
achievement." Journal of Policy Analysis and Management 30(3): 418-446.
Department of Education (2014). Review of the Australian Curriculum Final Report
Canberra.
Duckworth, A. L., C. Peterson, M. D. Matthews and D. R. Kelly (2007). "Grit: perseverance
and passion for long-term goals." Journal of personality and social psychology 92(6): 1087.
Duckworth, A. L., P. D. Quinn, D. R. Lynam, R. Loeber and M. Stouthamer-Loeber (2011).
"Role of test motivation in intelligence testing." Proceedings of the National Academy of
Sciences 108(19): 7716-7720.
Duckworth, A. L., P. D. Quinn and E. Tsukayama (2012). "WhatNo Child Left Behind leaves
behind: The roles of IQ and self-control in predicting standardized achievement test scores
and report card grades." Journal of educational psychology 104(2): 439.
Duckworth, A. L. and M. E. Seligman (2005). "Self-discipline outdoes IQ in predicting
academic performance of adolescents." Psychological science 16(12): 939-944.
Garcia, J. and G. Cohen (2012). A social psychological approach to educational intervention.
Behavioral foundations of public policy. E. Shafir. Princeton, Princeton University Press:
329-347.
Gatz, M. and M. J. Karel (1993). "Individual change in perceived control over 20 years."
International Journal of Behavioral Development 16(2): 305-322.
Heckman, J. J. (2000). "Policies to foster human capital." Research in economics 54(1): 3-56.
Heckman, J. J. (2006). "Skill formation and the economics of investing in disadvantaged
children." Science 312(5782): 1900-1902.
Heckman, J. J. (2008). "Schools, skills, and synapses." Economic inquiry 46(3): 289-324.
Heckman, J. J. and T. Kautz (2013). Fostering and measuring skills: Interventions that
improve character and cognition, National Bureau of Economic Research.
Heckman, J. J. and D. V. Masterov (2007). "The productivity argument for investing in
young children." Applied Economic Perspectives and Policy 29(3): 446-493.
DRAFT – FOR COMMENT, NOT QUOTATION
Heckman, J. J., S. H. Moon, R. Pinto, P. A. Savelyev and A. Yavitz (2010). "The rate of
return to the HighScope Perry Preschool Program." Journal of Public Economics 94(1): 114128.
Heckman, J. J. and S. Mosso (2014). The Economics of Human Development and Social
Mobility, National Bureau of Economic Research.
Heckman, J. J. and Y. Rubinstein (2001). "The importance of noncognitive skills: Lessons
from the GED testing program." American Economic Review: 145-149.
Hitt, C., J. Trivitt and A. Cheng (2014). "When You Say Nothing at All: The Surprisingly
Predictive Power of Student Effort on Surveys."
Hoover, E. (2013). "Colleges Seek 'Noncognitive' Measures of Applicants; Admissions
offices want to know about traits, like leadership, initiative, and grit, that the SAT doesn't
test." The Chronicle of Higher Education 59(19).
Kyllonen, P. C. (2005). The Case for Noncognitive Assessments. R&D Connections. E. T. S.
R. a. Development. New Jersey, Princeton.
Mischel, W. (2014). The Marshmallow Test: Mastering Self-Control, Little, Brown and
Company
Mischel, W., Y. Shoda and M. I. Rodriguez (1989). "Delay of gratification in children."
Science 244(4907): 933-938.
Psacharopoulos, G. and H. A. Patrinos* (2004). "Returns to investment in education: a
further update." Education economics 12(2): 111-134.
Reeves, R., J. Venator and K. Howard (2014). The Character Factor: Measures and Impact of
Drive and Prudence. C. o. C. a. Familes, Brookings Institute.
Sammons, P., K. Sylva, E. Melhuish, I. Siraj, B. Taggart, R. Smees and K. Toth (2014). "The
Effective Pre-school, Primary and Secondary Education Project (EPPSE 3-16+) Students’
views of school in Key Stage 4 at age 16." Institute of Education, University of
London/Department for Education http://www. ioe. ac. uk/research/153. html.
Tough, P. (2012). How Children Succeed: Grit, Curiosity, and the Hidden Power of
Character, Houghton Mifflin Harcourt.
West, M. R., M. A. Kraft, A. S. Finn, R. Martin, A. L. Duckworth, C. F. Gabrieli and J. D.
Gabrieli (2014). "Promise and Paradox: Measuring Students’ Non-cognitive Skills and the
Impact of Schooling."
DRAFT – FOR COMMENT, NOT QUOTATION
Appendix 1: full results for comparisons across English speaking countries only: all sample
VARIABLES
Female
Born Overseas
Speaks other than English
Grade level of student
Attended Preschool
Age started school
Parent's level of schooling
Mother employed part-time
(ref: full time)
Mother not employed
Father employed part-time
Father not employed
Wealth (standard deviation)
Canada (ref: Australia)
Ireland
New Zealand
UK (excluding Scotland)
Scotland
USA
Constant
Observations
R-squared
(1)
test scores
(2)
Self-control
(3)
perseverance
0.02**
(0.010)
0.11***
(0.022)
-0.14***
(0.016)
0.37***
(0.014)
0.18***
(0.023)
0.02***
(0.008)
0.07***
(0.003)
0.08***
0.11***
(0.012)
0.05
(0.029)
-0.02
(0.021)
0.07***
(0.017)
0.17***
(0.063)
0.00
(0.011)
0.03***
(0.003)
-0.02
-0.05***
(0.013)
0.14***
(0.028)
0.18***
(0.027)
0.06***
(0.013)
-0.03
(0.039)
-0.03**
(0.013)
0.03***
(0.004)
0.02
(0.013)
0.03**
(0.014)
-0.18***
(0.018)
-0.14***
(0.016)
0.10***
(0.008)
0.10***
(0.011)
0.24***
(0.015)
-0.26***
(0.020)
-0.43***
(0.022)
-0.46***
(0.024)
-0.24***
(0.016)
-4.52***
(0.143)
53,439
0.142
(0.026)
-0.05**
(0.019)
-0.11***
(0.027)
-0.10***
(0.026)
0.06***
(0.012)
0.10***
(0.015)
0.05**
(0.021)
-0.02
(0.021)
0.02
(0.020)
0.06**
(0.024)
0.20***
(0.018)
-1.27***
(0.196)
34,198
0.021
(0.022)
0.01
(0.015)
-0.15***
(0.033)
-0.00
(0.022)
0.09***
(0.009)
0.07***
(0.012)
0.07***
(0.017)
-0.13***
(0.017)
-0.02
(0.016)
-0.02
(0.020)
0.28***
(0.013)
-0.99***
(0.167)
34,565
0.034
DRAFT – FOR COMMENT, NOT QUOTATION
Appendix 2: full results for each English speaking country:
2.1 Australia
VARIABLES
Female
Born Overseas
Speaks other than English
Grade level of student
Attended Preschool
Age started school
Parent's level of schooling
Mother employed part-time
Mother not employed
Father employed part-time
Father not employed
Wealth-standard deviation
Constant
Observations
R-squared
(1)
Test scores
(2)
Self-control
(3)
perseverance
-0.01
(0.014)
0.02
(0.017)
0.15***
(0.023)
0.25***
(0.013)
0.30***
(0.028)
-0.09***
(0.007)
0.13***
(0.003)
0.15***
(0.010)
0.02**
(0.012)
-0.06***
(0.020)
-0.18***
(0.016)
-0.05***
(0.007)
-3.61***
(0.141)
0.01
(0.014)
0.03
(0.019)
0.25***
(0.024)
0.01
(0.013)
0.08**
(0.039)
-0.05***
(0.010)
0.05***
(0.004)
0.05***
(0.017)
-0.00
(0.017)
-0.09***
(0.026)
-0.11***
(0.025)
0.01
(0.010)
-0.70***
(0.167)
-0.16***
(0.011)
0.19***
(0.020)
0.25***
(0.018)
-0.02
(0.013)
0.14***
(0.028)
-0.03***
(0.009)
0.06***
(0.004)
0.05***
(0.016)
0.10***
(0.014)
-0.08***
(0.023)
-0.10***
(0.024)
0.04***
(0.011)
-0.65***
(0.163)
11,530
0.119
7,424
0.020
7,611
0.037
DRAFT – FOR COMMENT, NOT QUOTATION
2.2 Canada
VARIABLES
Female
Born Overseas
Speaks other than English
Grade level of student
Attended Preschool
Age started school
Parent's level of schooling
Mother employed part-time
Mother not employed
Father employed part-time
Father not employed
Wealth-standard deviation
Constant
Observations
R-squared
(1)
Test scores
(2)
Self-control
(3)
Perseverance
0.03***
(0.009)
0.01
(0.022)
0.00
(0.018)
0.41***
(0.012)
0.17***
(0.014)
0.05***
(0.007)
0.06***
(0.002)
0.09***
(0.014)
-0.01
(0.011)
-0.08***
(0.023)
-0.02
(0.022)
-0.04***
(0.008)
-4.79***
(0.117)
0.06***
(0.014)
0.12***
(0.027)
-0.03
(0.021)
-0.00
(0.019)
0.09***
(0.029)
-0.04***
(0.013)
0.03***
(0.004)
0.02
(0.019)
0.01
(0.019)
-0.09***
(0.030)
0.00
(0.027)
0.04***
(0.010)
-0.19
(0.232)
-0.04**
(0.015)
0.13***
(0.020)
0.18***
(0.022)
0.14***
(0.020)
0.07**
(0.025)
-0.02*
(0.011)
0.05***
(0.002)
-0.01
(0.021)
0.02
(0.019)
-0.04
(0.037)
0.04
(0.027)
0.04***
(0.010)
-2.08***
(0.212)
15,976
0.097
10,019
0.010
10,026
0.033
DRAFT – FOR COMMENT, NOT QUOTATION
2.3 Ireland
VARIABLES
Female
Born Overseas
Speaks other than English
Grade level of student
Attended Preschool
Age started school
Parent's level of schooling
Mother employed part-time
Mother not employed
Father employed part-time
Father not employed
Wealth-standard deviation
Constant
Observations
R-squared
(1)
Test scores
(2)
Self-control
(3)
Perseverance
0.00
(0.017)
0.20***
(0.022)
-0.16***
(0.046)
0.08***
(0.011)
0.04
(0.024)
-0.09***
(0.014)
0.08***
(0.003)
0.10***
(0.017)
-0.03*
(0.015)
-0.25***
(0.027)
-0.18***
(0.016)
0.02
(0.011)
-0.98***
(0.157)
0.06***
(0.019)
0.03
(0.036)
-0.02
(0.063)
-0.01
(0.015)
-0.02
(0.024)
-0.03*
(0.017)
0.03***
(0.004)
-0.00
(0.024)
0.06***
(0.021)
-0.06
(0.035)
-0.17***
(0.029)
0.02
(0.015)
-0.25
(0.197)
-0.12***
(0.020)
0.08**
(0.034)
-0.02
(0.071)
0.04***
(0.014)
-0.08**
(0.031)
0.01
(0.016)
0.04***
(0.004)
-0.05*
(0.025)
0.01
(0.022)
-0.18***
(0.033)
-0.04
(0.026)
0.02
(0.014)
-0.88***
(0.188)
4,174
0.096
2,741
0.014
2,736
0.020
DRAFT – FOR COMMENT, NOT QUOTATION
2.4 New Zealand
VARIABLES
Female
Born Overseas
Speaks other than English
Grade level of student
Attended Preschool
Age started school
Parent's level of schooling
Mother employed part-time
Mother not employed
Father employed part-time
Father not employed
Wealth-standard deviation
Constant
Observations
R-squared
(1)
Test scores
(2)
Self-control
(3)
Perseverance
-0.03
(0.021)
0.20***
(0.024)
-0.39***
(0.035)
0.41***
(0.024)
0.32***
(0.046)
-0.11***
(0.025)
0.11***
(0.005)
0.14***
(0.019)
0.00
(0.021)
-0.13***
(0.034)
-0.18***
(0.029)
0.08***
(0.013)
-5.15***
(0.338)
0.06**
(0.022)
0.08***
(0.028)
-0.03
(0.039)
0.07**
(0.031)
0.13***
(0.036)
-0.08***
(0.023)
0.03***
(0.006)
-0.01
(0.026)
-0.06*
(0.031)
-0.13***
(0.040)
-0.10**
(0.045)
0.05***
(0.017)
-0.83**
(0.345)
-0.12***
(0.020)
0.20***
(0.025)
0.31***
(0.032)
0.02
(0.032)
-0.01
(0.038)
-0.06***
(0.020)
0.04***
(0.007)
0.04
(0.026)
-0.14***
(0.025)
-0.08*
(0.042)
0.12***
(0.037)
0.03**
(0.017)
-0.59
(0.392)
3,407
0.138
2,169
0.017
2,226
0.049
DRAFT – FOR COMMENT, NOT QUOTATION
2.5 Scotland
VARIABLES
Female
Born Overseas
Speaks other than English
Grade level of student
Attended Preschool
Age started school
Parent's level of schooling
Mother employed part-time
Mother not employed
Father employed part-time
Father not employed
Wealth-standard deviation
Constant
Observations
R-squared
(1)
Test scores
(2)
Self-control
(3)
Perseverance
-0.02
(0.016)
0.27***
(0.034)
0.04
(0.073)
0.27***
(0.021)
0.32***
(0.060)
-0.17***
(0.016)
0.08***
(0.005)
0.08***
(0.023)
-0.03
(0.020)
-0.19***
(0.038)
-0.03
(0.025)
0.04***
(0.012)
-3.14***
(0.274)
-0.01
(0.024)
0.09*
(0.050)
0.22**
(0.099)
0.16***
(0.040)
0.30**
(0.116)
-0.16***
(0.022)
0.03***
(0.006)
0.08***
(0.030)
-0.02
(0.033)
-0.05
(0.057)
-0.07*
(0.042)
0.14***
(0.016)
-1.69***
(0.546)
-0.20***
(0.028)
0.17***
(0.039)
0.51***
(0.075)
0.02
(0.038)
0.28***
(0.091)
-0.09***
(0.025)
0.03***
(0.006)
0.04
(0.028)
0.07**
(0.026)
0.03
(0.048)
-0.03
(0.045)
0.12***
(0.022)
-0.46
(0.532)
2,462
0.100
1,602
0.039
1,626
0.033
DRAFT – FOR COMMENT, NOT QUOTATION
2.6 United Kingdom (excluding Scotland)
VARIABLES
Female
Born Overseas
Speaks other than English
Grade level of student
Attended Preschool
Age started school
Parent's level of schooling
Mother employed part-time
Mother not employed
Father employed part-time
Father not employed
Wealth-standard deviation
Constant
Observations
R-squared
(1)
Test scores
(2)
Self-control
(3)
Perseverance
-0.01
(0.026)
0.19***
(0.036)
0.00
(0.041)
-0.02
(0.029)
0.51***
(0.034)
0.06**
(0.022)
0.09***
(0.005)
0.17***
(0.017)
0.02
(0.022)
-0.11***
(0.031)
-0.21***
(0.020)
-0.00
(0.011)
-1.37***
(0.391)
0.02
(0.021)
-0.02
(0.045)
0.17***
(0.048)
0.01
(0.027)
0.11**
(0.045)
0.04*
(0.018)
0.04***
(0.005)
-0.05
(0.030)
-0.05
(0.032)
-0.03
(0.045)
-0.03
(0.038)
0.00
(0.014)
-0.85**
(0.338)
-0.21***
(0.019)
0.29***
(0.043)
0.21***
(0.038)
-0.00
(0.023)
0.26***
(0.042)
0.01
(0.018)
0.01**
(0.007)
0.07***
(0.026)
0.09***
(0.026)
-0.08**
(0.037)
0.05
(0.034)
0.06***
(0.011)
-0.52*
(0.286)
7,263
0.067
4,676
0.009
4,785
0.029
DRAFT – FOR COMMENT, NOT QUOTATION
2.7 USA
VARIABLES
Female
Born Overseas
Speaks other than English
Grade level of student
Attended Preschool
Age started school
Parent's level of schooling
Mother employed part-time
Mother not employed
Father employed part-time
Father not employed
Wealth-standard deviation
Constant
Observations
R-squared
(1)
Test scores
(2)
Self-control
(3)
Perseverance
0.02*
(0.013)
0.10***
(0.030)
-0.18***
(0.019)
0.39***
(0.016)
0.04
(0.052)
0.02**
(0.010)
0.06***
(0.003)
0.05**
(0.018)
0.04**
(0.018)
-0.20***
(0.023)
-0.14***
(0.020)
0.13***
(0.011)
-4.78***
(0.166)
0.14***
(0.016)
0.04
(0.042)
-0.05*
(0.027)
0.07***
(0.019)
0.30**
(0.137)
0.00
(0.014)
0.02***
(0.004)
-0.02
(0.039)
-0.06**
(0.024)
-0.12***
(0.035)
-0.11***
(0.031)
0.07***
(0.016)
-1.24***
(0.258)
-0.02
(0.017)
0.11***
(0.039)
0.17***
(0.035)
0.06***
(0.015)
-0.17**
(0.083)
-0.03**
(0.016)
0.03***
(0.005)
0.01
(0.031)
-0.00
(0.019)
-0.17***
(0.043)
-0.01
(0.027)
0.10***
(0.012)
-0.53**
(0.217)
8,627
0.156
5,567
0.023
5,555
0.025
DRAFT – FOR COMMENT, NOT QUOTATION
.05
0
-.05
0
is
si
m
no
nm
is
si
ng
ng
is
si
m
ng
Pe
rs
ev
er
an
ce
se
lf co
nt
ro
l
l
f-c
on
tro
se
l
ng
is
si
N
on
-m
Pe
rs
ev
er
an
ce
-.1
-.05
-.1
mean value test scores index
.05
Appendix 3: Mean value of test scores by missing value status in non-cognitive abilities for
English Speaking countries
.4
.3
.2
0
.1
-.3 -.2 -.1
Associated change in index values
.5
Appendix 4 unconditional comparisons across English speaking countries
Australia
Canada
Ireland New Zealand
Test scores
Perseverance
UK
Scotland
Self-Control
USA