The role of noncognitive skills
in academic success
David Payne
Vice President and
Chief Operating Officer
Higher Education Division
Patrick Kyllonen
Sr. Research Director
Center for Academic &
Workplace Readiness &
Success
R&D Division
21st Century Knowledge and Skills: The New
Curriculum and the Future of Assessment
Center for Enrollment Research, Policy, and Practice (CERPP)
Marriott Downtown, Los Angeles, CA
January 11-13, 2012
Confidential and Proprietary. Copyright © 2010 Educational Testing Service. All rights reserved.
1/18/2012
Overview of ETS’s
Commitment to the
Measurement of Noncognitive
Attributes
2
Confidential and Proprietary. Copyright © 2010 Educational Testing Service. All rights reserved.
Assessing Noncognitive Attributes
1. Areas of use in higher education
– Graduate and professional Admissions
– Undergraduate Admissions
– Workforce
2. Implementation issues
– Leading change in well-established
processes
– Concerns about unintended consequences
– Demands on evaluators
3
Confidential and Proprietary. Copyright © 2010 Educational Testing Service. All rights reserved.
1/18/2012
Overview of the Rest of the Talk
• What does “noncognitive skills” mean?
– Economics, Psychology, “21st Century Knowledge &
Skills”
• Why do we think these are important?
– Leaders (education, workforce) say so
– Prediction studies (education, labor market) show
• Do they change? Or are they fixed?
– Intervention studies (K-12) show they can be developed
• Why should they be assessed?
– Admissions, skill development, outcomes
• How do we measure them?
– Self report, others’ ratings, situational judgment tests
4
Confidential and Proprietary. Copyright © 2010 Educational Testing Service. All rights reserved.
WHAT DOES
“NONCOGNITIVE SKILLS”
MEAN?
5
Confidential and Proprietary. Copyright © 2010 Educational Testing Service. All rights reserved.
1/18/2012
Economics perspective
• Schooling produces both “cognitive skills,” measured by test
scores, and “noncognitive skills,” unmeasured
• Years of schooling predicts labor market outcomes—cognitive
skills account for only 20%; therefore 80% of the “years of
schooling” benefit is due to noncognitive skills (Bowles, Gintis, &
Osborne, 2001)
• Early childhood interventions do not raise IQ, but improve
noncognitive skills which affect education, employment,
earnings, and crime (Heckman et al., 2010)
• GED holders have equal cognitive skills as HS grads, but have
worse labor market outcomes, commit more crime (Heckman &
Rubinstein, 2001)
6
Confidential and Proprietary. Copyright © 2010 Educational Testing Service. All rights reserved.
1/18/2012
2010
2011
2012
Deeper Learning and 21st Century Skills, Washington, DC:
National Academy of Sciences / National Research Council.
7
Confidential and Proprietary. Copyright © 2010 Educational Testing Service. All rights reserved.
“21st Century Skills”
Cognitive
• (Non-routine) problem
solving
• Critical thinking
• Systems thinking
• Study skills
• Adaptability
• Creativity
• Meta-cognitive skills
Interpersonal
(social)
• Complex
communication
• Emotional/ social
intelligence
• Teamwork/
Collaboration
• Leadership
• Cultural sensitivity
• Tolerance for diversity
Confidential and Proprietary. Copyright © 2010 Educational Testing Service. All rights reserved.
Intrapersonal
(emotional, selfregulatory)
•
•
•
•
•
•
•
•
•
•
Anxiety
Self-efficacy
Self-concept
Attributions
Work Ethic
Persistence
Organization
Time management
Ethics & Integrity
Life-long learning
Why do we think noncognitive attributes are important?
LEADERS SAY
NONCOGNITIVE ATTRIBUTES
ARE IMPORTANT
9
Confidential and Proprietary. Copyright © 2010 Educational Testing Service. All rights reserved.
1/18/2012
Percentage Employers Rating Skill as “Very Important”
Oral Communications
Teamwork/Collaboration
Professionalism/Work Ethic
Written Communications
Critical Thinking/Problem Solving
Ethics/Social Responsibility
Leadership
Information Technology
Creativity/Innovation
Lifelong Learning/Self direction
Diversity
English Language
Mathematics
Science
4-year college High School
graduates
Graduates
95%
70%
94
75
94
80
93
53
92
58
86
63
82
29
81
53
81
36
78
43
72
52
88
62
64
30
33
9
Conference Board et al (2008)
10
Confidential and Proprietary. Copyright © 2010 Educational Testing Service. All rights reserved.
1/18/2012
Taxonomy of Valued Academic Skills:
Mission Statements from 35 Universities
(College Board/Michigan State University)1,2
I. Knowledge, learning, mastery of
general principles
II. Continuous learning, intellectual
interest and curiosity
III. Artistic cultural appreciation and
curiosity
IV. Multicultural tolerance and
appreciation
V. Leadership
VI. Interpersonal skills
VII. Social responsibility, citizenship
and involvement
VIII. Physical and psychological
health
IX. Career orientation
X. Adaptability and life skills
XI. Perseverance
XII. Ethics and integrity
Schmitt, Oswald, Kim, Imus, Drzakowski, Friede, & Shivpuri (2007).
2 Oswald, Schmitt, Kim, Ramsay, & Gillespie (2004).
1
Confidential and Proprietary. Copyright © 2010 Educational Testing Service. All rights reserved.
Why do we think noncognitive attributes are important?
PREDICTION STUDIES OF
EDUCATION & LABOR
MARKET OUTCOMES
12
Confidential and Proprietary. Copyright © 2010 Educational Testing Service. All rights reserved.
1/18/2012
Noncognitive predictions of school grades1
by school level
correlation
K-8
High School
College
Conscientiousness
.28
.21
.23
Openness
.24
.12
.07
Agreeableness
.30
.05
.06
Emotional Stability
.20
.01
-.01
Extraversion
.18
-.01
-.03
Cognitive ability
.58
.24
.23
Socioeconomic Status
.32
.32
.32
1
Confidential and Proprietary. Copyright © 2010 Educational Testing Service. All rights reserved.
Poropat, A. (2009). Psychological Bulletin.
“Using <NELS data> …. The main finding is that
eighth grade misbehavior1 [based on 8th grade
teacher evaluations] is important for earnings
over and above eighth grade test scores.
Moreover, controlling for educational
attainment, childhood misbehavior is associated
with earnings at all educational levels. By
contrast, achievement test scores are only
related to earnings for young men with
postsecondary degrees.”
Segal, Carmit (2011). Misbehavior, Education, and Labor Market Outcomes.
(Stanford dissertation) www.econ.upf.edu/~segal/MisbehaviorJan2011.pdf .
1absenteeism,
Confidential and Proprietary. Copyright © 2010 Educational Testing Service. All rights reserved.
14
disruptiveness, inattentiveness, tardiness,
and homework completion
1/18/2012
Noncognitive skills measured at
age 18 predict labor market
outcomes for 30+ year olds
• Data
(Swedish military enlistment data)
– Noncognitive: 30 min clinical interview to measure responsibility,
independence, outgoingness, persistence, emotional stability, initiative
– Cognitive: synonyms, induction, metal folding, technical comprehension
– Outcomes: wages, educational attainment, experience, family status, etc.
• Participants
– 14,703 18-19 yo Swedish enlistees , examined at age 32 – 41 yo
• Findings
– Noncognitive > cognitive predictions; a change in 1 SD in the predictor = …
• Changes in wages: 9% vs. 5% (d = .33 vs. .15)
• Changes in employment status: 3.3% vs. 1.1%
• Changes in probability (annual earnings < 10th percentile): 4.7% vs. .2%
(when noncognitive controlled for) (1.5% when noncognitive not
controlled for)
Lindqvist, Erik, & Vestman, Roine (2011). The Labor Market Returns to Cognitive and Noncognitive Ability: Evidence from the Swedish
Confidential and Proprietary. Copyright © 2010 Educational Testing Service. All rights reserved.
Enlistment. American Economic Journal: Applied Economics, 3 (1), 101-128.
NON-COGNITIVE SKILLS CAN
BE IMPROVED
16
Confidential and Proprietary. Copyright © 2010 Educational Testing Service. All rights reserved.
1/18/2012
Social Emotional Learning
(SEL)
• “the process of acquiring core
competencies to
–
–
–
–
–
–
recognize and manage emotions,
set and achieve positive goals,
appreciate the perspectives of others,
establish and maintain positive relationships,
make responsible decisions, and
handle interpersonal situations constructively”
Durlak, Weissberg, Dymnicki, Taylor, & Schellinger (2011)
17
Confidential and Proprietary. Copyright © 2010 Educational Testing Service. All rights reserved.
1/18/2012
Social Emotional Learning
(SEL)
•
•
18
Findings (N = 270,034 K-12 students) from treatment vs. control
studies (Durlak et al., 2011)
SEL treatment resulted in improvements in…
–
Effect size
– social and emotional skills
.57
– attitudes
.23
– positive social behavior
.24
– conduct problems
.22
– reducing emotional stress
.24
– academic performance
.27
Confidential and Proprietary. Copyright © 2010 Educational Testing Service. All rights reserved.
1/18/2012
Confidential and Proprietary. Copyright © 2010 Educational Testing Service. All rights reserved.
20
Confidential and Proprietary. Copyright © 2010 Educational Testing Service. All rights reserved.
HOW CAN NON-COGNITIVE
DATA BE USED?
21
Confidential and Proprietary. Copyright © 2010 Educational Testing Service. All rights reserved.
1/18/2012
Uses of Noncognitive Data
• Admissions
– Noncognitive skills can supplement grades and
standardized test scores for admissions decisions,
increases validity, less adverse impact
• Skill Development
– Students can be given feedback, resources, and action
plans to improve their noncognitive skills
• Outcomes
– Programs, interventions can be evaluated for their
noncognitive outcomes as well as cognitive outcomes
22
Confidential and Proprietary. Copyright © 2010 Educational Testing Service. All rights reserved.
1/18/2012
What are the approaches for measuring noncognitive
skills?
HOW DO WE MEASURE
THEM?
23
Confidential and Proprietary. Copyright © 2010 Educational Testing Service. All rights reserved.
1/18/2012
Approaches to measuring noncognitive
skills
• 24
Self-assessments
Objective measures
(Situational Judgment Test; Implicit Bio-data
Association Test; Conditional
Reasoning Test)
Ratings-by-others, Interviews
For high-stakes
assessments ratings-byothers are best because
they’re not fakeable
Confidential and Proprietary. Copyright © 2010 Educational Testing Service. All rights reserved.
How can we assess noncognitive
attributes?
• Self assessments
– Simple self report (subject to coaching/faking)
– Forced choice
• Objective tests
– Situational judgment tests
– Implicit association test
– Conditional reasoning test
• Others’ ratings (higher validity than self-ratings; Conneley & Ones,
2011)
– Teacher grades, ratings in K-12
– Letters of Recommendation; ETS’s Personal Potential Index (PPI) in Higher
Education
– Interviews
Confidential and Proprietary. Copyright © 2010 Educational Testing Service. All rights reserved.
25
Confidential and Proprietary. Copyright © 2010 Educational Testing Service. All rights reserved.
For each statement, evaluators rate the applicant on a
five-point scale that ranges from “Below Average” to “Truly
Exceptional (Top 1%)"
•Critical Thinking and Problem Solving
•
Generates good ideas
•
Forms opinions based on logic and facts
•
Understands different points of view
•
Knows how to apply knowledge to solve
problems
•Persistence and Resilience
• Remains calm under pressure
• Accepts feedback without getting defensive
• Can overcome challenges and setbacks
• Does not give up easily
•Motivation and Work Ethic
•
Works hard
•
Is goal-oriented
•
Shows initiative
•
Meets deadlines
•Leadership and Teamwork
• Is skilled in handling social situations
• Has a talent for influencing people
• Works well in group settings
• Gives criticism/feedback to others in a helpful
way
•Ethics and Integrity
•
Shows respect for classmates and teachers
•
Is worthy of trust from others
•
Is engaged in school or community
citizenship activities
•
Treats all people fairly
•Communication Skills
• Writes essays well
• Expresses ideas clearly in short written text
(emails, notes, memos)
• Expresses ideas clearly in oral presentations
• Effective at getting point across in
conversations and discussions
Confidential and Proprietary. Copyright © 2010 Educational Testing Service. All rights reserved.
The case for noncognitive skills assessment
SUMMARY
28
Confidential and Proprietary. Copyright © 2010 Educational Testing Service. All rights reserved.
1/18/2012
Summary
• Employers and educators say noncognitive skills are
important
– They may facilitate content skills, or be valuable in their own right
• Prediction studies show that noncognitive skills are
important
– Educational achievement, labor market outcomes
• Noncognitive skills can be assessed for admissions, and
also for skill development, and outcomes
• How we measure noncognitive skills is important
• Including noncognitive skills in admissions, outcomes, skill
development could lead to positive changes in quality and
diversity
Confidential and Proprietary. Copyright © 2010 Educational Testing Service. All rights reserved.
Discussion Questions
• How important are noncognitive
skills?
• Can they be measured reliably?
• How can they be used (e.g., college
admissions, student learning
outcomes, self help, school
monitoring)
30
Confidential and Proprietary. Copyright © 2010 Educational Testing Service. All rights reserved.
1/18/2012
Questions? Comments?
31
Confidential and Proprietary. Copyright © 2010 Educational Testing Service. All rights reserved.
References & Bibliography
•
•
•
•
•
•
•
Almlund, M., Duckworth, A. L., Heckman, J. & Kautz, T. (2011) Personality Psychology and
Economics, (January 17, 2001 version) ¨IZA Workshop: Cognitive and Non-Cognitive Skills¨, January
25-27, (Bonn, Germany). Available as:
http://iza.org/conference_files/CoNoCoSk2011/heckman_j130.pdf.
Almlund, M., Duckworth, A. L., Heckman, J. & Kautz, T. (2011) Personality Psychology and
Economics, (January 17, 2001 version) ¨IZA Workshop: Cognitive and Non-Cognitive Skills¨, January
25-27, (Bonn, Germany). Available as:
http://iza.org/conference_files/CoNoCoSk2011/heckman_j130.pdf.
Baird, L. L., & Knapp, J. E. (1981). The inventory of documented accomplishments for graduate
admissions: Results of a field trial study and its reliability, short-term correlates, and evaluation.
Princeton, NJ: Educational Testing Service. (ERIC Document Reproduction Service No. ED214940)
Borghans, L. , Duckworth, A. L., Heckman, J. J., &: ter Weel, B. (2008) “The Economics and
Psychology of Personality Traits,” The Journal of Human Resources, XLIII(4), pp. 972-1059.
Borghans,L, ter Weel, B & Weinberg, B. A. (2008) “Interpersonal Styles and Labor Market
Outcomes,” The Journal of Human Resources, XLIII(4), pp. 815-858.
Bowles, S., Gintis, H., & Osborne, M. (2001) “The Determinants of Earnings: A Behavioral Approach,”
Journal of Economic Literature,” 39(4), pp. 137-1176.
Casner-Lotto, J., & Barrington, L. (2006). Are they really ready to work? Employers’ perspectives on
the basic knowledge and applied skills of new entrants to the 21st century U.S. workforce. New
York: The Conference Board. Retrieved October 22, 2008, from http://www.conferenceboard.org/pdf_free/BED-06-Workforce.pdf
Confidential and Proprietary.
Confidential and Proprietary. Copyright © 2010 Educational Testing Service. All rights reserved.
Copyright © 2009 Educational
References & Bibliography
•
•
•
•
•
•
•
Connelly, B.S., & Ones, D.S. (in press). An other perspective on personality: Meta-analytic integration
of observers’ accuracy and predictive validity. Psychological Bulletin.
Crede, M., & Kuncel, N. R. (2008). Study habits, skills, and attitudes: The third pillar supporting
collegiate academic performance. Perspectives on Psychological Science, 3(6), 425-453.
Durlak, J.A.,, Weissberg, R.P., Dymnicki, A.B., Taylor, R.D., Schellinger, K.B. (2011). The Impact of
Enhancing Students’ Social and Emotional Learning: A Meta-Analysis of School-Based Universal
Interventions. Child Development, 82(1), 405-432.
Heckman, J. & Rubenstein, Y. (2001) “The Importance of Noncognitive Skills: Lessons from the GED
Testing Program,” American Economic Review, 91(2), pp. 145-149.
Heckman, J. Moon, S. H., Pinto, R., Savelyev, P. & Yavitz, A. (2010) ¨A New Cost-Benefit and Rate of
Return Analysis for the Perry Preschool Program: A Summary,¨ In Reynolds, A. J., Rolnick, A. J.,
Englund, M. M., & Temple, J. A., eds., Childhood Programs and Practices in the First Decade of Life
,pp. 366-380 (New York: Cambridge University Press).
Heckman, J.J., Malofeeva, L., Pinto, R. &Savelyev, P. (2010). Understanding the mechanisms through
which an influential early childhood program boosted adult outcomes. Department of Economics,
University of Chicago.
Kyllonen, P. C., Lipnevich, A. A. , Burrus, J. & Roberts, R. D. (2008) Personality, Motivation, and
College Readiness: A Prospectus for Assessment and Development (Princeton, Educational Testing
Service).
Confidential and Proprietary. Copyright © 2010 Educational Testing Service. All rights reserved.
Confidential and Proprietary.
Copyright © 2009 Educational
References & Bibliography
•
•
•
•
•
•
•
Lindqvist, E. & Vestman, R. (2011) ¨The Labor Market Returns to Cognitive and Noncognitive Ability:
Evidence from the Swedish Enlistment,¨ American Economic Journal: Applied Economics, 3, pp. 101128.
Noftle, E. E. :L& Robins, R. W. (2007) “Personality Predictors of Academic Outcomes: Big Five
Correlates of GPA and SAT Scores,” Journal of Personality and Social Psychology, 93(1), pp. 116-130.
Oswald, F. L., Schmitt, N., Kim, B. H., Ramsay, L. J., & Gillespie, M. A. (2004). Developing a biodata
measure and situational judgment inventory as predictors of college student performance. Journal
of Applied Psychology, 89, 187-207.
Roberts, B. W., Walton, K. & Viechtbauer, W. (2006). Patterns of mean-level change in personality
traits across the life course: A meta-analysis of longitudinal studies. Psychological Bulletin, 132, 125.
Sackett, P.R., Schmitt, N., Ellingson, J.E., Kabin, M. (2001). High stakes testing in employment,
credentialing, and higher education: Prospects in a post-affirmative-action world. American
Psychologist, 56(4), 302-318.
Schmitt, N., Oswald, F. L., Kim, B. H., Imus, A., Drzakowski, S., Friede, A., & Shivpuri, S. (2007). The
use of background and ability profiles to predict college student outcomes. Journal of Applied
Psychology, 92, 165-180.
Stricker, L. J., Rock, D. A., & Bennett, R. E. (2001). Sex and ethnic-group differences on
accomplishments measures. Applied Measurement in Education, 14(3), 205–218.
Confidential and Proprietary. Copyright © 2010 Educational Testing Service. All rights reserved.
Confidential and Proprietary.
Copyright © 2009 Educational
References & Bibliography
•
•
The Conference Board, Corporate Voices for Working Families, Partnership for 21st Century Skills,
Society for Human Resource Management (2006). Are they really ready to work? Employers’
perspectives on the basic knowledge and applied skills of new entrants to the 21st century
workforce. Retrieved November 1, 2009, from:
www.21stcenturyskills.org/documents/FINAL_REPORT_PDF9-29-06.pdf
Willingham, W. W., & Breland, H. M. (1982). Personal qualities and college admissions. New York:
College Entrance Examination Board.
Confidential and Proprietary. Copyright © 2010 Educational Testing Service. All rights reserved.
Confidential and Proprietary.
Copyright © 2009 Educational