The Assessment of Intrapersonal and
Interpersonal Competencies
National Academy of Sciences
April 13, 2017
Wayne Camara
Horace Mann Research Chair and Senior Vice
President of Research
Uses of Intrapersonal and Interpersonal Assessments
1. Admissions and Selection
2. Placement (Developmental vs Credit bearing
courses)
3. Personal Growth and Formative Improvement
a) Proactive/Preparatory (K-12)
b) Proactive/Preparatory (prior to enrollment)
c) Remedial
4. Research
5. Accountability in Higher Education
2
Assessments should be driven by intended use
1. Intended purpose of assessment and
claims regarding score interpretations
2. Criteria (G&T, academic success,
engagement, ability to benefit, retention)
3. Relevant context or moderators
(commuting or residing at college, adult
student)
4. Substance (content) and method (format)
of assessment
3
Previous Efforts to Examine Intra- and Interpersonal
Constructs in Higher Education
4
ACT Holistic Framework (Camara et al, 2015)
Core Academic
Skills
Cross-Cutting
Capabilities
Behavioral Skills
Education &
Career Navigation
• English Language Arts
• Mathematics
• Science
•
•
•
•
Technology and Information Literacy
Collaborative Problem Solving
Studying and Learning
Thinking and Metacognition
• Acting Honestly
• Maintaining Composure
• Socializing with Others
•
•
•
•
Getting Along Well with Others
Sustaining Effort
Keeping an Open Mind
Self Knowledge
Environmental Factors
Integration
Managing Career and Education Actions
5
5
Holistic Model of College Success
Science
ELA
Math
Critical Thinking
Technology and Information
Literacy
Studying and Learning
Persistence
Dependability
Self-Confidence
College GPA
Socialization
Academic Self-Efficacy
Goals
6
Holistic Model of College Success
Science
ELA
Math
Critical Thinking
Technology and Info. Literacy
Studying and Learning
Persistence
Goal Striving
Socialability, Optimism
(Dependability, SelfConfidence)
Academic Self-Efficacy
Fit, Supports
(Socialization
Academic Self-Efficacy)
Goals
College
Graduation
7
Behavioral Framework Illustration
8
Growth in Admission Testing
ACT Administrations
4.0
3.5
Millions
3.0
2.5
2.0
1.5
1.0
0.5
0.0
1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014
Total
National
State
• In 2014, 57% of high school graduates took the ACT.
9
Trends in Admissions Factors Considered
Important or Very Important
10
10
Measuring Intrapersonal and Interpersonal Constructs
• Rater biases
A personal statement is
to college admissions as • Lack of verification of author
an unstructured interview • Susceptible to external
is to personnel selection.
assistance
• Lack of standardized scoring
systems or rubrics for evaluation
Comprehensive review is
• ‘Context’ for school quality,
used to evaluate the
individual’s background,
whole person in context
overcoming adversity is largely
and come to a summary
left to one’s own judgment
judgement.
• Absence of research on college
decision processes.
• Missing variables (test, AP, IB)
11
Assessments
Existing Measures in Higher Ed New Techniques
• Engage (6-8, 9-12,
College transition)
• Success Navigator
(College transition)
• Personal Potential Index –
PPI (graduate school)
• Motivated Strategies for
Learning Questionnaire
(College students)
• LASSI
• Individual institutions
practices
• Anchoring Vignettes
• Forced Choice / Ipsative
method
• Natural language
processing – AI
• Gaming
• Big Data - aggregation
12
Validity, Reliability, Fairness
• Construct terms and definitions create confusion.
– Some terms suggest personality traits long considered to be
static. Other terms raise sensitivities about privacy especially in
public education.
• Evidence related to other variables may be most
essential type of validity for higher education adoption.
• Content evidence also essential – fidelity with what is
required to succeed in college (job analysis
methodology)
• Contextual factors – culture, climate, institution,
department.
– Difficult to transport validity evidence globally (teamwork,
sociability have different values in different cultures).
13
STEM
• Academic preparation is essential for STEM success –
but not determinative (Radunzel, Mattern, & Westrick,
2016)
– Many academically prepared students do not complete a STEM
degree – 47%
– While some students not academically prepared do complete a
STEM degree – 19%
• The highest STEM persistence and degree completion
rates are for students with both expressed and
measured STEM interests.
• The lowest rates were for students who had neither
expressed or measured interests in STEM.
14
Impediments in K-12
• Large scale assessments in K-12 today are driven
by accountability (not student centric) – evaluate
schools, teaching, educators.
• Intra- and Interpersonal constructs can not be
easily associated with a course, a teacher, or a
curriculum.
• Not ‘actively’ taught – worst yet, in some
communities constructs may be viewed as outside
the responsibility of a school (home factors).
• Items may be viewed as invasive and not relevant
to schools. (Contrast with career interests)
15
Impediments in Higher Education
• Demand is absent
– Additional hurdles may discourage applicants
– Gaining consensus on constructs, definitions, measures is
difficult
• Change is difficult
– Writing introduced in 2005 by ACT + SAT yet 80% of
colleges do not require that test.
• Admissions professionals feel they can assess these factors
with subjective measures – anecdotal evidence.
• Concern after legal rulings (Fisher v University of Texas,
Grantz v. Bollinger, Grutter v. Bollinger)
• The ‘criterion problem’ in higher education has largely
suppressed large scale research concerning new
predictors for college admissions (Camara, 2005).
16
Impediments to Research and Test Development
• Lack of demand by Higher Education – little
incentive for investment
• Institutional fit and Individual context
• Need operational research as well as experimental
studies
• Cross-institutional research collaboratives required
• Difference between experimental work and large
scale implementation
• Susceptibility of faking – overstated
• Similar acceptance issues as K-12, especially for
public institutions
17
On the positive side…
• Incremental validity
• Assess important outcomes – retention,
graduation, time to degree, engagement.
• Assess important constructs in demand by
employers – problem solving, teamwork,
technology literacy
• May reduce adverse impact against
underserved groups
• Provide a more holistic framework for
decisions – multiple measures.
18
Construct representation - Triathlon
• Triathlon’s have three events.
• Athletes often excel at different events.
• When you reduce a competition to just
2 of the 3 events it may result in
significantly different finishes!
• Ken finishes 1st in swimming, 5th in
running and 15th in biking. Overall he
would rank 7th. If only the first two
events are counted he would rank 3rd,
but if the last two events are counted
he would rank 10th.
• Similar to selecting a subset of relevant
constructs to focus on you may
inadvertently introduce construct
underrepresentation and bias.
19
Incremental Validity and Adverse Impact
Table 1. Correlations with First-Year GPA
(FYGPA)
Research based on ACT
Engage shows that
Intrapersonal and
Interpersonal factors,
particularly academic
discipline:
– Predicts important
outcomes
– Has smaller subgroup
differences
Predictor
ACT
Engage Academic Discipline
ACT + Academic Discipline
FYGPA
0.42
0.27
0.51
Table 2. Subgroup Differences on ACT versus Engage
Academic Discipline
Subgroup
White
Engage Academic
ACT
Discipline
FYGPA
M
SD
M
SD
M
SD
21.83
4.20
African American 17.72 3.32
Cohen's d
1.09
47.00
0.93 2.75
2.75
49.00
0.95 2.30 2.30
-0.26
0.48
20
Subgroup Impact at Differential Levels of Selectivity
Research based on ACT
Engage shows that
Intrapersonal and
Interpersonal factors,
particularly academic
discipline:
– Predicts important
outcomes
– Has smaller subgroup
differences
– Can increase diversity of
admitted students
Institutional
Selectivity
Race/ethnicity
ACT only
African American
3.2
American Indian
0.8
Asian American
2.4
Hispanic
2.1
High Selectivity Pacific Islander
(top 15%)
White
91.4
African American
9.4
American Indian
1.0
Asian American
2.1
Hispanic
3.3
Moderate
0.0
Selectivity (top Pacific Islander
50%)
White
84.1
African American
14.3
American Indian
1.1
Asian American
1.9
Hispanic
4.7
0.0
Low Selectivity Pacific Islander
(top 85%)
White
78.0
ACT and Engage
Academic
Discipline
3.6
1.0
2.5
2.5
90.4
11.2
1.1
2.1
3.8
0.0
81.8
16.9
1.1
1.9
5.3
0.0
74.8
21
What we know vs what we do
•
•
•
•
We know that performance in jobs and
•
in school (graduation, retention, grades,
performance) is influenced by multiple
factors…academic, behavioral, etc.
•
We know that employers value
behaviors and use high stakes
assessments to measure these skills.
We know that reading can confound
measurement of other constructs
•
(math, science and behaviors) and
introduce construct irrelevance resulting
in adverse impact.
We know that academic factors have the
largest subgroup differences.
•
But educational assessments focus on
academic skills such as math and ELA,
ignoring other important constructs.
But in education behaviors are rarely
incorporated in standardized assessment
systems and may be evaluated in
admissions by individual raters using
subjective processes.
But the common core has resulted in math
and science tests with heavy reading loads
and when behavioral skills are assessed
they are often measured in written formats
as opposed to simulations, SJTs (as in
industrial testing).
But we use academic assessments as the
primary or sole focus in accountability,
admissions, and achievement.
22
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service (ETS). All other trademarks are property of their respective owners. 36655.
Intrapersonal and Interpersonal Assessments for
Admissions
There are differences in access, resources, and experiences in
and out of school between ethnic groups and families which
explain roughly
There is a decision to measure some constructs (academic)
and not other constructs (behavior, cross-cutting skills)
The constructs we measure have larger adverse impact against
minorities, first-generation students, and low-income students
than factors we do not measure.
Institutions state they value the these other constructs and
consider them in their decisions – but, they are not measured
with standardized and reliable measures, nor part of the
systematic assessment system.
23