Exploring the Efficacy of Early
Intervention Based on
Psychosocial Risk Factors
Dale Tampke
University of North Texas
&
Alex Casillas
ACT, Inc.
© 2011 by ACT, Inc. All rights reserved.
Agenda
• Introduction to ENGAGE (formerly SRI)
{
{
{
What is ENGAGE?
Reports & Tool Shop
Research
• Case Study: Increasing First-Year Academic
Success and Retention at University of North Texas
{
{
{
Assessment and Intervention Process
Outcomes
Lessons Learned
• Summary & Recommendations
© 2011 by ACT, Inc. All rights reserved.
Introduction
• Historically, prediction of student success has
focused on academic achievement (e.g., test
scores, GPA)
• Growing evidence that success should be
modeled with additional data, including
psychosocial and behavioral factors
• Cognitive and psychosocial data are
independent; their combination improves our
predictive models and intervention strategies
© 2011 by ACT, Inc. All rights reserved.
Academic Behavior Assessments
• Students’ personal development influences
their ability to stay in school and be successful
• Psychosocial development can be structured
into 3 broad domains:
{
{
{
Motivation
Social Engagement
Self-Regulation
© 2011 by ACT, Inc. All rights reserved.
Introduction to ENGAGE
• ENGAGE measures aspects of these 3 domains.
• It contains two predictive indices
(based on ENGAGE scales + achievement scores):
{
{
Academic Success Index: likelihood of GPA 2.0 or higher
Retention Index: likelihood of returning second year
• All ENGAGE scores (scales and indices) are
presented as percentile scores (range 1 to 99)
© 2011 by ACT, Inc. All rights reserved.
What does ENGAGE measure?
Domain
Scales
Motivation
(Getting work done)
•
•
•
•
•
•
Academic Discipline
Commitment to College
Goal Striving
General Determination
Study Skills
Communication Skills
Social Engagement
(Getting along)
• Social Activity
• Social Connection
Self-Regulation
(Keeping your cool)
• Academic Self-Confidence
• Steadiness
© 2011 by ACT, Inc. All rights reserved.
Why ENGAGE?
• Grounded in research:
{
Developed based on meta-analysis regarding predictors of
academic performance and retention (Robbins et al., 2004)
• Helps educators considers the whole student:
{
Provides profile of relative strengths and needs
• Practical and flexible; institutions can:
{
{
{
{
Administer online or paper & pencil at any time
Reports available online 24/7
Identify areas of challenge for students in “at-risk” groups
Create a crosswalk of scales to services to use existing
resources
© 2011 by ACT, Inc. All rights reserved.
ENGAGE Sample Report
Profile
of scores
(scale scores
& percentiles)
Success Indices
(only on Advisor
Report)
Interpretive feedback
(sorted from relative
strengths to needs)
ENGAGE Aggregate Report
© 2011 by ACT, Inc. All rights reserved.
ENGAGE Tool Shop
© 2011 by ACT, Inc. All rights reserved.
Research: Relative Strength of
Predictors of College GPA
Robbins et al. (2006)
© 2011 by ACT, Inc. All rights reserved.
Use of ENGAGE to Identify
Students Academically at Risk
Accuracy of Identification
Selection Method
Drop Out
Academic Difficulty
Random
10%
20%
ENGAGE Success Indices
24%
46%
Note. Students scoring in the bottom 5% of each of these populations were flagged.
Students identified early and provided interventions are more likely
to succeed, and institutions benefit in a variety of ways:
• Improved accountability
• Increased revenue from tuition and fees
© 2011 by ACT, Inc. All rights reserved.
Average First-Year GPA at 4-Year Institutions, by ACT
and ENGAGE Academic Discipline Score
Low AD Scores
Moderate AD Scores
High AD Scores
4.0
Average First-Year College GPA
3.5
3.57
3.17
3.0
2.98
2.68
2.5
2.0
2.11
2.26
2.74
2.28
1.85
1.5
1.0
0.5
0.0
Bottom 25%
Middle 50%
ACT Composite Sc ore
© 2011 by ACT, Inc. All rights reserved.
Top 25%
Robbins et al. (2006)
First-Year College Retention at 4-Year Institutions, by
ACT and ENGAGE College Scores
Low ENGAGE
Moderate ENGAGE
High ENGAGE
First-Year College Retention for 4-Year Institutions
100%
97%
90%
89%
92%
90%
80%
84%
79%
70%
60%
73%
74%
67%
50%
40%
30%
20%
10%
0%
Bottom 25%
Middle 50%
ACT Composite Score
© 2011 by ACT, Inc. All rights reserved.
Top 25 %
Robbins et al. (2006)
Case Study:
Increasing First-Year Academic Success
and Retention at
University of North Texas
© 2011 by ACT, Inc. All rights reserved.
University of North Texas
• 36,000+ students
• One of 15 of U.S. News and World Report’s
top “up and coming” universities
• Located in the 4th largest metropolitan area in
the U.S.
• Since 2000, a state and national leader for
increases in enrollment and degree production
© 2011 by ACT, Inc. All rights reserved.
Assessment & Intervention Process
• Administer ENGAGE to all incoming
first-year students
• Identify at-risk students and follow up with
immediate individualized intervention
• Use crosswalk of campus resources to link
students to appropriate assistance
© 2011 by ACT, Inc. All rights reserved.
Sample Crosswalk of Resources
ENGAGE Scale
Definition
Resources
Academic Discipline
Effort put into school work and
the degree to which students see
themselves as hardworking and
conscientious.
• Learning Center
• Office of Exploring Majors
• UNT Math Tutor Lab
• UNT Student Writing Lab
Social Connection
Feelings of connection and
involvement with school and
community.
•
•
•
•
Academic SelfConfidence
Belief in ability to perform well
in school.
• Learning Center
• Counseling and Testing
• Office of Exploring Majors
© 2011 by ACT, Inc. All rights reserved.
Off-Campus Student Services
Transfer Center
Residence Life
Recreation Center
© Herzog & Miller
© 2011 by ACT, Inc. All rights reserved.
Face-to-Face Meeting
• Advisor or student affairs staff meet with
student during first 5-6 weeks of semester
• Present ENGAGE results by focusing first on
strengths, then needs
• Focus resources to aid student in:
{
{
building on one strength and addressing one area
of improvement
using crosswalk of campus resources
© 2011 by ACT, Inc. All rights reserved.
Measuring Outcomes
• Goal of Intervention:
{
Increase student success and retention
• Indicators :
{
{
{
{
Fall GPA
Percentage of students in good academic standing
(GPA ≥ 2.0) at end of fall semester
Percentage of students returning for Spring semester
Percentage of students returning for 2nd-year (Fall
term)
© 2011 by ACT, Inc. All rights reserved.
Intervention Results
(Based on “At-Risk” Students)
Interventiona
No Interventionb
Average Fall GPA
2.24
2.14
Good academic
standing
73%
63%
Returned in Spring
93%
89%
Returned 2nd year
74%
70%
Indicator
Note. a N = 160
bN
= 262.
© 2011 by ACT, Inc. All rights reserved.
Lessons Learned
• Identify and intervene with at-risk students
early in the semester
• Provide students with resources designed to
increase their opportunity to succeed
{
{
Focus on most important dimensions first
Be strategic in your use of resources
• Track student progress
{
Use your data/feedback to constantly improve the
process
© 2011 by ACT, Inc. All rights reserved.
© Mark Parisi
© 2011 by ACT, Inc. All rights reserved.
One Final Hurdle:
Leading the Horse to Water
• How do you engage and motivate students to
utilize resources effectively?
© 2011 by ACT, Inc. All rights reserved.
Summary & Recommendations
• The combination of cognitive and psychosocial
characteristics is most effective at predicting
outcomes of interest.
• Results of ENGAGE can be used in a variety of ways
{
{
{
Early identification of at-risk students
Student advising
Institutional planning & resource management
• Be clear on Goals
{
Satisfaction, learning, and persistence are not the same
© 2011 by ACT, Inc. All rights reserved.
Summary & Recommendations (cont.)
• Take advantage of campus resources
{
{
{
{
Refer identified students to resources and
interventions
Use a “whole student” approach for service
delivery
Do not be afraid of intrusive advising
Track student progress and outcomes
© 2011 by ACT, Inc. All rights reserved.
Questions?
Correspondence regarding this presentation
should be addressed to:
Dale Tampke
Alex Casillas
Dean of Undergraduate Studies
Senior Research Associate
University of North Texas
ACT, Inc.
[email protected]
[email protected]
For an annotated bibliography of
ENGAGE research studies, please visit:
http://www.act.org/engage/bibliography.html
© 2011 by ACT, Inc. All rights reserved.