Preliminary Validity of the Preschool Self-Regulation Assessment (PSRA)
Radiah Smith-Donald, Kirsten Carroll, Paul Goyette, Molly Metzger, Ta-Tanisha Young, C. Cybele Raver
Three Types of Validity
Background
Self-regulation and school readiness
•Components of self-regulation:
-Attention regulation (focusing and shifting attention, executive control)
-Behavior regulation (impulsivity, ability to wait)
-Emotion regulation (managing excitement, frustration, distress)
•Developmental research indicates significant correlations between preschoolers’ selfregulation and school readiness1
•Head Start’s stated goals (2005):
-Increase the school readiness of young children in low-income families
-Provide programs to support social and emotional development, development of
cognitive and language skills, and physical development
•The PSRA is a one-on-one direct assessment measure developed to evaluate children’s
self-regulatory skills in a standardized situation. It uses tasks adapted from well-validated,
lab-based measures of preschoolers’ self-regulation and a report of children’s behavior
throughout the assessment to capture children’s attention and planning skills, impulse
control, how well they follow directions, and their emotional responses.
Inter-rater Reliability
Construct Validity
Method
Method
Method
Do different observers code a child’s performance in the same way?
Which developmental constructs does the PSRA actually capture?
•Interrater reliability was assessed by videotaping and double-coding
about one-third (36.5%) of assessments.
•Task performance on the PSRA and global reports of behavior and
attention were analyzed separately. Raw scores were standardized.
How does children’s PSRA performance relate to their performance on
other measures?
•Reliability correlations were calculated between the assessor’s and
reliability coder’s scores (Cronbach’s alpha for continuous variables and
Cohen’s kappa for categorical variables)
•Factor analysis using principal component extraction was used to see if
observed behavior and task performance clustered together as expected.
Results (n = 23)2
Results4,5 (N = 63)
•Bivariate correlations between constructs were calculated.
1
See the work of Clancy Blair, Susanne Denham, Nancy Eisenberg et al., John Fantuzzo et al., Sheryl
Olson, and Cybele Raver.
Task
Balance Beam
Pencil Tap
Tower Task
Description
Targeted self-regulation skills
Balance Beam
Walk along this line as slowly as you can
Attention/planning
Pencil Tap
Tap 1 time when I tap 2 times
Attention/planning
Tower Task
Take turns adding blocks
Impulse control
Tower Cleanup
Toy Sort
Toy Wrap
Reliability
0.98
1.00
0.91k
Task
Toy Wait
Toy Return
Snack Delay
0.98
0.96
Tongue
PositiveA
0.90
DefiantA
Reliability
0.81
1.00
0.84k
0.91
0.75
0.87
Attention &
Following
Directions
Clean up blocks from the tower
Follow adult directions
Toy Sort
Clean up fun, small toys
Follow adult directions
Don’t peek while I wrap this surprise
Impulse control
Toy Wait
Wait to open surprise
Impulse control
Toy Return
Return of toy after short play period
Follow adult directions
Snack Delay
Wait for beep to get M&M
Impulse control
Tongue
Hold M&M on tongue w/o eating
Impulse control
Assessor Report
Global ratings of children’s attention, affect, Attention, Impulse control, Follow
and engagement largely adapted from the adult directions, Positive and
Leiter-R socio-emotional rating scale
negative emotions, Engagement
Control
Attention &
Impulse
Control
Balance Beam, Pencil Tap, Tower Cleanup, Toy Sort
Positive
Emotion
All correlations <.10
Toy Wrap, Toy Wait, Snack Delay
Cohen’s kappa (for categorical variables)
A averaged across all tasks
Displays positive affect, Engaged with tasks, Socially engages assessor
4
Full sample: 63 Head Start students, 3-5 yrs old
67% Latino, 25% Black, 5% Biracial, 3% other
Implications3
PSRA saves time & money, yet yields high-quality data
•PSRA is not biased by identity of trained observer
•PSRA is not biased by location of observer (live vs. tape)
-Data quality was not weakened by collecting it during the
assessment process
•Obtained reliable data with relatively little training
**Special thanks to the staff of the Chicago School Readiness Project and to the
families and staff at participating Head Start sites
r =.62
Sustains attention, Not easily distracted, Waits to hear directions
2
Toy Wrap
Impulse
r =.45
k
Tower Clean Up
Assessor Report
r =.40
3
Goyette et al poster gives more information concerning reliability
Items with skewed distributions or little variance across children were excluded from factor
analyses.
5
• Bivariate correlations were calculated between PSRA task and assessor
report constructs and:
1. Teacher report of behavior and social competence (SCBE, BPI)
2. Preschoolers’ early academic skills (Head Start NRS)
Results (N = 63)
PSRA tasks
PSRA Tasks
Task
Concurrent Validity
NRS subtest
PSRA tasks
Assessor Report
Attention &
Impulse
Following directions Control
Attention &
Impulse control
Positive
Emotion
Early Math
.08
.22t
.23t
.30*
Letter Naming
.10
.23t
.10
.16
Verbal Skills
.12
.38*
.25*
.29*
Ext. Behavior
-.12
-.22t
-.30*
.16
Int. Behavior
-.31*
-.21t
-.28
.02
Social skills
.53*
.28*
.30*
.14
* p < .05
t
p < .1
•Magnitude of the relationship between PSRA skills and teacher report of
behavior and social skills is similar to that between preschoolers’ selfregulation skills and early academic skills.
Items with factor loadings of 0.4 or higher were included in that construct.
Implications
Implications
•Self-regulation constructs are related, but distinct
-In particular, positive emotion was not related to other selfregulation constructs
•Impulse control and assessor-reported attention were associated
with early academic skills.
•Children who did well on attention/planning tasks also did well on
tasks targeting the tendency to follow adult directions
-Perhaps both task types require thoughtful execution and
concern for rules
•Task reduction – can information on preschoolers’ skills in each area
be captured with fewer tasks?
•Children’s behavior with teachers and peers was associated with
behavior in the structured assessment, but at low levels
-Effect of classroom vs. structured assessment with a stranger?
•While these skills are clearly distinct, they also seem to develop
together (results are in line with previous work).
-Could targeting self-regulation skills contribute to school
readiness?