Progress Monitoring
of Language: The
Narrative Language
Measures (NLM)
Douglas B. Petersen, PhD, CCC-SLP
Trina D. Spencer, PhD, CSP, BCBA-D
Response to Intervention (RTI)
• A framework for matching educational
services to individual student’s needs.
• Allocating the appropriate level of instruction
and intervention is dependent on reliable,
valid, and practical tools to monitor student
performance in key domains.
– Reading
– Language
Assessment Drives RTI
• Screening
– Universal screening (e.g., Fall, Winter, Spring)
– When students’ scores are compared to a performance
criterion, this is considered benchmarking
– Identifies children who may require intensified services
or more frequent and targeted monitoring
• Progress monitoring
– Regular sampling of performance (e.g., weekly, monthly)
– Data are used to make decisions about
• Adjustments in intervention
• Entry into and exit from tiers
• Need for Special Education
Requirements for
Progress Monitoring Tools
1.
2.
3.
4.
5.
Strong reliability and validity
Time efficient and easy to administer and score
Alternate forms of standardized tasks
Measure socially important outcomes
Sensitive to growth due to intervention
-Deno, 2003
*Curriculum Based Measurement (CBM)
*Also called General Outcome Measurement (GOM)
CBMs for SLPs
• Available CBMs for reading, writing, and math.
• Language, a critical construct, has been
neglected.
• There are few language instruments that fulfill
the technical requirements needed for proper
screening and progress monitoring of
language.
Ukrainetz, 2006
• “There is an urgent need for a
simple, quick, standardized measure of language that
can be administered repeatedly…in a manner akin to
the continuous ‘dibbeling’ that is used to monitor
progress in code learning.”
• “It behooves the profession and research community
to claim the job of developing a set of very
brief, standardized benchmark measures of
vocabulary, grammar, and narrative…”
Narrative Language as a Critical
General Outcome
• Narrative language is associated with academic
achievement.
• Narrative proficiency at age 5 is one of the best
predictors of future reading comprehension
(Catts, Fey, Tomblin, & Zhang, 2002; Dickinson & McCabe, 2001;
Griffin, Hemphill, Camp, & Wolf, 2004.)
Narrative Language as a Critical
General Outcome
• Narratives are socially important.
• Narrative tasks include many aspects of
language content, form, and use.
• Both language comprehension and production
can be measured through narration.
Analysis of Narratives
• Story Structure Analysis
– Entire story is the unit of analysis
– Story completeness
– Story Grammar: setting, initiating event, internal response,
plan, attempt, consequence, resolution
• Linguistic Analysis
– Words and clauses are the units of analysis
– Language complexity
– Language features: vocabulary, subordination, modifiers,
causal and temporal markers and other aspects of grammar
Quality Narratives
• Narrative – an account of causally-related events
in temporal order (aka. a story)
• Quality Narrative…
– has a basic plot/episode structure
• Initiating Event
• Attempt/Action
• Consequence
– includes sophisticated language features that convey
causal and temporal relations between key events
(Ukrainetz, 2006).
Quality is impacted by components and the clarity
of the relations between components!!!
Narrative Language Measures
(NLM)
• A set of stories (Level 1 = 40; Level 2 = 25)
– Relatable content
– Consistent length, story structure, and language
complexity
– Aligned with information on typical 4-5 year old
(level 1) and 6-7 year old (level 2) children’s
narrative skills
Spencer & Petersen, 2010
LanguageDynamicsGroup.com
Equivalent Features of
the NLM:SA Stories
STORY STRUCTURE
•
•
•
•
•
•
•
•
•
•
•
Opening with time element
Main Character is named 6 times
Secondary Character is unnamed
Setting (Main Character’s initial activity and
general location)
Initiating Event (Problem)
Internal Response (Emotion) related to the
Problem
Plan
Attempt by Main Character to solve the
Problem (Main Character asks the Secondary
Character for help)
Consequence (the Secondary Character helps
the Main Character)
Ending (what results from the Secondary
Character’s help)
Reaction (Ending Emotion related to the
Consequence)
LINGUISTIC COMPLEXITY
•
•
•
•
•
•
•
•
•
•
•
•
108-110 words
Dialogue exchange between two characters
(marked by quotations)
2 Temporal conjunctions ‘then’
1 Left-branching temporal subordinate clause
using ‘After’
1 Left-branching temporal subordinate clause
using ‘When’
3 Right-branching causal subordinate clauses
using ‘because’
1 Complex subordinate clause not associated
with dialogue
1 Adversative conjunction ‘but’
1 Adjectival (relative) subordinate clause
using ‘that’
1 Pre-noun modifier (adjective)
1 Double pre-noun modifier (adjectives)
2 Pre-verb modifiers (adverbs)
Narrative Language Measures
(NLM)
• A set of administration procedures for…
– Retell
– Story Comprehension
– Personal Generation
• A set of scoring guides
– Real time
– Transcript
Narrative Language Measures
(NLM)
• Test of Narrative Retell (TNR)
– General Outcome Measure (or CBM)
– Most developed
– Most often used in our research
• Test of Story Comprehension (TSC)
– Easier task for very young children
– Comprehension of Wh-questions is needed
• Test of Personal Generation (TPG)
– Clinically important
– Limitations preclude it from being a GOM/CBM
Characteristics of Curriculum
Based Measures (Deno, 2003)
• sample authentic child behaviors that reflect key
outcomes
• have standardized administration and scoring
procedures
• be time efficient, economical, and easy to use
• meet the requirements of technical adequacy
• be sensitive to growth due to intervention or
change over time
Test of Narrative Retell (TNR)
• Utilizes NLM stories: Level 1 (40); Level 2 (25)
• Assesses language using a retell format
– Captures both language comprehension and
production.
– Retell and Story Comprehension .43 (p = .005*)
– Retell and Personal Generation .64 (p = .0001*)
– Story Comprehension and Personal .23 (p =.10)
Level 1 TNR with Pictures
• Lay out pictures before beginning the story.
• As you read the story, point to the picture that
depicts the part of the story.
• Remove the pictures when the child retells the story.
Time Efficient, Economical, and Easy
• Administration takes about 1-2 minutes each.
• Administration materials: TNR booklet, digital
audio recorder, and a pencil (pictures for
preschoolers).
• Very easy to learn: follow script, read slowly
and clearly, provide encouragement, but don’t
prompt.
• Children like the stories.
TNR Scoring Options
1. Use a digital voice recorder, transcribe, and
score from the transcription
2. Use a digital voice recorder and score from
audio
3. Score in real time (although may want to
have the recording as a back up)
TNR Preliminary Psychometric Data
• Concurrent Validity
– The Renfrew Bus Story and TNR (r = .88)
– Index of Narrative Complexity and TNR (r = .93)
• Reliability
–
–
–
–
Alternate Form Reliability (.73-.88, p < .0001*)
Fidelity of Administration (91%)
Interrater Reliability (96%)
Transcript and Real Time Scoring Reliability (89%)
• Sensitivity
– INC average gain was 9.9 points
– TNR average gain was 13.7 points
Test of Narrative Retell
Mean Scores
d= .80
TNR Scores
Test of Story Comprehension (TSC)
• Utilizes NLM stories: Level 1 only (40)
• Comprehension questions are considered
easier, but understanding wh-questions are
necessary to do well.
– Primarily captures language comprehension
– Retell and Story Comprehension .43 (p = .005*)
– Retell and Personal Generation .64 (p = .0001*)
– Story Comprehension and Personal .23 (p =.10)
Mean Scores
Test of Story Comprehension
d= .52
Test of Personal Generation (TPG)
• Assesses language using a conversation elicitation
technique adapted from Peterson & McCabe
(1983) and McCabe & Rollins (1994).
• Examiner elicits personal stories while playing
with a young child in a very natural manner.
• Provide at least 3 opportunities to tell a personal
story.
• Significantly limited by content relatability and
the child’s willingness to tell a story.
• This is the most difficult narrative task we use.
TPG Variation – Level 2
• Immediately following a child’s retell narrative
(TNR), elicit a personal generation by
asking, “Has something like that ever
happened to you?”
• Samples a different dimension of their
narrative skills in a very efficient manner
– Content limitations
– Motivation limitations
TPG Scoring Guide
• Can be used for scoring TNR, TPG, or any
other story the child generates (e.g., fictional).
• To use this scoring guide, the narrative has to
be transcribed first.
• More time consuming and complicated to
use, but it is more comprehensive.
• Best used for intervention planning; can be
used in research and infrequent progress
monitoring.
The NLM: A Promising Tool for Supporting RTI of Language
EXPLORING THE POSSIBILITIES
Assessment in the RTI Context
• Early Identification
– Identify children who may require intervention
• Level
• Gain
• Progress Monitoring
– Before, during, and after intervention, performance
is probed to document learning
– Results used to make adjustments in intervention
Seasonal Assessment Using the NLM
(aka benchmarking or universal screening)
• Suggestions for Early Identification
– Use child’s best score out of 3 administrations for data analysis
– Use a criterion score (level of performance) based on local
norms to identify children who would benefit from intervention
– Use gain scores (or lack of gain) to identify children who would
benefit from intervention
14
14
12
12
10
10
8
6
8
Child 1
6
4
4
2
2
0
0
Fall
Winter
Spring
Child 2
Fall
Winter
Spring
Dynamic Approach to Early
Identification
Test
All children were assessed prior to
beginning the intervention (3 times)
Teach
Test
Children received a brief, narrative
intervention as a whole class
All children were re-assessed
following the intervention (3 times)
Responsiveness: Identification
+
-
Responder
Gainer
Leveler
Minimal
Responder
Gain
+
Level
-
TNR: Level = 7; Gain = 2
TSC: Level = 5; Gain = 1
Responsiveness: TNR
Minimal
Responder, 27.8
0%
Responder, 58.3
0%
Gainer, 5.60%
Leveler, 8.30%
Responsiveness: TSC
Minimal
Responder, 19.4
0%
Responder, 47.2
0%
Gainer, 11.10%
Leveler, 22.20%
Progress Monitoring to
Document Learning
• NLM Total Scores
– Most sensitive to change
– Most closely related to general language abilities
Progress Monitoring to
Document Learning
Baseline
TNR Total Score
Intervention
Maintenance
Progress Monitoring to
Document Learning
Baseline
TPG Total Score
Intervention
Maintenance
Progress Monitoring to
Document Learning
Baseline
TPG Variation
(Level 2)
Intervention
Maintenance
Progress Monitoring to Make
Adjustments to Intervention
• NLM Element Scores
– Helps clinician target specific narrative and
language features for individual children
• Differentiated intervention in small groups
– Helps clinician determine when targets are
mastered
– Helps clinician make decisions about intensity
(e.g., number of sessions, number of trials, explicitness of
prompting, etc.)
TNR Scores
Element Analysis Using Grids
Baseline
Intervention
=2 points
=1 point
=0 points
LEVEL 1 - PreK
LEVEL 2 – K1
LEVEL 1 - PreK
LEVEL 2 – K1
Free NLM materials available at
LanguageDynamicsGroup.com
For more information, contact:
Doug Petersen: [email protected]
Trina Spencer: [email protected]