Reading Comprehension is
Not a Single Ability
Hugh W. Catts
Florida State University
California Speech-Language-Hearing Association
April 2016
Reading Comprehension?
• Educational/clinical
implications
Reading Comprehension?
LARRC:
Language and Reading
Research Consortium
Institute of Education Sciences (Grant# R305F100002)
Investigators:
Laura Justice (PI)
Shelley Gray (Co-PI)
Hugh Catts (Co-PI)
Tiffany Hogan (Co-PI)
Kate Cain (Co-PI)
Ron Nelson
Diane Nielsen
Laida Restrepo
Stephen Petrill
Jim Bovaird
Richard Lomax
Shayne Piasta
Ann O’Connell
Mindy Bridges
Jill Pentimonti
larrc.ehe.osu.edu
Reading Crisis
• NAEP data show that 65%
of 4th graders and 64% of 8th
graders are reading below
grade level on national
assessments
• PISA data appear to show
that American children are
losing ground to those from
other countries
PISA Results
2009
2012
Shanghai-China
Korea
Finland
Hong Kong
Singapore
Canada
New Zealand
Japan
Australia
United States
Shanghai-China
Hong Kong
Singapore
Japan
Korea
Finland
Canada
Taiwan
Ireland
Poland
Estonia
Liechtenstein
Australia
New Zealand
Netherlands
Belgium
Macro-China
Switzerland
Germany
United States
LARRC:
Language and Reading
Research Consortium
Institute of Education Sciences (Grant# R305F100002)
Investigators:
Laura Justice (PI)
Shelley Gray (Co-PI)
Hugh Catts (Co-PI)
Tiffany Hogan (Co-PI)
Kate Cain (Co-PI)
Ron Nelson
Diane Nielsen
Laida Restrepo
Stephen Petrill
Jim Bovaird
Richard Lomax
Shayne Piasta
Ann O’Connell
Mindy Bridges
Jill Pentimonti
larrc.ehe.osu.edu
7
Vocabulary
Syntax
Discourse
Inference Making
Comprehension
Monitoring
Reading
Comprehension
Sight Word
Decoding
Phono Awareness
Working Memory
Intervention
Proximal gains
Improvement in reading comprehension
Mental
Model
Text
Coherent Understanding
Situation Model
Knowledge
Knowledge
Sara first let loose a team of gophers. The
plan backfired when a dog chased them
away. She then threw
a party but
1. Where
didthe
Sara put the
guests failed to bring their
motorcycles.
gophers?
Furthermore, her stereo system was not
loud enough. Obscene phone calls gave her
2. number
Why did
want the guests
some hope until the
wasSara
changed.
tothe
bring
theirneon
motorcycles?
It was the installation of
blinking
lights across the street that finally did the
trick. She framed3.
the What
ad fromdid
thethe
classified
ad say?
section and now has it hanging on her wall.
GETTING RID OF
BAD NEIGHBORS
Knowledge Gap
• Schools have dramatically reduced time spent teaching
subject matters like science and social studies in lieu of
generic reading comprehension instruction
• Teach strategies like “finding the main idea” or
“comprehension monitoring.”
• Students do read subject matter material within RC lessons –
but it is one subject one day and another the next – no time
to build knowledge
• Skills may help but knowledge is the building block of more
knowledge
Knowledge Matters
http://www.knowledgematterscampaign.org
• We don’t have a reading crisis, we have a knowledge
crisis – knowledge is literacy
NAEP Reading Topics
4th Grade
8th Grade
Blue crabs
Brazilian beetles
Antarctica penguins
Ellis Island
Woman astronaut
Telescopes and space
Cultural stereotypes (N)
Fishing with granddaddy (N)
Oregon trail
Great white shark
Anasazi Indians
19th Amendment
Robots
Cane Toads
Bus schedules
Chinese Emperor (N)
Knowledge Matters
http://www.knowledgematterscampaign.org
• We don’t have a reading crisis, we have a knowledge
crisis – knowledge is literacy
• Daniel Willingham -Teaching content is teaching
reading
https://www.youtube.com/watch?v=RiP-ijdxqEc
• Reading comprehension is not a skill but a condition
you create (Robert Pondiscio)
David Pearson
Valencia, S., Wixson, K., &
Pearson, P.D. (2014). Putting text
complexity in context. The
Elementary School Journal, 13,
270-289.
Pearson, P.D., Valencia, S., &
Wixson, K. (2014). Complicating
the world of reading assessment:
Toward better assessments for
better teaching. Theory into
Practice, 53, 236-246.
RAND Reading Study Group (2002)
Sociocultural Context
Setting, Supports, Resources
Text
Activity
Genre
Complexity
Subject Matter
Find a solution
Evaluate an argument
Write a book report
Prepare for test
Reader
Decoding skills
Language abilities
Cognitive abilities
Background Knowledge
RAND model
• Reading ability/disability is a more fluid concept
than typically thought
• Given different combinations of text and activity,
the best reader can be expected to perform quite
poorly and the poorest reader quite well
• Reading ability is not solely “beneath the skin and
between the ears” (Mehan, 1993)
• Because of the interactive nature of the process,
we are unable to reduce RC performance down to
a single score
• It’s not a single thing
Reading Comprehension Tests
•
•
•
•
Examined how 4 commonly used
standardized reading comprehension
tests compared with each other (QRI,
GORT-3, WJPC-3, PIAT)
Bivariate correlations ranged from
.45-.68 (median = .54)
Lower correlations at higher grades
Overlap in Diagnosis was only 40%
Variability across measures of RC
• Measurement error
• Different formats of assessment
• Different content
Method of Assessment
Collins, A., Compton, D., Gilbert, J., & Lindstrom, E. (2015). Performance variation
across reading comprehension assessments: Examining the unique contributions
of response format, text genre, and child skills.
Keenan, J., Betjemann, R., & Olson, D. (2008). Reading comprehension tests vary
in the skills they assess: Differential dependence on decoding and oral language.
Scientific Studies in Reading, 12, 281-300.
Cutting, L.E., & Scarborough, H. (2006). Prediction of reading comprehension:
Relative contributions of word recognition, language proficiency, and other
cognitive skills can depend on how comprehension is measured. Scientific Studies
of Reading, 10, 277-299.
Muijselaar, M., Swart, N., Steenbeek-Planting, E., Verhoeven, L. & de Yong, P. (in
press). The dimensions of reading comprehension in Dutch children: Is
differentiation by text and question necessary? Journal of Educational Psychology.
Method of Assessment
Collins, A., Compton, D., Gilbert, J., & Lindstrom, E. (2015). Performance
variation across reading comprehension assessments: Examining the unique
contributions of response format, text genre, and child skills.
•
•
•
•
•
•
79 fourth-grade students
Read 3 narrative and 3 expository passages
Answered open-ended or multiple choice questions or retell
Counterbalanced across passages
Bivariate correlations ranges from .37-.48
Interaction of assessment method and ability
Method of Assessment
Collins, A., Compton, D., Gilbert, J., & Lindstrom, E. (2015). Performance variation
across reading comprehension assessments: Examining the unique contributions
of response format, text genre, and child skills.
Keenan, J., Betjemann, R., & Olson, D. (2008). Reading comprehension tests vary
in the skills they assess: Differential dependence on decoding and oral language.
Scientific Studies in Reading, 12, 281-300.
Cutting, L.E., & Scarborough, H. (2006). Prediction of reading comprehension:
Relative contributions of word recognition, language proficiency, and other
cognitive skills can depend on how comprehension is measured. Scientific Studies
of Reading, 10, 277-299.
Muijselaar, M., Swart, N., Steenbeek-Planting, E., Verhoeven, L. & de Yong, P. (in
press). The dimensions of reading comprehension in Dutch children: Is
differentiation by text and question necessary? Journal of Educational Psychology.
Variability across measures of RC
• Measurement error
• Different forms of assessment
• Different content
LARRC
•
•
•
•
•
Approximately 700 3rd grade children
Presented with 8 passages adapted from QRI
Read 4, Listened to 4
2 narrative and 2 expository
Answered 5-8 questions
Bivariate Correlations between
Passages (Percent Correct)
Reading Comprehension Passages
RC .1
RC.1 The Trip to the Zoo
RC.2
.434
RC.2 Bomb Beetle
RC.3
RC.4
.456
.406
.497
.474
RC.3 The Horned Frog
.465
RC.4 A New Friend from Europe
Listening Comprehension Passages
LC .1
LC.1 The Friend
LC.2 Puffin
LC.3 Windshield Wiper
LC.4 A Special Birthday for Rosa
LC.2
.498
LC.3
LC.4
.467
.370
.411
.365
.418
1-10 of N=21 with mean passage % correct close to
sample mean (70%)
11-21 of N=21 with mean passage % correct
close to sample mean (70%)
1
1
0.9
0.9
0.8
0.8
0.7
0.7
0.6
0.6
0.5
0.5
0.4
0.4
0.3
0.3
0.2
0.2
0.1
0.1
0
0
1
2
3
4
5
6
Passages 1-4: Reading Comprehension
Passages 5-8: Listening Comprehension
7
8
1
2
3
4
5
6
7
8
N=12 with mean passage percent correct
close to -1 SD (55.2%)
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
1
2
Passages 1-4: Reading Comprehension
Passages 5-8: Listening Comprehension
3
4
5
6
7
8
Variability across measures of RC
• Measurement error
• Different forms of assessment
• Different content
• Create a latent variable
Latent factor
Reading Comprehension
Measure 1
Measure 2
Measure 3
Predicting Reading Comprehension Over Time:
Third through Fifth Grade Children (Lonigan, 2015)
RCOMP T1
Gates T1
(R2 = .56)
WJ-PC T1
(R2 = .60)
TSREC T1
(R2 = .56)
RCOMP T3
RCOMP T2
Gates T2
(R2 = .59)
WJ-PC T2
(R2 = .66)
TSREC T2
(R2 = .59)
Gates T3
(R2 = .57)
WJ-PC T3
(R2 = .61)
TSREC T3
(R2 = .57)
Predicting Reading Comprehension Over Time:
Third through Fifth Grade Children (Lonigan, 2015)
1.0
1.0
RCOMP T1
Gates T1
(R2 = .56)
RCOMP T3
RCOMP T2
WJ-PC T1
(R2 = .60)
TSREC T1
(R2 = .56)
1.0
Gates T2
(R2 = .59)
WJ-PC T2
(R2 = .66)
TSREC T2
(R2 = .59)
Gates T3
(R2 = .57)
WJ-PC T3
(R2 = .61)
TSREC T3
(R2 = .57)
Can we improve reading comprehension by
intervention in general knowledge and skills?
Vocabulary
Syntax
Discourse
Inference Making
Comprehension
Monitoring
Reading
Comprehension
Sight Word
Decoding
Phono Awareness
Working Memory
Methods and Measures
xxxx
Vocabulary
-PPVT-4
-Expressive Vocabulary Test - 2
-CELF-4 Word Classes
Grammar
-CELF-4 Word Structure
-CELF-4 Recalling Sentences
-TEGI
-TROG-2
-Morphological Lexical Judgment
-Wagner Morphological Derivation Task
Higher-level Skills
-Inference Making Stories
-Comprehension Monitoring: Knowledge
Violations
-Comprehension Monitoring: Inconsistencies
-Narrative Structure Task – Picture or
Sentence Arrangement
Cross-sectional sample from Year
1 and 2 data – 245 G3 children
Working Memory
-WJ-III Numbers Reversed
-WJ-III Auditory Memory
-Memory Updating Task
Decoding Abilities & Reading
Precursors
WRMT-R/NU – Word ID
WRMT-R /NU – Word Attack
TOWRE-2
Pressure Points
39
Analysis
• Multiple measures of each construct
• Used CFA to create factor scores
• Conducted both OLS and quantile multiple
regression
• Latter uses a weighted procedure to
examine relationships at selected quantiles
or percentiles of reading comprehension
Pressure Points
40
OLS Multiple Regression
Overall model parameter estimates
Standard
Variable
Estimate Error
t-value p-value
Intercept 0.00
0.04
0.02
0.985
Vocabulary 0.28
0.06
4.56
<.001
Grammar 0.33
0.07
4.48
<.001
High-Level Language 0.18
0.05
3.88
<.001
Word Reading 0.16
0.05
3.36
<.001
Memory 0.05
0.05
1.03
0.302
R2=.69
Pressure Points
41
Analysis
• Multiple measures of each construct
• Used CFA to create factor scores
• Conducted both OLS and quantile multiple
regression
• Latter uses a weighted procedure to
examine relationships at selected quantiles
or percentiles of reading comprehension
Pressure Points
42
Quantile Multiple Regression
Quantile specific parameter estimates
Vocabulary
High-Level Language
Grammar
Word Reading
Memory
0.8
0.6
0.4
0.2
0
0.9
0.7
0.5
0.3
0.1
0.9
0.7
0.5
0.3
0.1
0.9
0.7
0.5
0.3
0.1
0.9
0.7
0.5
0.3
0.1
0.9
0.7
0.5
0.3
-0.2
0.1
Coefficient
1
Quantile of Reading Comprehension
Pressure Points
43
Poor comprehenders
Specific comprehension deficit – problems in
reading comprehension despite normal or near
normal word reading abilities
25% of those with poor comprehension in 3rd
grade and above
Poor comprehenders
• About 1/3 met criteria for LI (also see Adlof,
2010; Nation et al., 2004)
• Parents reported that only 18% had received
speech and language services
Poor Comprehenders
• Often did not have a reading problem until later in
school
• Normal readers in 2nd grade
• Problems emerged in 4th grade
• Late-Emerging Poor Reader
Language Intervention
Academic Language
• Discipline-general language knowledge and skills that
cut across content areas and allow students to
communicate about and gain discipline-specific
knowledge
• vocabulary, syntax and discourse structures found
across content areas in print
• Core academic language skills (CALS; Uccelli et al.,
2014, Applied Psycholinguistics) – Grades 4-8
Academic Vocabulary
• Tier 2 words (analyze, enormous, evidently)
- context of storybooks or text
- teach synonyms and antonyms
- provide opportunity to forget and relearn
• Morphosyntax (include, inclusion, inclusive)
- unpacking complex words
- learn how morphemes work
Bringing Word to Life: Robust Vocabulary Instruction
(Beck, McKeown, & Kucan, 2013)
Unlocking Literacy: 2nd Ed. (Henry, 2010)
Complex Syntax
• Unpacking of noun phrases
- prepositional phrases
- relative clauses
• Subordination
– because, although, unless, even though, if,
whenever, provided that, though, since,
while, whereas
Complex Syntax
“ The girl from across the street with the
boyfriend who has the big dog is sick.”
“I am pleased to report that at the April 13th UPC
meeting, UPC members voted to APPROVE the
Communication Science and Disorders QER Review
Report with a written follow up, by the Program, of
the recommendations in the report to be completed
by the next QER review.”
Complex Syntax
• Unpacking of noun phrases
- prepositional phrases
- relative clauses
• Subordination
– because, although, unless, even though, if,
whenever, provided that, though, since,
while, whereas
Cohesion
• Cohesive devices tie one portion of the text to
another
- conjunctives (e.g., alternatively, however, nonetheless)
- enumeration (first, next, then)
- anaphoric pronouns
- text structure
Text as an language lesson
• Vocabulary
• Syntax
• Text structure
• Pick an appropriate book or text
• Teach the language in the book
LARRC
•
•
•
•
766 children PreK – 3rd Grade
25 week language intervention
Four 30 minute sessons a week
Vocabulary, text structure, inferencing,
comprehension monitoring
• Some proximal gains
• No effect on standardized RC/LC tests
Lesaux et al. (2014)
Effects of academic vocabulary
intervention for linguistically diverse
adolescents: Evidence from a
randomized field trial. American
Educational Research Journal, 5.,
1159-1194.
•
•
•
•
n = 2082 (70% LM)
20-week, 45 minute daily
70 academic words
Word-learning strategies
morphological awareness
• near (proximal) and far transfer
• Vocabulary, morphology, reading
comprehension, writing
Vocabulary
Syntax
Discourse
Inference Making
Comprehension
Monitoring
Reading
Comprehension
Sight Word
Decoding
Phono Awareness
Thinking
Working Memory
System 1
- automatic
- intuitive
- attention limited
- “seat of the pants”
System 2
- deliberate
- logical
- rational
- resource demanding
Thinking While Reading
• Reading strategies promote
thinking
“
- comprehension monitoring
- question generating
- summarization
- graphic organizer
Reading Strategies
Ciullo, S., et al. (2016). A synthesis of research on
Informational text reading interventions for
elementary students with learning disabilities.
Journal of Learning Disabilities, 49, 257-271.
Swanson, E. et al. (2014). A synthesis and metaanalysis of reading interventions using social studies
content for students with learning disabilities.
Journal of Learning Disabilities, 47, 178-195.
Thinking While Reading
• Reading strategies
- promote cognitive engagement
- limited effect
IKEA Desk
• Think about the desks you
have put together in the
past
• Find the “main” piece to
work with and start
assembling
• As you work, stop and ask
yourself if it looks correct
Thinking While Reading
• Reading strategies
- promote cognitive engagement
- limited effect
- not automatically transferable
- not specific to topic and task
Sociocultural Context
Setting, Supports, Resources
Therapy Goals
1. Improve reading comprehension
Text
Genre
Subject Matter
Complexity
Activity
Identify theme
Compare perspective
Synthesize texts
Reader
Decoding skills
Language abilities
Cognitive abilities
Reading
Comprehension
1908-2015
Sociocultural Context
Setting, Supports, Resources
Therapy Goals
1. improve evaluation of arguments
Text
Genre
Subject Matter
Complexity
Activity
Evaluate argument
Appreciate narrative
Identify theme
Compare perspectives
Reader
Decoding skills
Language abilities
Cognitive abilities
Background knowledge
2. appreciate narratives
3. identify underlying theme
4. compare author perspectives
5. background knowledge
6. academic language
Should we measure reading comprehension?
Assessment
• No compelling reason to give a domain general
reading comprehension test
- not single thing
- not reliable
- does not inform practice
• EOY assessment – science, history, literature, etc
Text-Task Scenarios
• Pearson and colleagues argue that we specifically
measure children abilities on specific comprehension
text-tasks situations
• For an assessment to be valid it must reflect a
reasonable range of variation on various texts and
tasks - RC is not a single thing
• Assess whether a child can evaluate an argumentative
text on a familiar topic written with grade level
complexity – summarize vs. take a stance and support
claim
• Appreciation of a narrative
• Contrast perspectives
Assessment
• Measure components
- word recognition speed and accuracy
- academic language knowledge
- domain knowledge
- think alouds (Wade, 1990)
Thank you
[email protected]