Assessment and Learning Disabilities
Identification
In Preparation
Step 1
Locate our Tools
Step 2
GO TO
https://www.govote.at/1cb379
http://tinyurl.com/ORRTIiLD
Purpose
• Basic characteristics/Challenges of LD
Identification
• Research and test validity
• Competing models of LD ID
– Research
• Validity
• Reliability
@Jwill_EdNW
#ORRTIi16
3
Objectives
• Become a more informed judge of LD
identification methods.
• Upon leaving the presentation you will have
questions you wish to pursue answers to.
Construct of LD
• A construct can be considered “valid” when a
community of scholars reaches a paradigm
surrounding the construct’s meaning and
generalizability
What are learning disabilities
• What are learning disabilities?
• Do you recognize them when you see them?
• What are common characteristics?
– Write your response(2 mins)
– Find someone approximately the same height as you
and share (5 mins)
What are Learning Disabilities?
What are Learning Disabilities?
Learning Disabilities is a Latent
Construct
• The essential aspect of the construct is
unexpected underachievement
• LD does not exist independently of how it is
measured; All measures are imperfect
indicators of latent constructs
Conceptual
Model
Classification
Hypothesis
Definition
Identification
criteria
Measurement
LD IDENTIFICATION MODELS
IDEA: Regulations
• IQ/Achievement discrepancy
• Response to Intervention (RTI)
• Other alternative research-based procedures
1. IQ/Achievement Discrepancy
• Codified by Public Law 94-142 (EAHCA)
– “severe discrepancy between achievement and
intellectual ability”
• Allowed by 34 states
– Variability between states
– 14 provide no guidance on what a “discrepancy is”
Maki, K. E., Floyd, R. G., & Roberson, T. (2015). State learning disability eligibility criteria: A
comprehensive review. School Psychology Quarterly,30(4), 457.
2. Instructional Response
• Low achievement is
unexpected because deficits
exist despite generally
effective instruction
• LD identification is based on
direct measurement of
basic academic processes
• Low achievement
• Instructional response
criteria
RTI
• 8 states solely RTI
• 9 specify RTI in combination
• 34* allow as is stated by the law
• *includes Washington DC
Maki, K. E., Floyd, R. G., & Roberson, T. (2015). State learning
disability eligibility criteria: A comprehensive review. School
Psychology Quarterly,30(4), 457.
3RD METHOD: PSW
Cross Battery Assessment
Inclusionary Criteria
1. Achievement Deficit
2. Related Processing
Weakness
3. “Normal” Cognitive Ability
based on assessment of 7
CHC broad strata
Flanagan et al., 2007
16
Concordance/Discordance
Method
Inclusionary Criteria
1. Achievement Deficit
2. No Significant Difference
between Achievement Deficit
and Processing Weakness
3. Significant Difference
between Achievement Deficit
and Processing Strength
4. Significant Difference
between Processing
Weakness and Strength
Hale & Fiorello, 2004
17
PSW
• 14 states specify that PSW can be used
• 12 do not specify
• 25 do not allow the approach
Maki, K. E., Floyd, R. G., & Roberson, T. (2015). State learning
disability eligibility criteria: A comprehensive review. School
Psychology Quarterly,30(4), 457.
PSW
• 14 states specify that PSW can be used
• 12 do not specify
• 25 do not allow the approach
Maki, K. E., Floyd, R. G., & Roberson, T. (2015). State learning
disability eligibility criteria: A comprehensive review. School
Psychology Quarterly,30(4), 457.
How can different conceptual
frameworks be compared?
• Processes for the identification of a disorder should be:
– Reliable
– Valid
– Useful
• Questions about the reliability, validity, and utility of
identification processes are ultimately empirical
– Apply proposed criteria and compare resulting groups:
•
•
•
•
•
Academic achievement
Cognitive profiles
Intervention response
Brain functioning
Behavior, attention, anxiety, adult outcomes...
Which side are you on?
• https://www.govote.at/1cb379
• What method does your school use?
• How would you describe it’s
– Reliability
– Validity
– Usefulness
RELIABILITY: A FUNDAMENTAL
CHALLENGE TO LD IDENTIFICATION
All Identification methods are
unreliable at the individual level
• Latent construct: no measure perfectly
measures “reading” or “verbal ability”
• Measures are not perfectly reliable
• As a result, students near the cut point change
group membership
Dot = No LD
Tri =
Discrepant &
LA
Circle = LA
only
Francis et al., 2005
Dot = No LD
Tri =
Discrepant &
LA
Circle = LA
only
Evidence for Unreliability
• IQ achievement discrepancy methods are not
reliable.
• RTI methods are not reliable.
• PSW methods are not reliable
VALIDITY OF DIFFERENT METHODS
FOR LD ID
IQ Achievement Discrepancy
• IQ- discrepant and non- discrepant low
achievers do not differ
• IQ does not predict intervention response
• No difference in brain activation profiles
Francis et al., 1996
Instructional Response
• Adequate and inadequate responders differ in
achievement, cognitive skills, behavior,
subsequent response to instruction, and
neurobiological
• Response to instruction, despite challenges to
reliability, demonstrates good validity across
criteria and definitions (
Inadequate responders demonstrate
lower cognitive ability
Fletcher et al., 2011
Inadequate Responders: Tier 3
Cognitive Correlates of Inadequate
Response
0.6
0.4
0.2
0
C
DFC
-0.2
F
Responder
-0.4
-0.6
-0.8
Phonological
Awareness
Listening
Comprehension
Miciak et al., 2014
Rapid Naming
Processing Speed
Matrix Reasoning
Verbal Knowledge
Grade 1 Intervention Response
• Simos et al., 2005 after
grade 1 intervention in
Mathes et al., 2005.
Baseline MEG Patterns for Adolescent
Adequate and Inadequate Responders
• Rezaie et al., 2011
PSW
Some History
Learning Disabilities Association of America White Paper
• (1) the statutory definition of LD should be maintained
• (2) neither an IQ achievement discrepancy or a failure to respond to
intervention is sufficient for LD identification
• (3) PSW methods make the most empirical and clinical sense
• (4) comprehensive psychoeducational evaluations should occur for
the purposes of LD identification
• (5) the results of cognitive and neuropsychological results should be
utilized for intervention planning and LD identification
These conclusions do not represent a consensus
PSW to Inform Intervention
• Arguments:
– target cognitive deficits and provide training and
improve outcomes or
– tailor instruction to build on cognitive strengths
and accommodate cognitive weaknesses
• LD kids need something different.
Literature Says
• Aptitude-treatment interventions:
– the evidence for such interactions is at best
fragmentary and often contradicted (
Pashler, McDaniel, Rohrer, &
Bjork, 2008)
– no evidence for a group by treatment interaction
such that cognitively focused interventions aimed
at students with specific cognitive deficits produce
better effects
(Kearns & Fuchs, 2013).
Validity of PSW Methods for LD
Identification
Identify students as “LD”
or “not LD” by C/DM and
XBA
Intensive
Intervention in
Reading
Hypothesis: To the extent PSW status is
educationally meaningful, students should
respond differently to the same intervention.
Miciak et al. (2015)
Evaluate Posttest
Performance
Results
• Conducted 39 contrasts to evaluate whether LD status or
inclusionary criteria specified by the C/DM & XBA methods were
significant predictors of intervention response.
• Among 39 contrasts : 4 were statistically significant.
• Only one contrast (Gc => Word Reading) predicted > 1% of the
variance at posttest.
• Academic Achievement Pretest predicted between 53% and
69% of the variance at posttest.
But what about that one?
Cross tabulation of predictions based on r2 = .828 and a cut
point for pass/fail of 25th percentile
Pretest only
Pass
Fail
Pass
670
76
Fail
76
178
Total number of misclassifications =
Pretest + Gc
Status
152
Cross tabulation of predictions based on r2 = .838 and cut
point for pass/fail of 25th percentile
Pass
Fail
Pass
672
73
Fail
74
181
Total number of misclassifications =
147
Reliability
• Simulation Studies
– Stuebing, Fletcher, Branum-Martin, & Francis,
2012
– Taylor, Miciak, Fletcher, & Francis, 2016
• Empirical Studies
– Miciak, Fletcher, Stuebing, Vaughn, & Tolar, 2014
– Miciak, Taylor, Denton, & Fletcher, 2014
– Kranzler, Floyd, Benson, Zaboski, & Thobodaux,
2015
44
Simulation of PSW Methods (Stuebing et al., 2012)
• Of 10,000 assessments:
– CDM: 1,558 identified as LD (8,436 as not LD); 25 correct,
so 1,533 are false positives and get the wrong treatment
– XBA: 678 would be identified as LD (9,322 not LD); 353
correct, 325 are false positives and get the wrong
treatment
Are PSW methods interchangeable?
Approach
Approach
C/DM
XBA
Miciak, Fletcher, et al., 2014
C/DM
XBA
-
30.0
0.31
-
Are PSW methods (C/DM) robust
across different tests?
• What is the level of agreement achieved by two
comparable, but different assessment batteries
utilized for LD identification within the C/DM?
(Word ID, Fluency, Comprehension)
• What is the level of agreement achieved by the
two assessment batteries on the academic
domain of eligibility for LD?
Miciak, Taylor, et al., 2014
Results
•
•
•
•
•
Kappa = .28
Percent agreement = 65%;
Percent positive agreement = 62%
Percent negative agreement = 67%
Also little overlap in the achievement domain
identified as most impaired
Are these results sample specific?
• Simulated > 70,000 latent correlations
between a cognitive strength, cognitive
weakness, and academic weakness
• Compared agreement for battery 1 and
battery 2
Taylor et al, (in review)
Agreement Tables
Test 2
Achievement
Value
< 85
Test 1
Yes
Yes
No
No
46.7
52.4
52.4
3519.1
• Positive agreement does not exceed
chance levels of agreement.
Complete Parameterization:
Positive Agreement
• With 1 exception, every
condition has the
potential to produce PA
=0
• Even when reliability of
both tests is high, PA
rarely exceeds .60
• Mean positive
agreement = .4
51
All methods are unreliable, right?
• Cognitive discrepancy methods like the C/DM are less
reliable than low achievement or RTI methods:
– Increased complexity
– More factors measured = more opportunity for error
– Difference scores are always less reliable than their
constituent scores; they contain error from two measures
82
94
Within 95% CI
88
Difference score
range:
-1 - 23
99
11?
93
105
What does evidence support
• Learning disabilities are dimensional
• All methods based on strict cut points will
demonstrate limited reliability
• Cognitive Discrepancy models demonstrate
poor validity
• RTI models demonstrate good validity
Advantages/disadvantages
• Handout 2
–Response form
Conclusions
• Focus assessment on academic need
• Identification decisions should incorporate
multiple academic measures
Final Thought
• Move away from the historical focal point of
“which students qualify for help”
•
to simply which students need help
Reflection
http://tinyurl.com/ORLDID9
[email protected]
[email protected]
THANK YOU