381
META-ANALYTIC REVIEW OF RESPONSIVENESS-TOINTERVENTION RESEARCH: EXAMINING FIELD-BASED AND
RESEARCH-IMPLEMENTED MODELS
Matthew K. Burns
James J.
Appleton
Jonathan D. Stehouwer
University of Minnesota
A
responsiveness-to-intervention (RTI)
approach to diagnosing LD is a leading alternative to current practice. This study conducted a meta-analytic review of research on
four existing large-scale RTI models and
other models implemented for research.
Twenty-four effect sizes and unbiased estimates of effect (UEE) were computed. Results
found a larger UEE for studies of existing RTI
models than those implemented by university
faculty for research, but both were strong.
The UEE for student achievement and sys-
temic outcomes both exceeded 1.0, but the
UEE for systemic outcomes among field-based
RTI models was nearly twice as large as for student outcomes. Further, RTI models implemented for research led to an UEE of 1.14 for
student outcomes and 0.47 for systemic outcomes. The mean percentage of
nonresponders in the studies was 19.8% =
SD 12.5), and
(
an average of 1.68% (
SD 1.45) of the student population was placed into special education. Implications for practice and future
research are included.
=
Changing current educational practice is a frustrating and discouraging
process whereby disagreement is fundamental to success and success is unlikely to occur (Goor, 1995). The mandate for the education of children with disabilities was established in 1975 with the original authorization of the
Individuals with Disability Education Act (IDEA). Yet special education has not
experienced much substantive and lasting change in the 30 years since its formal inception, despite frequent and meaningful controversies (Ysseldyke,
Algozzine, & Thurlow, 2000). An especially intense debate has developed in
special education around proposed changes in the reauthorization of IDEA
because some seemingly dramatic reforms have been discussed. Perhaps the
most notable could be the diagnosis of specific learning disabilities (LD).
Although the diagnosis of LD in the schools has a long history of controversy,
beginning almost immediately after the initial implementation of IDEA
(Aaron, 1997; Meyer, 2000; Myers & Hammill, 1990; Ysseldyke, Algozzine, &
Epps, 1983), most states continue to use diagnostic procedures outlined in
1975 (Tomasi & Weinberg, 1999). Alternative procedures discussed as potential federal mandates represent a significant departure from current practice.
The President’s Commission on Excellence in Special Education (PCESE,
2001) fueled debate when it endorsed a responsiveness-to-intervention (RTI)
diagnostic approach for LD. Gresham (2001) defined RTI as identifying a child
with LD only after academic behaviors do not significantly change pre- and
postimplementation of a validated intervention. The concept of determining
special education eligibility based on a lack of progress is not new (e.g., Fuchs
& Fuchs, 1998; Vellutino et al., 1996), and advocates argue that federal special
Downloaded from http://jpa.sagepub.com at UNIV OF UTAH on August 12, 2008
© 2005 SAGE Publications. All rights reserved. Not for commercial use or unauthorized distribution.
382
always included an RTI component in its definition of LD by
mandating learning experiences appropriate for the child’s age and ability levels before assessing for a severe discrepancy. However, a single definition of RTI
has yet to be established, and the meaningful differences that exist between the
definitions and models that have been presented by various groups have &dquo;inadvertently contributed to the confusion surrounding RTI&dquo; (Fuchs, Mock,
Morgan, & Young, 2003, p. 159).
Regardless of the RTI model proposed, each generally involves group problem solving, close monitoring of student progress, implementation of researchbased individual interventions, and consideration for special education services only after a student fails to respond adequately (Fuchs et al., 2003). There is
a long history of research searching for validated behavioral and academic
interventions (e.g., Kavale & Forness, 2000; Shapiro, 2004; Shinn, Walker, &
Stoner, 2002; Swanson, 1999, 2000; Swanson, Hoskyn, & Lee, 1999). Previous
studies that addressed the use of monitoring progress to responsiveness used
norm-referenced achievement tests as postintervention criteria for adequate
response (e.g., a standard score of 90 or higher; Torgesen, Alexander, Wagner,
Rashotte, Voeller, & Conway, 2001) or the determination of the slope of student growth between pre- and postintervention measures (Vellutino et al.,
1996). In recent years, a more common approach to progress monitoring
education law has
involves the
of curriculum-based measurement (CBM) data to determine
of responsiveness over a 2-month period (Deno, Fuchs,
estimate
slopes
&
Marston,
Shin, 2001; Fuchs, 2003; Fuchs & Fuchs, 1998; Fuchs, Fuchs, &
Speece, 2002), for comparisons to postintervention benchmarks (e.g., 40
words/minute for reading fluency at the end of the first grade, Fuchs, 2003;
reading 50 words/minute in third grade, Burns, Tucker, Hauser, Thelen,
Holmes, & White, 2002), or to compare the slope and level of the data to peers
(Fuchs, 2003; Speece, Case, & Molloy, 2003). The optimal method of assessment to serve the needs of children is still being determined by ongoing
research.
Perhaps one aspect of RTI that research has conclusively supported is the
effectiveness of a group problem-solving model (Tilly, 2002). Fuchs et al.
(2003) identified four group-level problem-solving models that are consistent
with RTI, two of which are used to make decisions regarding special education
eligibility (see Fuchs et al., 2003, for a detailed description of each): Heartland
Agency (Iowa) Model (Heartland; Ikeda, Tilly, Stumme, Volmer, & Allison,
1996); Ohio’s Intervention Based Assessment (IBA; Telzrow, McNamara, &
Hollinger, 2000); Pennsylvania’s Instructional Support Teams (IST; Kovaleski,
Tucker, & Duffy, 1995); and Minneapolis Public School’s Problem-Solving
Model (MPSM; Minneapolis Public Schools, 2001). These four are widely presented as large-scale implementations of RTI currently in practice that have
their roots in prereferral group problem solving (Fuchs et al., 2003).
A previous review of research on prereferral intervention assistance teams
found strong effects for student (e.g., increased task completion and decreased
behavioral and academic difficulties) and systemic outcomes (e.g., reduction
of referrals to and subsequent new placements in special education), but the
average effect size associated with teams that were implemented for research
use
as an
Downloaded from http://jpa.sagepub.com at UNIV OF UTAH on August 12, 2008
© 2005 SAGE Publications. All rights reserved. Not for commercial use or unauthorized distribution.
383
than twice as large as those studies that included teams
in
the
field (Burns & Symington, 2002). It was hypothesized
already existing
that the difference in size of effect was due to inconsistent implementation
among the latter group (Burns & Symington, 2002), which could significantly
purposes
was more
hinder future
developments in group problem solving (Burns, Vanderwood, &
Ruby, 2005). Therefore, a comparison of similar variables between existing RTI
models and those implemented for research purposes could suggest whether
similar implementation issues affect results.
National policy regarding LD diagnosis has widespread implications for children, given that those diagnosed as LD represent over 50% of all children identified with a disability and approximately 5% of the total school population
(Lerner, 2002). Thus, whereas some suggest a cautious transition toward an RTI
model for eligibility and diagnosis (Fuchs et al., 2003), others generally oppose
this transition, considering it a dramatic shift from current practice (Hale,
Naglieri, Kaufman, & Kavale, 2004). Although the unknown creates anxiety,
school personnel tend to exhibit a somewhat perpetual search for and fascination with that which is new (Ellis, 2001). Therefore, as was the case with many
seemingly new educational innovations, RTI has been embraced with enthusiasm among some scholars (Gresham, 2002; Gresham et al., 2004;
Reschly &
Ysseldyke, 2002), government officials (Pasternack, 2002), professional associations (Fuchs et al., 2003), and many school psychologists (Reschly, 2003).
However, important questions remain unanswered, including (a) Do we have
validated intervention models and measures to ensure instructional validity
(Vaughn & Fuchs, 2003)? (b) Are there adequately trained personnel to implement an RTI model (Vaughn & Fuchs, 2003)? (c) Are parents and others prepared for the noncategorical approach to special education service delivery that
will likely be the result of an RTI model (Fuchs et al., 2003)? and (d) How many
children would be identified as nonresponders within an RTI model (Hale et
al., 2004)? As should be the case in an evidence-based practice, research data
are needed to answer these questions and to determine if RTI is worthy of a des-
ignation as scientifically based practice (Fuchs et al., 2003).
Some of these unanswered questions are being addressed by current
researchers, with some having sociological implications (e.g., parental acceptance of a noncategorical approach to special education) that make them difficult to answer before implementation. Kavale and Forness (2000) suggested
that meta-analytic research procedures were necessary when considering policy changes in special education. Thus, research on four existing large-scale RTI
models could provide data with which some important questions could be
answered. However, given the differential effectiveness found between group
problem-solving teams in research and in practice (Burns & Symington, 2002),
a similar comparison of the effectiveness of RTI models used in research to
those that exist in practice seems necessary to determine if the effectiveness of
a model differs according to the setting in which it is used and to determine in
which settings the models are most effective. Therefore, the purpose of the current study was to review existing research on RTI and the four specific models
of RTI currently in practice (Heartland, IBA, IST, and MPSM). This metaanalysis was designed to address three questions:
Downloaded from http://jpa.sagepub.com at UNIV OF UTAH on August 12, 2008
© 2005 SAGE Publications. All rights reserved. Not for commercial use or unauthorized distribution.
384
the large-scale RTI models currently in practice
those
as compared
developed for research?
2. Does RTI lead to improved systemic and student outcomes?
3. On average, what percentage of the student population was determined to have a disability under RTI?
1. How effective
are
to
It should be noted that although the purpose of meta-analytic research is
often to compare different approaches, the current study explored only the
effectiveness and outcomes associated with RTI and made no attempt to compare RTI to traditional special education or any other service delivery model.
METHOD
Data Collection
The PsycINFO, ERIC, and Education Abstracts databases were searched for
articles between October 11th and 15 th , 2004. This search for articles was not
restricted by any a priori specified date range. The following terms were
searched for, with the subsequent results listed in parentheses: response (and
responsiveness) to intervention (145 hits); response (and responsiveness) to
instruction (108); Heartland model (35); Heartland (108); intervention-based
assessment (18); instructional support team (137); and Minneapolis problemsolving model (23). Articles that did not directly address RTI or one of the four
models were eliminated, leaving 21 total articles. Next, the references of the
relevant studies were examined and potentially applicable entries that were not
yet identified were collected. This increased the potential pool to 31 articles.
Finally, articles identified through the search and reference lists were narrowed
by comparing them to the following criteria:
1. The
study implemented
one
an
intervention
(e.g.,
intensive remedial
al., 2001) or a systemic intervention (e.g.,
of the four problem-solving models) with children experienc-
instruction; Torgesen
et
ing academic difficulties or identified as learning disabled.
2. The study provided measures of either individual student learning,
such as postintervention and/or growth assessments, or systemic
outcomes (e.g., number of children identified as LD).
3. The study used a unit of analysis that was either the individual students or school buildings. Thus, studies that provided school district or statewide data were not used unless the data could be converted to the school building unit level.
4. The study included at least one between-group comparison and/
or at least one within-group comparison of the outcomes.
5. The study presented quantitative data that could be used to compute effect sizes. Means and standard deviations for both experimental and control groups, or pre- and postimplementation, were
necessary. However, the study was also included if enough data
were provided to compute the necessary means and standard deviations or if statistical analyses provided enough data to compute
an effect size.
6. The study was written in
English.
Downloaded from http://jpa.sagepub.com at UNIV OF UTAH on August 12, 2008
© 2005 SAGE Publications. All rights reserved. Not for commercial use or unauthorized distribution.
385
Of the articles found through various search
lists, 21 met the inclusion criteria and
erence
and by perusing refincluded in the meta-
engines
were
analysis.
Categorization
Articles
of Studies
categorized according to three criteria. First, studies were categorized
descriptive of one of the four RTI models already in practice
(Heartland, IBA, IST, and MPSM, as identified by Fuchs et al., 2003) or of an
intervention model developed and implemented for research purposes.
Approximately half of the studies (11 of 21 ) addressed one of the existing RTI
as
were
either
models, with 2 that addressed Heartland, 3 that addressed IBA, 4 that
addressed IST, and 2 that addressed MPSM. The remaining 10 studies examined interventions implemented for research purposes.
Burns and Symington (2002) found large effects for prereferral group
problem-solving models using various student and systemic outcomes. Given
the demonstrated relevance of this distinction, these two categories were also
used to assign current studies into one of two groups. Measures that were
placed into the student outcomes group included assessments of academic
skill, estimates of growth in a particular skill, and observations of time on task
and task completion related to academic interventions. Systemic variables
included referrals to and/or placements in special education, student time in
special education services, and number of students retained in a grade. It
should be noted that some studies addressed more than one effect. Thus, some
studies received more than one designation, which resulted in 11 (52.4%)
being classified as student outcomes and 14 (66.7%) as system outcomes.
Finally, data that were aggregated, presented, and/or analyzed with students as the unit of analysis (e.g., average reading score among 138 students)
were classified as student units, and those that examined schools as the unit of
analysis (e.g., mean number of new placements into special education among
the 36 participating school buildings) were classified as school units. Studies
could again receive more than one designation if more than one effect was
examined. Therefore, 13 (61.9%) were classified as school units and 12
(57.1 %) were classified as student units. Analyses were not conducted based on
this variable alone, but these data were used to compute effect sizes within comparable units of analysis.
A second person, who was a school psychology doctoral candidate, also
coded 66% of the studies in order to assess interrater consistency. The number
of studies that were placed into the same category by both coders was divided
by the total number of studies. The result of this calculation demonstrated
100% agreement.
Effect Size Calculation
The statistic of interest in
meta-analyses
is the effect size
Valentine, & Charlton, 2000). Cohen’s d (Cohen, 1988)
(ES; Cooper,
computed by subthe mean of the control group from the mean of the experimental
group, or the mean of the pretest from the mean of the posttest. Next, the
was
tracting
Downloaded from http://jpa.sagepub.com at UNIV OF UTAH on August 12, 2008
© 2005 SAGE Publications. All rights reserved. Not for commercial use or unauthorized distribution.
386
product was divided by the pooled standard deviation of the two groups ((SDI
+ SD2) /2) in order to express the difference between measurements in a common standard deviation (Cooper et al., 2000). Redundancy within studies was
addressed by pooling effect sizes among data that represented the same effect.
For example, a study may use multiple measures of reading (e.g., phoneme segmentation fluency and oral reading fluency), present means of subscales from
the same tool, or present data for referrals to and placements in special education. In these examples, the individual effect sizes were not independent and
a mean ES was computed.
After computing the ES and the measures of central tendencies for the various groupings, the aggregated data were used to compute an unbiased estimate of effect (UEE) using the formula outlined by Hedges (1982). The UEE
is a weighted estimator of effect using d and the sample size for each individual study. Thus, UEE was computed only if the aggregated data were all from
the same unit of analysis (school or student).
RESULTS
As stated above, 25 total effect sizes were computed. These effect sizes
ranged from 0.18 to 6.71, with a mean ES of 1.49 (SD 1.43) and a median ES
of 1.09. Computing an UEE weighting these results according to sample size
was inappropriate at this point given the differing units of
analysis (i.e., student
=
and school).
The variance of the 25 effect sizes was examined by computing a
Mahalanobis D to identify significant outliers. This examination led to the
removal of one effect size that was a substantial outlier in the positive direction.
Recalculation of the remaining 24 effect sizes indicated a range from 0.18 to
3.04. The mean ES for these data was 1.27 (SD .94), with a median ES of 1.02.
The remaining sections will report results for the data summarized by these 24
effect sizes.
Table 1 displays the mean, median, and standard deviations of the effect
sizes associated with the variables of interest to the current study. Eight studies
examined RTI models designed by university faculty and implemented for
research purposes, and 16 presented data from existing RTI models in the
field, both of which led to mean effect sizes greater than 1.0, but researchimplemented RTI models led to a median ES of .86. The UEE for studies of
existing RTI models was 1.42, and 0.92 for those implemented by university faculty for research.
=
Table 1
Unweighted Mean and Median Effect Sizes for Categories and Total
Downloaded from http://jpa.sagepub.com at UNIV OF UTAH on August 12, 2008
© 2005 SAGE Publications. All rights reserved. Not for commercial use or unauthorized distribution.
387
Student and systemic outcomes were also examined and resulted in mean
and median effect sizes that ranged from 0.72 to 1.53. The systemic effects were
approximately one third larger than the student effects. Unbiased estimates of
effect for these two variables were 1.02 for student and 1.54 for systemic outcomes. Thus, the UEE was larger for systemic outcomes at a magnitude that was
slightly smaller but similar to d.
A further comparison between university-based and field-based RTI models
was conducted by examining student and systemic outcomes of these studies.
RTI models currently implemented in the field ( n 11) led to a mean ES of
1.73 (SD 0.99) for systemic variables but resulted in a mean ES of 0.62 (SD
0.33) for student outcomes ( n 5). A reverse relationship was noted for RTI
models implemented by university faculty for research, with a mean ES of .47
(SD .07) for systemic outcomes (n 2) and 1.23 (SD = .95) for student outcomes ( n
6). UEE was computed for these variables as well. Field-based RTI
models resulted in a UEE of 0.94 for student outcomes and 1.80 for systemic
outcomes. RTI models implemented for research led to a UEE of 1.14 for student outcomes and 0.47 for systemic outcomes.
Concern was expressed that using an RTI approach for LD diagnoses would
lead to a dramatic increase in the number of children identified as nonresponders, and therefore labeled as LD (Hale et al., 2004). The current study
examined this by recording percentages of nonresponders within the included
studies. Eleven studies reported a percentage of children who were classified as
nonresponders. The range of nonresponders was from a low of 2.7% to a high
of 44.0%, with an average of 19.8% (SD 12.5). This number represents the
percentage of children who were identified with learning difficulties and participated in the intervention but did not meet the individual study’s operational definition of adequate responsiveness to the intervention. A 12th study
(Torgesen et al., 2001) reported 68.6% nonresponders, but participants were
previously identified as LD according to traditional methods and were already
receiving special education services. Thus, the results of Torgesen et al. suggest
that 31.4% of students already identified with LD responded to an RTI
=
=
=
=
=
=
=
=
approach.
In addition to calculating the percentage of nonresponders within the
study, four studies that used existing RTI models (IST and MPSM) reported the
percentage of the student population within the schools that was referred to or
placed into special education. In those studies reporting referral rates, an average of 1.26% (SD 0.65) of the student population was referred for a special
education eligibility assessment. In those studies reporting placement rates, an
average of 1.68% (SD 1.45) of the population was placed into special education. It should be noted that these studies examined referral to or placement
in special education and did not necessarily examine both. Finally, the same
four studies also reported the percentage of the student population that was
referred to the problem-solving team within the RTI model, the average of
=
=
which was 5.98% ( SD 2.97). Given that approximately 6% of the student population participated in the RTI model and less than 2% were then later considered for special education, it seems that on average just over 4% of the student population benefited from RTI, but the exact extent of the influence of
RTI remains uncertain.
=
Downloaded from http://jpa.sagepub.com at UNIV OF UTAH on August 12, 2008
© 2005 SAGE Publications. All rights reserved. Not for commercial use or unauthorized distribution.
388
DISCUSSION
study investigated the effectiveness of large-scale RTI models
as compared to those developed for research and examined systemic and student outcomes. Both field-based and university-based RTI models led to strong
effects, with the former stronger than the latter, but larger effects were found
for systemic rather than student outcomes. These results suggest the following
answers to the research questions posed.
The current
1. How effective
pared
to
those
are
the
developed
large-scale
RTI models
currently
in
practice
as com-
for research?<’
One readily apparent finding in this research is the consistently strong
effects of RTI implementations currently in practice (field-based). Whether calculated using Cohen’s d or UEE, field-based efforts consistently demonstrated
stronger effects than university research-based efforts. These results seemingly
contrast with the stronger university-based results noted by Safran and Safran
(1996) and subsequently supported by meta-analytic research (Burns &
Symington, 2002). Previous research hypothesized that stronger universitybased results were probably due to infidelity in implementation (Burns &
Symington, 2002). One possible suggestion for the current finding is the longer
duration of implementation for these field-based efforts as compared to the
university-based models. These increased effects may reflect refining of the RTI
process at these sites over this extended period and could suggest that implementation and refinement over a period of years could increase the likelihood
of success. Implementation was used to hypothesize as to why university-based
prereferral teams were more successful than field-based teams and why fieldbased RTI models led to stronger effects than university-based models. Thus,
implementation appears to be a critical issue, but these are merely hypotheses
that require additional empirical scrutiny.
Another explanation of these findings may lie with the quasi-experimental
design of the studies. Such designs allow for threats to the internal validity of
studies that when not controlled may produce effects that can be confounded
with the effects of the experimental stimulus (in this case, field-based RTI models) (Campbell & Stanley, 1963). Illustrating the importance of randomized
controlled trials (RCT) in establishing causal validity, well-constructed RCTs
are listed as providing the best evidence of effects and included in the What
Works Clearinghouse category of studies &dquo;meeting evidence standards,&dquo; whereas even the best-constructed quasi-experimental designs are unable to attain
this level of prominence (What Works Clearinghouse, 2004). A tangible example of uncontrolled confounding effects would be if schools that chose to
implement RTI models were composed of staff already more committed to prereferral intervention.
2. Does RTI lead to
improved systemic and student outcomes?
This study clarified that sites implementing RTI had both improved systemic and student outcomes (UEE 1.54 and 1.02, respectively). An examination by type of study and outcome revealed that field-based efforts had slightly
=
Downloaded from http://jpa.sagepub.com at UNIV OF UTAH on August 12, 2008
© 2005 SAGE Publications. All rights reserved. Not for commercial use or unauthorized distribution.
389
smaller improvements in student outcomes (but still substantial at UEE 0.94),
but systemic outcomes that were over three times larger than research-developed efforts (UEE 1.80 vs. 0.47, respectively). Finding that both systemic and
student outcomes improved with an RTI model in use is a promising sign. The
tremendous difference in systemic outcomes between field-based and researchbased efforts may, again, be a function of length of implementation. Given that
student outcomes are comparable, it would appear that researchers should
focus on this difference in future examinations by gathering data to clarify
potential explanations for this finding.
Cohen (1988) classified an effect size of .80 or larger as representing a
strong effect. All but two mean and median effect sizes exceeded .80, with one
being .72. Thus, the RTI models previously researched demonstrated strong
effects in improving student learning and systemic variables. Although there is
some controversy regarding the RTI approach to identifying LD (Fuchs et al.,
2003; Gresham et al., 2004), the strong effects noted in the current study suggest reason for optimism.
=
=
3. On average, what
have
a
disability
percentage of the student population
was
determined to
under RTI?
An additional variable of interest to the study is the percentage of the student population who were identified as LD within an existing RTI model. On
average, less than 2% of the student population was identified as LD among
studies examining field-based RTI models. Previous estimates of LD prevalence
concluded that 5% of the student population experienced a learning disability
(Lerner, 2003). The Twenty-Fourth Annual Report to Congress on the Implementation
of the Individuals with Disabilities Education Act (United States Department of
Education, 2002) reported that 2,887,217 school-aged children were identified
with a learning disability, which equals 5.7% of the total estimated student population in this country. Although it is difficult to predict the effect a federal mandate for RTI would have on LD prevalence nationwide, it seems that large-scale
implementation of various RTI models led to fewer students identified as LD.
Thus, concerns about large numbers of children identified as nonresponsive
leading to dramatic increases in LD (Hale et al., 2004) were not validated by the
current data. However, the definition of nonresponsiveness could have varied
between models, and conclusions based on these data should be made cau-
tiously.
Vellutino et al. (1996) presented data relevant to the LD prevalence question. In attempting to elicit the academic characteristics of 118 students who
emerged as reading disabled using exclusionary criteria only, Vellutino et al.
found that early and labor-intensive intervention effectively discriminated
between difficult-to-remediate and readily remediated readers. In addition, the
use of response to early intervention to distinguish readers resulted in only 3%
of students from the population scoring below the 30th percentile after one
semester of remediation versus 9% using exclusionary only criteria (Vellutino
et al. ) . Moreover, a subsection of the group identified as poor readers not only
improved their reading as a function of the intervention but also neared normal levels of reading and consistently performed at this higher level relative to
Downloaded from http://jpa.sagepub.com at UNIV OF UTAH on August 12, 2008
© 2005 SAGE Publications. All rights reserved. Not for commercial use or unauthorized distribution.
390
other groups of poor readers. These findings suggested not only that RTI
would not increase the number of students receiving special education services but could also serve to reduce those numbers. Likewise, the Vellutino et al.
findings seemed to indicate that RTI did this by elucidating group differences
among poor readers, thereby supporting those readers who are capable of performing at near-average levels to do so. These data also suggest an entwined
relationship between student skill growth and LD diagnosis.
Limitations
Several limitations should be explicated regarding this study. As one might
expect, some searches using derivations of problem-solving model names (e.g.,
problem solving) produced extremely lengthy and seemingly irrelevant lists
(e.g., 8,748 records for &dquo;problem solving&dquo;). Researchers conducted cursory
examinations of these lists but reserved closer examinations for searches using
complete model names or partial model names with more manageable lists of
records (e.g., &dquo;Heartland&dquo;). It is possible, but not probable, that a relevant
study could have been found within the lengthy results of such a search and not
within a search using the full model name.
In the current analysis, two of the field-based models use RTI as a prereferral process (Ohio and Pennsylvania), whereas the other two models (Heartland
and Minneapolis) use RTI to determine special education eligibility. The current examination of a wide range of field-based RTI models combines outcomes that may provide other useful information if evaluated separately.
None of the articles and data used in this meta-analysis was solicited per se,
but relevant unpublished data referenced in articles were requested from those
researchers. Whereas this effort was an attempt to offset any &dquo;file drawer&dquo;
research shortcomings and did result in the inclusion of unpublished data,
researchers could extend this effort by actively soliciting unpublished research.
Finally, some caution should be exercised when interpreting these data
given the relatively large standard deviations for the effect sizes. Removing outlier data reduced the size of the effect sizes, but two standard deviations were
still as large as the median effects. Therefore, conclusions from those two categories should be made carefully. Moreover, there was a relatively small number of studies for some variables (e.g., four studies examining incidence levels
for LD), which suggested the need for caution when interpreting those findings as well.
Suggestions for Future Research
Perhaps the most important result of any research is to point out the need
for additional research. It seems that the effect of RTI on LD prevalence needs
additional empirical scrutiny using controlled rather than quasi-experimental
studies. Moreover, controlled studies of systemic outcomes in general appear
warranted. Specifically, RCT studies are needed to examine the effect of RTI
implementation on referrals to and placements in special education, student
time in special education services, and number of students retained in a grade.
Finally, implementation fidelity remains a crucial aspect of any intervention or
system of interventions. Therefore, the effect of fidelity in implementation on
RTI appears
to
be crucial data.
Downloaded from http://jpa.sagepub.com at UNIV OF UTAH on August 12, 2008
© 2005 SAGE Publications. All rights reserved. Not for commercial use or unauthorized distribution.
391
As stated earlier, almost three million children in this country are diagnosed with a learning disability and there is a long history of controversy surrounding the diagnosis. It is difficult to speculate how such a controversial
approach was implemented or what would happen if it changed. Scriven
(1983) stated in his response to Glass’s (1982) meta-analysis of special education research, &dquo;I cannot say what I think the pessimist could say about our
research and our practice in special education at this point, but I think the
optimist could say that we have a wonderful opportunity to start all over&dquo;
(p. 84). That statement was made over 20 years ago and although starting all
over again may be an extreme
position, it seems the current reauthorization of
IDEA presents an unprecedented opportunity to examine current practice and
reform as needed. However, reforms should occur only after extensive research
regarding proposed initiatives is conducted. This will help ensure that new
approaches will not experience the same level of controversy as their predecessors.
REFERENCES
marked with an asterisk indicate
studies included in the meta-analysis.
Aaron, P. G. (1997). The impending
demise of the discrepancy formula.
Review of Educational Research, 67, 461-
References
502.
Burns, M. K., & Symington, T. (2002). A
meta-analysis of pre-referral intervention teams: Student and systemic outcomes.Journal of School Psychology, 40,
437-447.
H. M., Valentine, J. C., &
Charlton, K. (2000). The methodology of meta-analysis. In R. Gersten,
E. P. Schiller, & S. Vaughn (Eds.),
Cooper,
Contemporary special education research:
Syntheses of the knowledge base on critical
instructional issues (pp. 263-280).
Mahwah, NJ: Erlbaum.
Deno, S. L., Fuchs, L. S., Marston, D., &
Shin, J. (2001). Using curriculumbased
Burns, M. K., Tucker, J. A., Hauser, A.,
Thelen, R., Holmes, K., & White, K.
(2002). Minimum reading fluency
necessary for comprehension: A
potential criterion for curriculumbased assessments. Assessment for
Effective Intervention, 28, 1-7.
Burns, M. K., Vanderwood, M., & Ruby, S.
(2005). Evaluating the readiness of
prereferral intervention teams for use
in a problem-solving model: Review
of three levels of research. School
Psychology Quarterly, 20, 89-105
rate
D. T., & Stanley, J. C. (1963).
Experimental and quasi-experimental
designs for research. Boston: Houghton
Campbell,
Mifflin.
Cohen, J. (1988). Statistical power analysis
nd ed.).
for the behavioral sciences (2
Hillsdale, NJ: Erlbaum.
to establish
standards
for
students
with
growth
disabilities.
School
learning
Psychology
Review Special Issue: Mini-series: Issues
in data-based decision making in special
measurement
education, 30, 507-524.
Ellis, A. K. (2001). Research on educational
innovations (3
rd ed.). Larchmont, NY:
Eye on Education.
*Foorman, B. R., Francis, D. J., Fletcher, J.
M., Schatschneider, C., & Mehta, P.
(1998). The role of instruction in
to read: Preventing reading
failure in at-risk children. Journal of
Educational Psychology, 90, 37-55.
Fuchs, D., Mock, D., Morgan, P. L., &
Young, C. L. (2003). Responsivenessto-intervention : Definitions, evidence,
and implications for the learning disabilities construct. Learning Disabilities
Research & Practice,
18, 157-171.
learning
Downloaded from http://jpa.sagepub.com at UNIV OF UTAH on August 12, 2008
© 2005 SAGE Publications. All rights reserved. Not for commercial use or unauthorized distribution.
392
L. S. (2003). Assessing intervention responsiveness: Conceptual and
technical issues. Learning Disabilities:
Research &
Practice, 18, 172-186.
Fuchs, L. S., & Fuchs, D. (1998). Treatment validity: A unifying concept for
reconceptualizing the identification
of learning disabilities. Learning
Disabilities Research &
Practice, 13, 204219.
*Fuchs,
*Fuchs, L. S., Fuchs, D., &
Prentice, K.
mathematinstruction:
problem-solving
Comparing students at risk of mathematics disability with and without risk
of reading disability. Journal of
(2004). Responsiveness
to
ical
, 293-306.
Learning Disabilities, 37
Fuchs, L. S., Fuchs, D., & Speece, D. L.
(2002). Treatment validity as a unify-
ing construct for identifying learning
disabilities.
Learning Disability
Quarterly, 25, 33-46.
Glass, G. (1983). Effectiveness of special
education. Policy Studies Review, 2, 6578.
Goor, M. B. (1995). Leadership for special
education administration: A case-based
approach. Fort Worth, TX: Harcourt
Brace.
Gresham, F. (2001, August). Responsiveness
to intervention: An alternative approach
to the identification of learning disabilities. Paper presented at the Learning
Disabilities Summit: Building a
Foundation
for
the
Future,
Washington, DC.
Gresham, F. M. (2002). Responsiveness
to
intervention: An alternative approach
to the identification of learning disabilities. In R. Bradley & L. Danielson
(Eds.), Identification of learning disabilities : Research to practice. The LEA series
on special education and
disability (pp.
467-519). Mahwah, NJ: Erlbaum.
Gresham, F. M., Reschly, D., Tilly, W. D.,
III., Fletcher, J., Burns, M. K., Christ,
T. Prasse, D., Vanderwood, M., &
Shinn, M. (2004). Comprehensive
evaluation of
learning disabilities: A
response-to-intervention perspective.
(4), 34-35.
Communiqu&eacute;, 33
Hale, J. B., Naglieri, J. A., Kaufman, A. S.,
& Kavale, K. A. (2004). Specific learning disability classification in the new
Individuals with Disabilities Education
Act: The danger of good ideas. The
School Psychologist, Winter, 6-13, 29.
*Hartman, W. T., & Fay, T. A. (1996). Costeffectiveness of instructional support
teams in Pennsylvania. Journal of
Educational Finance, 21, 555-580.
*Heartland (2004). Heartland AEA 11
annual progress report. Retrieved on
10/15/04 from http://www.aea11.
k12.ia.us/downloads/2004apr.pdf.
Hedges, L. (1982). Estimation of effect
size from series of independent
experiments. Psychological Bulletin, 92,
490-499.
*Ikeda, M. J., & Gustafson, J. K. (2002).
Heartland AEA 11 ’s problem solving
process: Impact on issues related to special
education. Research report no. 200201. Johnston, IA: Heartland Area
Education Agency 11.
Ikeda, M. J., Tilly, D. W., Stumme, J.,
Volmer, L., & Allison, R. (1996).
Agency-wide implementation of problem-solving consultation: Foundations,
current implementation, and future
directions. School Psychology Quarterly,
11, 228-243.
Kavale, K. A., &
Forness, S. R. (2000).
decisions
in special education:
Policy
The role of meta-analysis. In R.
Gersten, E. P. Schiller, & S. Vaughn
(Eds.), Contemporary special education
research: Synthesis of the knowledge base
on critical instructional issues (pp. 281326). Mahwah, NJ: Erlbaum.
*Kovaleski, J. F., Gickling, E. E., &
Morrow, H. (1998). High versus low
implementation of instructional support teams: A case for maintaining
program fidelity. Remedial and Special
Education, 20, 170-183.
*Kovaleski, J. F., Tucker, J. A., & Duffy, D.
J. (1995). School reform through
instructional
The
support:
Initiative
(Part I).
Pennsylvania
(8) (insert).
Communiqu&eacute;, 23
Downloaded from http://jpa.sagepub.com at UNIV OF UTAH on August 12, 2008
© 2005 SAGE Publications. All rights reserved. Not for commercial use or unauthorized distribution.
393
Lerner, J.
W.
(2002). Learning disabilities:
Theories, diagnosis, and teaching strateth ed.). Boston: Houghtongies (8
Mifflin.
*Marston, D., Muyskens, P., Lau, M., &
Canter, A. (2003). Problem-solving
model for decision making with highincidence disabilities: The Minneapolis
experience. Learning Disabilities Research & Practice, 18
(3), 187-200.
*McMaster, K. L., Fuchs, D., Fuchs, L. S., &
Compton, D. L. (2005). Responding to
nonresponders: An experimental field
trial of identification and intervention
methods. Exceptional Children, 71, 445463.
*McNamara, K. (1998). Adoption of intervention-based assessment for special
education. School Psychology International, 19, 251-266.
*McNamara, K., & Hollinger, C. (1997).
Intervention-based assessment: Rates
of evaluation and eligibility for specific learning disability classification.
Psychological Reports, 81, 620-622.
*McNamara, K., & Hollinger, C. (2003).
Intervention-based assessment: Evaluation rates and eligibility findings.
Exceptional Children, 69, 181-194.
Meyer, M. S. (2000). The ability-achievement discrepancy: Does it contribute
to an understanding of learning disabilities ? Educational Psychology Review
Special Issue: Intelligence Part II, 12,
315-37.
Minneapolis
Public
Schools
(2001).
Problem-Solving Model: Introduction for
all staff. Minneapolis, MN: Author.
Myers, P. I., & Hammill, D. D. (1990).
Learning disabilities: Basic concepts
assessment practices and instructional
th ed.). Austin, TX: PROstrategies (4
ED.
Pasternack, R. (2002, June). Learning disabilities and early intervention strategies:
How to reform the special education referral and identification process. Testimony
presented to the United States House
of Representatives’ Subcommittee on
Education Reform Committee on
Education and the Work Force,
Washington, DC.
President’s Commission on Excellence in
Special Education (2001). A new era:
Revitalizing special education for children
and their families. Washington, DC:
U.S. Department of Education.
Reschly, D. J. (2003, December). What if
LD identification changed to reflect
research findings ? : Consequences of LD
identification changes. Paper presented
at the Responsiveness-to-Intervention
Symposium, Kansas City, MO.
*Reschly, D. J., & Starkweather, A. R.
(1997). Evaluation of an alternative special education assessment and classification program in the Minneapolis Public
Schools. Minneapolis, MN: Minneapolis Public Schools.
Reschly, D. J., & Ysseldyke, J. E. (2002).
Paradigm shift: The past is not the
future. In A. Thomas & J.
Grimes
(Eds.), Best practices in school psychology
th ed., pp. 3-21). Bethesda, MD:
(4
National
Association
of
School
Psychologists.
*Rock, M. L., &
Zigmond, N. (2001).
Intervention assistance: Is it substance or symbolism. Preventing School
Failure, 45, 153-161.
Safran, S. P., & Safran, J. S. (1996).
Intervention assistance programs and
prereferral teams: Directions for the
twenty-first century. Remedial and
Special Education, 17, 363-369.
Scriven, M. (1983). Comments on Gene
Glass. Policy Studies Review, 2, 79-84.
Shapiro, E. S. (2004). Academic skill problems : Direct assessment and intervention
rd ed.). New York: Guilford.
(3
Shinn, M. R., Walker, H. M., & Stoner, G.
(2002). Interventions for academic and
behavior problems nd
(2 ed.). Bethesda,
MD: National Association of School
Psychologists.
D. L, & Case, L. P. (2001).
Classification in context: An alternative
*Speece,
approach to identifying early reading
disability. Journal of Educational
Psychology, 93, 735-749.
Downloaded from http://jpa.sagepub.com at UNIV OF UTAH on August 12, 2008
© 2005 SAGE Publications. All rights reserved. Not for commercial use or unauthorized distribution.
394
Speece,
D. L.,
Case, L. P., & Molloy, D. E.
(2003). Responsiveness
to
general
education instruction as the first gate
to learning disabilities identification.
Learning Disabilities Research &
Practice, 18, 147-156.
Swanson, H. L. (1999). Reading research
for students with LD: A meta-analysis
in intervention outcomes.
Learning Disabilities, 32,
Journal of
504-532.
Swanson, H. L. (2000). What instruction
works for students with learning disabilities ? Summarizing the results
from a meta-analysis of intervention
studies. In R. Gersten, E. P. Schiller, &
S. Vaughn (Eds.), Contemporary special
education research: Syntheses of the knowledge base on critical instructional issues
(pp. 1-30). Mahwah, NJ: Erlbaum.
Swanson, H. L, Hoskyn, M., & Lee, C.
(1999). Interventions for students with
learning disabilities: A meta-analysis of
treatment
outcomes.
New
York:
Guilford.
Telzrow, C. F., McNamara, K., & Hollinger,
C. L. (2000). Fidelity of problem-solving implementation and relationship
to
student performance. School
Psychology Review, 29, 443-461.
Tilly, W. D., III. (2002). Best practices in
school psychology as a problem-solving enterprise. In A. Thomas & J.
Grimes (Eds.), Best practices in school
th
(4 ed., pp. 21-36).
Bethesda, MD: National Association
psychology
of School
Psychologists.
Tomasi, S. F., & Weinberg, S. L. (1999).
Classifying children as LD: An analysis of current practice in an urban setting. Learning Disability Quarterly, 22,
31-42.
*Torgesen, J. K., Alexander, A. W., Wagner,
R.
&
K., Rashotte, C. A., Voeller, K. K. S.,
T. (2001). Intensive remedial instruction for children with
severe reading disabilities: Immediate
and long-term outcomes for two
instructional approaches.Journal of
Learning Disabilities, 34, 33-58.
Conway,
R. K., Rashotte,
C. A., Rose, E., Lindamood, P.,
Conway, T., & Garvan, C. (1999).
Preventing reading failure in young
children with phonological processing disabilities: Group and individual
responses to instruction. Journal of
Educational Psychology, 91, 579-593.
United States Department of Education.
(2002). Twenty-fourth annual report to
congress on the implementation of the
Individuals with Disabilities Education
Act. Washington, D.C.: US Government
Printing Office.
Vaughn, S., & Fuchs, L. S. (2003). Redefining learning disabilities as inadequate response to instruction: The
promise and potential problems.
Learning Disabilities Research &
*Torgesen, J. K., Wagner,
Practice, 18, 137-146.
*Vaughn, S., Linan-Thompson, S., &
Hickman, P. (2003). Response to
instruction as a means of identifying
students with reading/learning disabilities. Exceptional Children, 69, 391409.
*Vellutino, F. R., Scanlon, D. M., Sipay, E.
R., Small, S., Chen, R., Pratt, A., &
Denkla, M. B., et al. (1996). Cognitive
profiles of difficult-to-remediate and
readily remediated poor readers:
Early intervention as a vehicle for distinguishing between cognitive and
experimental deficits as basic causes
of specific reading disability. Journal of
Educational Psychology, 88, 601-638.
What Works Clearinghouse. (2004). WWC
study review standards. Retrieved
November 19, 2004, from http://
www.whatworks.ed.gov/reviewprocess/study_standards_final.pdf.
Ysseldyke, J. E., Algozzine, B., & Epps, S.
(1983). A logical and empirical analysis of current practice in classifying
students as handicapped. Exceptional
Children, 50, 160-166.
& Thurlow,
M. L., (2000). Critical issues in special
education rd
(3 ed.). Boston: Houghton
Mifflin.
Ysseldyke, J. E., Algozzine, B.,
Downloaded from http://jpa.sagepub.com at UNIV OF UTAH on August 12, 2008
© 2005 SAGE Publications. All rights reserved. Not for commercial use or unauthorized distribution.