Contributions of Instructional
Practices to Reading Achievement
in a Statewide Improvement Program
JOHN T. G U T H R I E
W I L L I A M D. SCHAFER
CLARE VON SECKER
TERRY ALBAN
University of Maryland
increasingly critical. However, when the school is the unit
of analysis, relatively little quantitative data are available
t'or use by researchers or policy makers. When policy makers e~amine characteristics of successful schools, they usually report organizational variables such as ongoing staff
development, a system for monitoring performance, content
learning goals, and school governance structures (Fullan,
1995). However, management and governance variables are
associated weakly with student achievement outcomes in
quantitative studies. In an extensive meta-analysis of within-school and between-school variables affecting achievement, Walberg (1986) reported that general school governance and management rarely explained a significant
amount of variance when accounting for student background factors and instructional quality. Therefore, attention to specific characteristics of reading programs rather
than general characteristics of schools seems warranted.
When education professionals such as teachers and state
department officials were surveyed to determine knowledge
required by teachers to run effective school reading programs, they emphasized content-specific pedagogy
(Reynolds, 1995). That is, to improve reading, teachers
were expected to be experts in strategies particular to reading instruction. Furthermore, when districts attempt to
improve their school reading programs systemically, they
construct reading-specific goals, materials, and instructional approaches (Santa, 1997). In the present high-accountability environment, we need studies on characteristics of
school reading programs that reliably increase achievement
on high-stakes assessments such as standardized tests and
performance measures. A reason to focus on reading is that
effectiveness of schools is relatively content specific. Mandeville and Anderson (1987) showed that in one statewide
sample of elementary schools, school effectiveness depended on content area. After controlling for student prior
A B S T R A C T The amount and direction of school change in
reading achievement on a statewide, high-stakes performance
assessment were identified. School change in reading achievement with instructional practices reported by teachers in primary and intermediate grades in 33 schools was predicted.
Meta-analyses of the effects of instructional practices in reading on achievement in 6 areas--including reading, writing,
language use, mathematics, science, and social studies--were
conducted. In Grade 5, achievement in reading, science, mathematics, and writing increased significantly. Practices of integrated instruction and use of abundant texts and resources
were associated positively with change in achievement. Basal
emphasis and comprehension instruction were associated negatively with achievement change. In Grade 3, few effects were
observed. Findings are interpreted in light of existing models
of integrated instruction and the roles of teacher knowledge in
achievement and learning.
!
¢xr~
Q...
r~
S c h o o l reading programs are increasingly scrutinized by
educational administrators and policy makers. Among
the reasons for the increased attention are relatively low
reading achievement among 9-year-olds on the National
Assessment of Educational Progress (NAEP; Campbell,
Voelkl, & Donahue, 1997), reading failure as a perceived
barrier to progress in mathematics and science (O'Sullivan,
Reese, & Mazzeo, 1997), and the standards-based school
improvement programs in many states and districts (Feuer,
Holland, Green, Bertenthal, & Hemphill, 1998). States and
districts attempting to produce higher achievement in reading usually place sanctions for failure on schools in which
reading achievement is exceptionally low (Yen & Ferrara,
1997). Accountability pressure is not often directed at the
individual teacher nor the individual grade level but at the
elementary school as a whole (Kortez, Linn, Dunbar, &
Shepard, 1991).
These political pressures heighten the school as the unit
of analysis for reading programs. Understanding the characteristics of effective schoolwide reading programs is
Address correspondence to John Guthrie, 3304 Benjamin Building, University of Maryland, College Park, MD 20724.
211
• 212
achievement and socioeconomic status (SES), schools predictably increased mathematics achievement but not reading. In addition, Wienert and Helmke (1995) reported that
even though students change teachers through the years,
student relative achievement level is highly stable. Consequently, detecting effects on achievement of reading programs when the school is the unit of analysis poses considerable challenges.
Studies of school reading programs usually address a
subset of the population or the instructional objectives of
the school. For example, studies of remedial school reading
programs have been conducted (Bean, 1995) in which particular students identified as low achievers are provided
specialized, small-group instruction. Studies of ~'Success
For All" have been referenced as school reading programs
(Slavin, Kartweit, & Wasik, 1994), although they are usually focused on a subset of students who require extra help in
the primary grades. Falling broadly within the remedial category are programs for reading designed for an entire
school devoted to reading-disabled or learning-disabled students (Gaskins & Gaskins, 1997).
Schoolwide reading programs are often provided for regular students in the form of book clubs (Goatley, Brock, &
Raphael, 1995), cross-age tutoring (Fox, 1994), or afterschool enrichment activities (Tucker & Fraser, 1995). In
those cases, although all students may be eligible, the
instructional activities are supplementary to the core of
reading language arts teaching. Other schoolwide reading
programs consist of nongraded education (Tanner & Decotis, 1995) or magnet school programs. However, those items
are grouping factors rather than pedagogical factors expected to increase achievement.
An approach to studying school reading programs in the
1970s consisted of identifying exemplary schools (Purkey
& Smith, 1983; Wilder, 1977). In those studies, outlier
schools that achieved beyond expectation on the basis of
SES were identified and described. Characteristics of reading programs identified in those studies included the following: a high priority on reading goals, an abundance of
books and materials, a large amount of time scheduled for
reading instruction, teacher opportunities for staff development in reading instruction, an emphasis on accountability
in reading, and parent involvement in the reading program,
as well as teachers' decision making in the reading program
and instructional leadership in reading from the school principal. Although those characteristics are well known, the
path from mediocrity to excellence in the school reading
program is not obvious. Further, there are few data to document that those characteristics quantitatively discriminate
high-achieving schools from low-achieving schools. Consequently, the distinguishing features of effective school reading programs are relatively little understood.
The current political and public agenda calls for information on school reading instruction when that instruction
has the following characteristics: (a) It serves as the main
curriculum in reading and language arts, (b) it is provided
The Journal of Educational Research
by regular teachers, (c) all students in the regular population (although diverse) participate in the instruction, (d)
teaching is provided for the 9-month academic year, and
(e) instruction is provided within schools that are not selective for particular types of students. Given that identity for
the school reading program, data are needed on the characteristics that reliably increase reading achievement.
Although school reading programs have rarely been studied in their full complexity, individual variables likely to be
present in the entire school reading program have been
subject to separate examinations.
The predominate element associated with a school reading program is the basal reader series. In a national survey
of elementary, school teachers, Baumann, Hoftman, Moon,
and Duffy-Hexter (1998) found that 83% of the teachers
reported using the basal as their centerpiece in the reading
program. The basal was supplemented by trade books for
56% of the respondents. Despite their popularity, basal
reading series have limitations. For example, Miller and
Blumenthal (1993) found that basal programs undersupported reading comprehension skills such as understanding
the main idea and comprehending cause--effect relationships. They concluded that the materials and guidelines for
teachers were inadequate to elicit higher order cognitive
processes, initiate complex writing processes, or set tasks of
reading lengthy text sections. The limitations also were evident in a study by Baumann (1984) who found that direct
comprehension strategy instruction was superior to a basal
program for increasing reading comprehension skills. However, basal programs can be improved by combining them
with direct strategy instruction. Payne and Manning (1992)
reported that integrating direct comprehension strategy
instruction into a basal reader program produced higher
achievement on standardized reading tests than a basal program in isolation. Those findings led us to expect that emphasis on a basal program would not consistently lead to
increased reading achievement.
Comprehension strategy instruction is almost as widely
used as emphasis on a basal program. National surveys
reported that 98% of teachers in Grades 3-5 regularly provided strategy instruction (Baumann et al., 1998). That
instruction included an emphasis on summarizing, selfquestioning, self-monitoring, and using prior knowledge to
understand text. Brown, Pressley, Van Meter, and Schuder
(1996) reported that, for elementary students, cognitive
strategy instruction in reading has advantages over traditional basal-centered instruction. For instance, Collins
(1993) showed that reading comprehension strategy instruction was higher than a traditional instructional experience in
standardized tests of reading comprehension and also in
students' reports of self-perceived competence in using
strategies for reading and thinking. With adequate training
and extended experience in comprehension strategy instruc• tion, teachers report that they enjoy teaching and believe
students are learning from this intervention (Pressley,
Schuder, Bergman, & El-Dinary, 1992). However, learning
, March/April 2000 [Vol. 93 (No. 4)]
to provide strategy instruction is time consuming. Duf-fy
(1993) found thatr teachers require 4 years to progress from
novice to expert as strategy instructors. Because of the frequent empirical confirmations of the beneficial effects of
comprehension strategy instruction, we expected that this
quality of teaching would contribute positively to student
reading achievement.
A less prevalent but increasingly visible characteristic of
reading instruction is integration (Au, 1997: Santa, 1997).
Integrating reading instruction often begins with connecting
reading with writing and linking reading to learning in a
content domain such as science or social studies. In such
programs, teachers provide inquiry units in which students
read and write about a topic or conceptual theme (Bean.
1995). Limited investigations suggest that those integrations are effective. In one quasi-experimental comparison,
students participating in a year-long integration of science
and reading were rated higher than traditional students in
expository text comprehension, narrative text comprehension, use of reading strategies for learning from multiple
texts, and using new knowledge to solve problems (Guthrie
et al., 1998). In a related study, integration of reading strategy instruction with in-depth science teaching produced
higher reading achievement test scores than traditional
basal and science teaching (Romance & Vitale, 1992).
Those integrations included hands-on science activities as
well as a diversity of reading and writing activities; also,
instruction that merges narrative and informational books
around themes also produced higher reading comprehension than traditional instruction (Morrow, Pressley, &
Smith, 1995). Although integrations of reading and science
are frequently employed in middle schools (Bean, 1995;
Guthrie, Alao, & Rinehart, 1997: Lapp, Flood, & Ranckbuhr, 1995), experimental evidence has not been collected
at the middle school level. According to those findings and
other suggestive evidence (Guthrie et al., 1996), we expected that integrated reading language arts and content instruction would increase reading achievement.
Integrated reading instruction depends heavily on a printrich environment with multiple text resources. Unfortunately, using trade books and multimedia resources extensively
in reading instruction was reported by only 16% of elementary school teachers (Baumann et al., 1998). However, education policy makers regard using books and multimedia as
a critical issue (Reutzel, Sudweek, & Hollingsworth, 1994),
and limited evidence from international studies (Elley, 1992)
and NAEP comparisons (Campbell et al., 1997) suggests
that using a wide range of trade books for classroom instruction is associated with relatively high reading achievement.
Teachers who are likely to have students with relatively high
reading achievement use narrative and information books in
social studies, science, and literature in a wide range of difficulty levels that match students' reading levels. Those practices increase students' amount of reading (Guthrie et al.,
1995), which is a strong contributor to reading achievement
(Cunningham & Stanovich, 1997). Consequently, we
213
expected that using a wide range of books and resources for
-. reading instruction would increase readirig achievement.
The popular social constructivist paradigm of learning
suggests an important role for collaboration (Salomon,
1998). In that theoretical perspective, reading comprehension is regarded as an interpersonal rather than an intrapersonal construct. Students are encouraged to build meanings
from stories and information books on the basis of combining their own experience and the interpretations of peers
with the text. Whereas a range of qualitative investigations
has been reported (Allen & Reed, 1997), a few quantitative
studies also are available. Almasi (1995) conducted a true
experiment in which goal group students socially constructed meanings from narrative text across 12 weeks. They
were rated higher in reading comprehension than comparison students who spent an equal amount of time reading but
did not participate in intensive interpersonal interchange.
Similar results were found for reading science information
books in a collaborative, social constructivist setting
(Meloth & Deering, 1994 ). Further, traditional structures of
cooperative learning in reading in which students are
rewarded in a competitive environment for interpersonal
support within the group are well established (Stevens,
Madden, Slavin, & Famish, 1987). However, implementation of collaborative learning is often incomplete. Students
are often off task from reading and experience lower
engaged reading time (Berliner, 1986). Therefore, we were
equivocal regarding the probable effects of collaboration on
reading achievement.
Writing activities within the reading program are frequently reported by outstanding teachers (Pressley, Yokoi,
Rankin, McDonald, & Minstretta, 1997). That practice may
be well advised because reading and writing appear to be
mutually influential. Shanahan and Lomax (1986) reported
path analytic data suggesting that the reading skills of word
analysis, vocabulary, and passage comprehension are reciprocally related to the writing skills of spelling, sentence
structure and composition, and written story organization.
Furthermore, students randomly assigned to a treatment
condition of story composition were rated higher than a control group in measured reading comprehension following
the treatment (Denner, McGinley, & Brown, 1989). Writing
instruction is often blended with reading instruction in both
primary grades (Bond. Ross, Smith, & Nunnery, 1995) and
intermediate grades (Pressley et al., 1997). Although the
effect of the writing component is seldom disentangled from
the reading component in instruction, we expected writing
instruction to increase reading achievement.
The important instructional characteristics in a school
reading program shift from primary level (Grades 1-3) to
intermediate level (Grades 4-5). Surveys of teaching practice show that intermediate teachers emphasize comprehension strategies, vocabulary, response to literature, comprehension monitoring, and trade books for instruction. In
contrast, primary teachers emphasize phonics and word
identification, journal writing, spelling, oral language activ-
214
ities, and reading aloud to children (Baumann et al., 1998).
Confirming their emphasis, teachers' staff development
needs are perceived to be radically different for primary and
intermediate teachers (M. E Doan & Doan, 1998). Consequently, we expected that the following variables would
increase achievement in Grade 5 but not in Grade 3: (a)
comprehension instruction; (b) integrated reading, writing,
and content: (c) text and resource diversity: (d) collaboration, and re) writing. We were equivocal about the basal
emphasis because there was lack of data in empirical literature for the primary grades.
Substantial methodological problems surround the
attempt to identify characteristics of school reading programs that increase reading achievement. One challenge is
that survey data often show negative correlations between
student achievement and instructional characteristics. For
instance, Lundberg and Linnakyla (1993) found negative
correlations between comprehension strategy instruction
and student reading achievement both across countries and
within countries. One interpretation of the negative correlation is that direct strategy instruction decreases achievement. Another interpretation is the compliance effect. Lundberg and Linnakyla (1993) reported:
Negative coefficients would be obtained if teachers of poorly achieving students had a stronger tendency to give high
frequency responses to items on the teaching questionnaire
but reflecting provision of [help] with those skills which better readers acquired earlier. Thus the high ratings by teachers
of low achieving students reflect adaptation to students'
needs. (p. 86)
A simple correlation is ambiguous and cannot be interpreted either as a causal effect or a compliance effect.
In this study, we attempted to characterize improving
schools. We collected data in 2 successive years on the basis
of the same statewide performance assessment. Schools that
were increasing in achievement were identified and compared with schools that were decreasing in achievement.
Using instructional characteristics of the school reading
program, we predicted school change. We did not attempt to
predict school achievement level in an absolute sense, but
school increase in a relative sense. An instructional characteristic typical of classrooms in which student achievement
is increasing may be operating as a causal variable.
It is possible that an instructional characteristic (e.g.,
comprehension strategy instruction) may be occurring in a
school reading program because a teacher perceives student
need for comprehension strategies. Under that condition,
teachers would provide more comprehension strategy
instruction for high-achieving students because of the
teachers' perception that it was needed by the students. To
examine that possibility, we conducted a covariance analysis (ANCOVA) in this study. We adjusted Grade 5 students'
achievement change scores for the level of those same students in Grade~3. If high comprehension strategy instrucron
were associated with a high achievement increase of Grade
5 students after prior achievement was equated, we had a
T h e J o u r n a l of E d u c a t i o n a l R e s e a r c h
source of evidence that the instructional variable was having a causal effect on the achievement outcome. If it were
true that teachers gave comprehension strategy instruction
to highly achieving students because they perceived that the
students needed it. then the ANCOVA should yield no significant association of comprehension strategy instruction
and reading achievement outcomes. The dual approaches of
(a) predicting change in achievement and (b) predicting
reading achievement change controlled for previous
achievement levels were intended to increase the plausibility of causal attributions for the instructional variables.
Method
Schools and Teachers
All of the districts in Maryland were invited to participate in our study. Three districts agreed to ask all of their
teachers in all of their elementary schools to participate;
33 schools cooperated. The districts were middle income,
suburban and did not represent the low-income urban nor
high-income suburban districts in the state. The accountability coordinator in each district was invited to participate in a study of the National Reading Research Center
for "examining how school-level reading instruction is
related to change in measured a c h i e v e m e n t . . , on schoollevel achievement on the MSPAP content area scores." All
Table 1.mBackground Characteristics of Intermediate and
Primary Teachers
Teacher
Variable
Intermediate
n
%
Primary
n
%
34
33
33
---
Grade
1
2
3
4
5
Gender
Female
Male
Degree
BA/BS
BA/BS + 15
MA/MEd.
MA/MEd. + 30
Ethnicity
Asian
Hispanic
Pacific Islander
African American
Native American
Caucasian
Years teaching
1-5
6-t6
17-30
--
--
--75
79
--49
51
78
75
77
---
139
12
90
8
223
5
97
2
35
45
61
1I
23
29
40
7
39
67
104
16
17
29
45
7
0
6
1
1
0
137
0
4
1
1
0
89
1
11
I
1
1
201
54
45
54
35
29
35
71
80
76
.4
5
.4
.4
.4
87
31
35
33
March/April 2000 [Voi. 93 (No. 4)]
classroom teachers (K-6).were asked to complete a questionnaire on reading instriaction.
Questionnaires were returned by 545 teachers. The teachers were evenly distributed across Grades 1-5 (see Table 1).
Ninety-seven percent of the primary teachers and 90% of
the intermediate teachers were women. For the total group.
education levels consisted of bachelor's degrees = 18%,
bachelor's plus 15 credits = 25%, master's degrees = 45%.
master's + 30 = 8%: ethnicities included Caucasian = 84%,
Hispanic = 5%, African American = 4%. Of the primary.
teachers, 35% had 17-30 years of teaching: 33% of the
intermediate teachers had 17-30 years of teaching. Teacher
characteristics are reported in Table 1. For both intermediate and primary teachers, a large majority were female Caucasians with a master's degree. They had a wide range of
years teaching and evenly represented all grade levels of
1-3 for primary and 4-5 for intermediate.
Measure of Achievement
The measure of achievement in this study was the
MSPAP performance assessment used statewide in Maryland. MSPAP is a state-mandated performance test of
achievement given each May to a census sample of third,
fifth, and eighth graders in reading, writing, language use,
mathematics, science, and social studies. The assessment
tasks are collections of interrelated activities or items that
are organized around a theme (e.g., powerful forces of
nature or westward expansion). Usually the tasks require
students to solve a problem or to explain their responses.
Activities comprising the tasks can be (a) group or individual activities; (b) hands-on observation or reading activities; or (c) extended written responses, limited written
responses, lists, charts, graphs, diagrams, webs, and drawings. Examinees are randomly assigned to independent
clusters that are administered in separate locations. All content areas have approximately 90 rain of assessment time
per cluster on each of 5 consecutive days. However, a substantial number of responses are coded for more than one
content area. For example, there were 24 items for reading
in Grade 3 and 30 items in Grade 5, but some of the student
responses also led to items on other content scales. The
number of items for each content area in Grades 3 and 5 is
reported in Table 2. The data were based on a technical
report (Maryland State Department of Education, 1997)
that showed the results of the 1996 administration used in
this investigation. Teachers who were not necessarily the
classroom teachers for the students administered the tasks
over a 5-day period within each school.
The performance assessment was scored by 36 trained
teams (Maryland State Department of Education, 1997).
During the scoring activity, any reader who scored below
70% accuracy received additional training immediately to
assure quality control. The interrater agreement tbr all items
on the test was satisfactory. For Grade 3, 90-100% agreement was obtained for 38% of the items; 80-89% agree-
215
Table 2.--Reliabilities of the Maryland State Performance
Assessment Program
Subject/grade
No. of items
Sample
Cronbach's alpha
Reading
3
5
24
30
7.499
7.500
.82
.77
Writing
3
5
14
2O
7.499
7,500
.68
.66
Mathematics
3
5
25
25
7,499
7,499
.88
.89
Science
3
5
17
26
7,499
7,500
,84
.87
Social studies
3
5
19
17
7,499
7,500
.87
.83
Language use
3
5
14
20
7,500
7,500
.93
.92
ment was obtained for 45% of the items; and 70-79% was
obtained for 15% of the items. For Grade 5, 90-100% was
obtained for 31% of the items; 80-89% agreement was
obtained for 52% of the items; and 70-79% was obtained
for 15% of the items. Those results slightly oversimplify a
complex interrater agreement scoring scheme (Maryland
State Department of Education, 1997).
Internal consistency reliability of the MSPAP in 1996 is
reported in Table 2. The Cronbach alpha for reading in
Grade 3 was .82 and .77 for Grade 5. It is evident that the
other content areas had reliabilities ranging from .93 to .66.
All are satisfactory except writing, which is lower but adequate. The reliabilities are computed for individual students, but the test scores are intended for school-level
reporting and not for individual interpretation.
Validity of the MSPAP is based partly on its reflection of
the Maryland learning outcomes. The mathematics outcomes were based on content standards from the National
Council of Teachers of Mathematics; the science outcomes
were tbunded on Project 2061 by the American Association
for the Advancement of Science. Social studies outcomes
were consistent with the work of the Association of American Geographers and Commission on History in the
Schools. Reading outcomes were similar to the stated goals
of the NAEP reading assessment objectives. The objectives
included reading for literary experience, reading to be
informed, and reading to perform a task.
The reading section of the MSPAP appears to possess
face validity. It is a pertbrmance-based assessment that uses
authentic texts. Reading selections are full-length, published works rather than excerpts contrived for use in a test; the
text is placed in realistic situations. Connections among science content such as plant growth, and reading skills such
216
as summarizing or identifying trends in texts are natural
integrations drawn from classroom activities.
Validity of the MSPAP has been examined by comparing
it to the Comprehensive Test of Basic Skills, No. 4
(CTBS/4) administered in 1991. Convergent validity estimates ranged from .73-.77 for reading, language arts, and
mathematics for third-, fifth-, and eighth-grade students
(Yen & Ferrara, 1997). Administrations of the same tests in
1995 showed correlations of .56-.66 on reading, language
use, and mathematics scale scores (Maryland State Department of Education, 1997). Teacher ratings of student
achievement and MSPAP were moderately correlated at
.41-.49 in Grade 3, .43-53 in Grade 5, .43-.61 in Grade 8
(Maryland State Department of Education, 1997), and are
consistent with data reported by Yen & Ferrara (1997).
Construct validity of the MSPAP is variable, according to
factor analyses. For example, although the reading scale
formed one factor in Grade 8, the reading scale formed two
factors in Grades 3 and 5, which was attributed to the range
of text types and variety of purposes in the elementary level
of the MSPAP (Maryland State Department of Education,
1998). The construct validity was also examined by determining the extent of differential item functioning. Items are
identified annually that function differently for ethnic
groups of African Americans, Asians, Hispanics, and Caucasians or for the performance of girls in comparison with
boys. Those items are flagged for attention in item construction in a succeeding year. There were only 4 items out
of 48 showing differential functioning in Grade 3 in 1997
and I item out of 54 for Grade 5 that showed differential
functioning.
Measures of lnstruction
A questionnaire on school reading instruction was constructed on the basis of research on classroom variables
expected for reading comprehension. Constructs were
intended to represent the following: (a) basal emphasis, (b)
books and resources, (c) comprehension instruction, (d)
integrated reading, writing and content, (e) social collaboration, and (f) writing. Each construct item was generated
according to the construct definitions in the literature. The
item-writing team included a professor in reading, a professor in measurement, a graduate student with experience
teaching reading, and a measurement graduate student with
teaching experience. In this study, we defined the six constructs in the following paragraphs (see the Appendix).
Basal emphasis: The teacher implements a basal program
comprehensively. Materials for language arts are drawn
from the basal, and the sequence of reading skills recommended in the basal is followed. Examples include: "I follow the teacher's guide for comprehension in the basal I
use" and "I rely heavily on the materials and resources contained in my basal workbook program." The scale had five
items with a Cronbach alpha reliability of .79.
The Journal of Educational Research
Books and resources: The teacher emphasizes literaturebased units that employ trade books and literature. Multiple
sources, including classroom books, library and media center, and community organizations, are used for instruction.
A range of professional colleagues participates in thematic
units. Examples are: "Our media specialist collaborates
with us in the development of literature-based units" and
"We visit community resources that will enrich our understanding of what we are studying." The scale had five items
with a Cronhach alpha reliability of .40.
Comprehension strategy instruction: The teacher provides direct, explicit instruction in reading strategies such
as using background knowledge, summarizing, self-monitoring, learning new vocabulary, and accurate word identification. Emphasis is given to gaining both automaticity and
flexibility in the use and application of skills in a wide
range of texts. Examples are: "I encourage my students to
recall their prior knowledge before they begin a new reading assignment" and "I teach my students how and when to
reread passages in a book to clarify their answers." The
scale had six items with a Cronbach alpha reliability of .50.
Integrated reading, writing, and content: The teacher
emphasizes reading and discussion of authentic texts,
including exposition and narrative. Note taking and writing
in response to text are included. Knowledge-rich topics in
social studies and science, as well as literature, are used as
the content for reading and writing instruction. Examples
include: "The students in my class read long, challenging
texts" and "I send students to do research alone or in small
groups in the media center without notice." The scale had
nine items with a Cronbach alpha reliability of .55.
Social collaboration: The teacher emphasizes cooperative learning and provides explicit guidance in taking turns
and participating in a group. Children are encouraged to
disagree constructively and to write collaboratively. Examples include: "We form literature groups to facilitate discussion" and "Students in my class review and edit each
other's work." The scale had five items with a Cronbach
alpha reliability of .65.
Writing: The teacher emphasizes the writing process
including planning, drafting, and revising. Understanding
and following the directions in a writing prompt are taught,
accompanied by organization in composition. Examples
include: "We spend a lot of time discussing and practicing
how to read a writing prompt and how to organize a
response" and "My students are learning to use a graphic
organizer to write a paragraph." The scale had five items
with a Cronbach alpha reliability of .71.
The response format for all scales was a Likert-type scale
consisting of the following: 1 (not at all true of my class),
2 (not veo' true of my class), 3 (somewhat true of my class),
and 4 (very. true of my class). This format does not imply
strong judgment about the quality of instruction. It is a relatively low-inference set of statements about "'what my class
is like." The preliminary study showed that teachers using a
March/April 2000 [Vol. 93 (No. 4)]
wide range of approaches could represent their classrooms
fully with the items in this format. The four choices compelled teachers to decide for or against a statement.
We conducted a factor analysis on data from 545 teachers using pairwise deletion of cases with missing values. We
performed a principal components analysis with varimax
rotation, which yielded seven factors. There were multiple
eigenvalues larger than one. The largest seven were relatively different from each other, and the lower eigenvalues
were relatively similar to each other, which justifies the
selection of seven factors. A total of 44% of the variance
was attributable to the first seven factors, and the next seven
eigenvalues accounted for less than 15% more variance.
To examine each construct, we subjected the items
derived from the literature review and the preliminary study
to seven separate principal components analyses, one for
each theoretical construct. For example, all items listed
under the comprehension strategy instruction construct in
the Appendix were entered in the principal components
analysis for comprehension strategy instruction. Loadings
of all items on each construct, for example, comprehension
strategy instruction, are shown in the Appendix. A large
majority of the Ioadings on each construct exceeded .40. A
total of 39 (87%) of the items attained this level, which indicated adequate cross-validation from the preliminary study.
To retain the theoretical relevance and content validity of
the constructs, we used all items in analyses of instructional effects on achievement.
Administrative Procedures
The first and second authors invited the districts to participate in January 1996. Teachers and administrators were
informed that the data would be aggregated at the school
level and that individual teachers and schools would be
anonymous in the study. The school districts also were
requested to give permission to use their school-level
MSPAP content area scores and demographic data. Individual student data were not collected nor used. Teachers
returned the machine-scorable questionnaires to the local
accountabililty coordinator who also supplied the achievement data. The data were collected by the end of March
1996. Schools were informed that a school-level report
would be provided with school means and standard deviations along with district means and standard deviations on
each construct. Those reports were sent to all the schools in
September 1996.
Preliminary Study
To develop the questionnaire and pilot test the metaanalysis procedures, we conducted a preliminary study. In
one district in January 1994, all 225 teachers in Grades 1-5
in 15 elementary schools completed a school reading
instruction questionnaire. There were 35 items measuring
six constructs generated from the theoretical literature. The
217
constructs were intended to represent the following: comprehension instruction, integrated reading, writing and content, books and resources, classroom collaboration, basal
emphasis, and writing. A factor analysis was conducted
with a principal components analysis and a varimax rotation. Six factors were identified: the highest six eigenvalues
accounted for 50% of the variance. The items were all
found to possess either substantial loadings on the expected
construct, exceeding .40, or high face validity with a more
moderate loading. Therefore. all 35 items were retained for
the full study.
We conducted a meta-analysis to examine whether the dimensions of instruction predicted change in CTBS achievement scores in 1991. 1992, and 1993. The data were schoollevel aggregations of reading achievement scores in Grades
3 and 5. Changes in achievement from 1991-1992 and
1992-1993 were significantly predicted by a substantial
number of school-level mean scores from the school reading instruction questionnaire. Although limited by the small
sample of 15 schools from one district, the preliminary
study suggested that a larger study with MSPAP data, which
is the high-stakes assessment in Maryland, would be useful.
Results
We first introduce descriptive results. Correlations
among instructional variables are reported in Table 3. The
lower left triangle shows associations among variables for
teachers of the intermediate grades (4 and 5). It is evident
that integration is significantly correlated with collaboration
(.44) and assessment (.34). However, a large proportion of
variables are not significantly correlated. For teachers of the
primary grades (I-3), correlations are shown in the upper
right triangle. Writing correlated significantly with books,
comprehension instruction, and integration. Furthermore,
use of books and resources correlated significantly with
comprehension instruction, integration, writing, collaboration, and assessment familiarity. In the primary grades, a
high proportion of the correlations was significant, showing
association among instructional variables not present in the
intermediate grades.
Means and standard deviations of the instructional variables are reported in Table 4. It is evident that intermediate
teachers, rather than primary teachers, placed more emphasis on integration, books and resources, collaboration, and
writing and possessed higher assessment familiarity. Primary teachers placed more emphasis than did intermediate
teachers on basal programs and comprehension instruction.
The goal of the investigation was to identify characteristics of improving schools. We wanted to identify which of
the instructional characteristics of the school reading program was associated significantly with change in achievement across time. For that purpose, we used a metaanalytic approach (Hedges & Olkin, 1985). Meta-analysis
is frequently used to study results from multiple studies to
determine general magnitude as well as correlates of effect
218
The Journal of Educational Research
Table 3.--Correlations Among Instructional Characteristics at Primary. and Intermediate Grades
Instructional process
Basal
Books
Comprehension
Integration
Writing
Basal
Books
Comprehension instruction
Integration
Writing
Social collaboration
1.00
.10
.22
-.03
-.03
-.26
.28
1.00
.05
.08
.06
-.02
.60"*
.40"*
1.00
.21
.53**
.29
.07
.40"*
.15
1.00
.25
.44**
.30
..1.4"*
.46**
.37**
1.00
.16
Collaboration
-.21
.37"
.20
.65**
.33*
1.00
Note. Intermediate grades are displayed in lower left; primary grades are shown in the upper right section of
the table.
* p < .05. **p < .0l.
sizes. For a study comparing means of two independent
groups, an effect size is computed consisting of the mean
difference between a treatment and a comparison group divided by the pooled standard deviation of the groups. An
effect size is normally distributed, and when divided by its
standard error, is asymptotically equivalent to a standard
normal (z) variate. A squared standard normal follows a
chi-square distribution with one degree of freedom and can
be summed with similarly derived chi-squares. The result
is a chi-square with degrees of freedom equal to the number of effect sizes involved. Deviations from the chi-square
distribution represent effect size heterogeneity that may
possibly be explained. Effect sizes may be modeled with
means or by regressions and the corresponding reduction in
the chi-square heterogeneity statistic is tested with degrees
of freedom equal to the number of parameters estimated in
the model. The result tests the hypothesis that the estimated parameters are zero. Original data may be treated with
meta-analytic procedures identical to those used for previously published data. In this meta-analysis, the analytic
unit analogous to the study was the school, and the control
and experimental treatments were 1995 and 1996 administrations of MSPAP that were given to independent groups
of students.
In our case, the effect size is the mean difference in
Table 4.wMeans and Standard Deviations oflntermediate
and Primary Teacherson Instructional Characteristics
in Reading
Teacher
Intermediate
(4-5)
Instructional process
Basal
Books
Comprehension instruction
Integration
Social collaboration
Writing
M
1.68
1.44
2.57
2.00
2.49
2.67
SD
n
.72 147
.58 137
.34 144
.34- 127
.36 150
.31 148
Primary
( 1-3 )
M
SD
n
1.77
1.37
2.64
1.78
2.18
2.47
.69
.55
.31
.47
.56
.48
222
213
214
198
222
221
achievement in 1995 and 1996 in a given school at a given
grade level, divided by the pooled standard deviation for the
groups. Because we have 33 schools, there are 33 effect
sizes. These effect sizes are distributed as a chi-square with
33 degrees of freedom, 1 for every estimate. Deviations of
the effect sizes around the grand mean shows heterogeneity
of variance that can be associated with variance in instructional variables. This association is similar to the logic of
regression. The meta-analyist asks whether variance in
effect sizes (e.g., differences among schools in their
changes in reading achievement) is associated with instruction (e.g., variance among schools in their emphasis on various instructional characteristics). In the main analysis, we
used a set of six instructional characteristics representing
the school reading program to predict achievement change
in reading and other content areas. Beta weights for the
instructional characteristics showed a profile of variables
effecting achievement change.
Similar to regression analysis, meta-analysis can also test
for the unique contribution of an instructional variable. The
unique contribution refers to the proportion of variance in
effect sizes attributable to a given instructional variable (e.g.,
comprehension instruction) when all other instructional variables are controlled statistically (in this case, basal emphasis, books and resources, integrated instruction, collaboration, writing, and assessment familiarity). In this study, we
examined the unique contributions of individual instructional characteristics to the variance in effect size for reading
achievement change across the 33 participating schools.
To conduct the meta-analysis, we computed an effect size
for the change in achievement from 1995 to 1996 in each
school, at each relevant grade level. That is, for a given elementary school Grade 5, mean achievement in reading in
1995 was subtracted from the mean achievement in reading
in 1996. That difference was divided by the pooled standard
deviation of the scores. The resulting effect size is positive
if the school has been improving and negative if the school
has been declining in achievement. We next attempted tq
predict the effect siz,+s with all. the characteristics of the
school reading program. We asked whether between-school
variation in effect size for reading in Grade 5 was associat-
,
March/April 2000 [Vol. 93 (No. 4)]
219
Table 5.'Descriptive Statistics for Meta-Analysis of Reading Achievement Change (Grade 5)
Variable
n
Minimum
Maximum
M
SD
School M (1995)
School SD (1995)
Number of test takers in each school (1995)
School M (1996)
School SD (1996)
Number of test takers in each school t1996)
Effect size
35
35
35
33
33
33
32
498
31
13
494.18
21.12
4.00
-.47
548
63
196
539.00
50.38
196.00
.82
516.00
43.07
77.69
522.70
38.20
79.55
.1585
11.75
6.75
41.77
12.53
5.77
44.65
.2403
Table @--Effect Sizes of Achievement Change in
Six Contents
Subject
Minimum
Maximum
M
SD
.82
.79
1.01
1.10
1.05
.78
.16
.16
.22
.25
.26
.24
.24
.18
.34
.27
.25
.22
.48
.78
.49
.90
.18
.60
-.02
-.02
-.06
-. 12
-.28
-.08
.30
.35
.37
.40
.32
.28
Grade 5
Reading
Language use
Mathematics
Science
Social studies
Writing
-.47
-.26
-.36
"
-.,7
-. 16
-.17
Reading
Language use
Mathematics
Science
Social studies
Writing
-.73
-1.29
-1.48
-.96
-1.08
-.46
Grade 3
Note. n = 32 for Grade 5; n = 33 for Grade 3.
ed with b e t w e e n - s c h o o l variation in all the instructional
characteristics as a set. The statistical significance o f that
association was estimated with a chi-square test.
To conduct the meta-analyses, we used several descriptive statistics that are c o n v e y e d in Table 5. For example, the
school mean in 1995 was 516.00 for reading. The standard
deviation of the 35 school means available for 1995 was
1 1.75. In 1995, in the 35 schools, there was a mean of 77.69
students taking the reading portion of the MSPAP. In other
words, 2,719 students took the 1995 reading test and their
scores were used for these analyses. In 1996, the mean reading a c h i e v e m e n t was 522.70. The m e a n effect size for the
1995-96 change w a s . 16. As indicated in the table, the lowest performing school had a negative effect size o f - . 4 7 and
the highest performing school had a positive effect size of
.82. This standard deviation o f those effect sizes was .24. It
is the variance o f those effect sizes that is predicted by reading instruction in our recta-analyses.
In addition to reading, effect sizes were c o m p u t e d for
a c h i e v e m e n t change in all six content d o m a i n s (see Table 6
for Grades 3 and 5). For example, in Grade 5 science the
mean effect size was .25, the m i n i m u m was -.27, and the
m a x i m u m was 1.10. In G r a d e 3, all means of the effect sizes
were negative, reflecting the fact that a c h i e v e m e n t in all of
the d o m a i n s for the schools decreased from 1995 to 1996.
The meta-analysis consisted of predicting the school
a c h i e v e m e n t changes with characteristics of school reading
instruction. The o u t c o m e s are reported in Table 7. The reading instruction at the intermediate grade level successfully
predicted a c h i e v e m e n t change in Grade 5. In this analysis,
a c o m p o s i t e of six reading characteristics was used to predict the effect size. The chi-square o f this analysis was significant, Z2(6, N = 32) = 17.880, p < .007. The multiple R
for this analysis was .634 with an R 2 o f .402. That result
indicates that all o f the instructional characteristics as a
c o m p o s i t e had a positive effect on a c h i e v e m e n t change in
reading. The effect was a c o n s e q u e n c e of a profile o f individual instructional characteristics that will be presented
later in this section.
The effect o f the school reading p r o g r a m on all achievement d o m a i n s was evaluated with ) 2 analyses. As Table 7
shows, the effect of reading instruction was statistically significant on a c h i e v e m e n t change in mathematics, Z2(6, N =
Table 7.--Effects of School Reading Program on
Achievement Change in Six Contents (Grade 5)
Subject
Reading
w/o covariate
with covariate
Language use
w/o covariate
with covariate
Mathematics
w/o covariate
with covariate
Science
w/o covariate
with covariate
Social studies
w/o covariate
with covariate
Writing
w/o covariate
with covariate
Z2
p
R
R-'
17.880
19.358
.007
.004
.634
.629
.402
.396
9.953
9.463
.127
.149
.475
.464
.226
.215
18.377
20.101
.005
.003
.416
.435
.173
.189
16.093
16.950
.013
.009
~577
.592
.333
.350
1.414
2.532
.965
.865
.180
.241
.032
.058
I4.337
12.147
.026
.059
.515
.47..1-
.265
.225
Note. Dependent variable = 1995-I996 effect size, w/o = without.
220
The Journal of Educational Research
Table 8.--Effects of Reading Program and Individual Instructional Characteristics on Achievement
Change (Grade 5)
Reading
Language
use
17.880
.634
.402
9.953
.475
.226
Social
studies
Math
Science
Writing
16.093
.577
.333
.013
1,414
.180
.032
ns
18.377
.416
.173
.005
ns
14.337
.515
.265
.026
VariabLe
Total
Z:
R
R:
p
.007
Basal
B
X2
p
-.340
3.218
.073
-.157
.686
.407
-.078
.394
.530
-.296
2.613
.106
-.031
.026
.872
-.237
1.874
.17l
Books/resou~es
B
Z:
p
.461
8.651
.003
.100
.396
.529
.133
1.702
.192
.337
5.001
.025
,144
.823
.364
.281
3.881
.049
Comprehension
8
Ze
p
-.222
1.891
.169
-.322
3.933
.047
-.054
.267
.605
-.341
4.842
.028
-.005
.001
.975
-.284
3.773
.052
.204
1.148
.284
.073
.143
.705
.412
11.004
.00l
.172
.874
.350
.009
.002
.964
.151
.753
.386
Social collaboration
B
Z2
p
-.069
.109
.741
.076
.134
.714
-.145
1.145
.285
-.003
.000
1.000
-.041
.037
.874
.137
.521
.470
Writing
B
Z~
p
-.110
.429
.512
.360
4.506
.034
.120
1.204
.273
.17l
1.111
.292
.119
.493
.483
-.012
.006
.938
Integration
fl
~2
p
32) = 18.377, p < .005. Reading instruction was also significant in predicting achievement change in science, Z2(6, N
= 32) = 16.093, p < .013. The multiple R for science was
.577, with an R 2 of .333. The reading instruction also influenced writing scores significantly, X2(6, N = 32) = 14.337,
p < .026. The multiple R was .515, with an R 2 of .265. Reading instruction did not significantly predict achievement
change in language use scores or social studies scores on
the MSPAP.
The analyses of effect sizes reported in the preceding
paragraph were also conducted with a covariate. That is,
effect size of achievement change between 1995-1996 was
the dependent variable. A covariate was the previous
achievement of the same students in the same content areas
assessed in 1994. As Table 7 shows, reading instruction significantly predicted the reading effect size when the covariate was used, Z2(6, N = 32) = 19.358, p < .004. With the
covariate included, reading instruction predicted mathematics achievement change significantly, Z2(6, N = 32) =
20.101, p < .003. Reading instruction also predicted
achievement change in science, X2(6, N = 32) = 16.950, p <
.009. Finally, reading instruction predicted achievement
change in writing at a marginal level, X2(6, N = 32) =
12.147, p < .059. With the covariate included, reading
instruction did not predict social studies achievement
change nor did it predict language use achievement change.
To examine the individual instructional characteristics
contributing to the impact of reading instruction on reading
achievement, we undertook additional analyses. As Table 8
indicates, beta ([3) weights were computed for each instructional characteristic when other instructional characteristics
were controlled. When achievement change in reading was
the outcome, positive beta weights, and, consequently, positive instructional influences were observed for the variables of books/resources ([3 = .461) and integration of
instruction ([3 = .204). At the same time, negative beta
weights were observed for basal emphasis in instruction
(13 = - . 3 4 0 ) and direction comprehension instruction ([3 =
-.222). In addition, smaller negative betas were observed
for writing (13 = -.110) and social interaction in reading
(~ = -.069).
The betas convey a profile of characteristics. Achievement change was positive in classrooms where teachers
emphasized a Wide variety o f books and resources as the
basis for reading instruction. In addition, positive effects
were observed for the integration of reading instruction
March/April 2000 [Voi. 93 (No. 4)]
221
Table 9.--Effects of Reading Program and Individual Instructional Characteristics on Achi%~ement
Change ~Grade 3)
Reading
Language
use
Math
Science
Social
studies
Writing
3,311
,232
.054
5.971
.283
,080
8.055
.312
.097
7.179
.270
.073
7.710
.328
.108
6.116
.282
.080
ns
ns
ns
ns
ns
.136
.484
.487
.104
.343
.558
.041
.061
.805
.109
.504
.478
.033
.039
.843
.145
.933
.334
.334
5.505
.019
.206
3.000
.083
.119
.602
.438
.155
2.689
.I01
-.202
1.123
.289
-.266
2.366
.124
-.223
1.862
.172
-.335
5.004
.025
-.437
6.159
.013
-.274
4.21
.040
-.255
1.249
.264
-,364
3.048
.081
.043
.04.7
.828
-.220
1.451
.234
-.296
1.927
.165
-.351
4,528
.033
.079
.123
.726
.120
.336
.562
-.143
.538
.463
.073
.167
.683
.342
2.647
.I04
.092
3.025
.082
.064
.136
.712
.133
,711
.399
-.006
.002
.964
.243
3.164
.075
.168
1.095
.295
.154
2.580
.108
Variable
Total
X~,
R
Ra
p
ns
Basal
8
~2
p
Books/~sources
Z2
p
Comprehension
~:
p
Integration
~2
p
Social collaboration
g
g2
p
Writing
B
X2
p
Table 10.--Effects of School Reading Program on
Achievement Change in Six Contents (Grade 3)
Subject
Reading
Language use
Mathematics
Science
Social studies
Writing
X2
p
R
R2
3.3 l 1
.544.1
8.055
7.179
7,710
6.1 I0
,769
.489
,234
.305
.260
.4t 1
.232
.298
.312
.270
.328
.282
.054
.089
.097
.073
.108
.080
Dependent variable = 1995-1996 effect size. d f = 6 for all
subjects.
Note.
with writing and learning in content domains such as science and social studies. At the same time, the classrooms in
which positive achievement change was observed tended to
have relatively less emphasis on basal instruction, relatively less emphasis on direct strategy instruction, and very
moderate emphasis on writing and social interaction. Conversely, schools in which achievement change was relatively low or negative were characterized by instruction that
.273
.2.297
.130
.102
.3,1a
.558
emphasized a strong reliance on the basal reading program
and heavy emphasis on comprehension instruction, but relatively little use of a broad array of diverse trade books and
resources; and relatively little use of integi'ated reading,
writing, and content teaching.
When achievement in the other content areas at Grade 5
was predicted as a function of the individual instructional
characteristics in reading, a similar profile emerged. For
example, in science, positive achievement change was highest in schools that emphasized books and resources (13 =
.337), integration of instruction (13 = .172), and writing
instruction (13 = .171). At the same time, low change or negative change was occurring in school programs that heavily
emphasized basal instruction (13 = - . 2 9 6 ) a n d direct comprehension instruction (13 = -.341). It is therefore interesting
that mathematics achievement change was highly predicted
by integration of reading instruction with other content
domains (13 = .412). In addition, language use was predicted by an emphasis on writing (13 = .360), which was statistically significant (p < .034), even though reading instruction as a whole did not influence language use. Across
content domains, books and resources, as well as curriculum integration, always showed positive instructional
222
effects. At the same time, emphasis on basal reading and
direct comprehension instruction showed negative effects.
whereas writing emphasis and social interaction showed
mixed effects--positive in some content areas and negative
in others.
In Grade 3, meta-analyses of the effects of reading
instruction on achievement change were also conducted. As
Table 9 indicates, the composite of reading instruction characteristics used in this study did not significantly predict
achievement change in reading. In addition, reading instruction did not predict achievement change in any of the other
content domains, including language use, mathematics, science, social studies, or writing. Further analyses of individual characteristics were performed in Grade 3. Those analyses should not be considered too seriously because the
overall effects of the composite of reading instruction were
not significantly predictive. However, books and resources
had a consistently positive beta weight and basal instruction
had a consistently positive beta weight across the six contents (see Table 10). At the same time, integration had a
consistently negative beta weight and direct comprehension
instruction had a consistently negative effect. Those influences were not statistically significant, however, and should
not be viewed as substantive.
Discussion
It is evident that reading achievement change on the
statewide performance assessment in Maryland was directly associated with the characteristics of the school reading
program in the higher elementary grades. The high-impact
programs had a distinctive profile. In particular, they contained integration of curriculum and an abundance of books
and resources. Furthermore, they placed a low emphasis on
the basal program, comprehension strategy instruction,
writing techniques, and social interaction. Few studies
empirically address the effects of school-level reading program characteristics on reading achievement.
Our findings confirm and extend previous research
regarding the benefits of diverse books and resources and
the integration of reading instruction on reading achievement. The MSPAP measure of achievement in all subjects,
including reading, required extensive use of cognitive strategies of searching for information, comprehending multiple
texts, combining knowledge from text with knowledge from
such sources as science experiments and geographical maps,
and expressing that knowledge coherently in writing. Those
cognitive strategy demands of the MSPAP are similar to
those of the performance assessment used by Guthrie and
colleagues in several investigations (Guthrie et al., 1996;
Guthrie et al., 1998; Guthrie et al., 1999). As shown in one
study, those strategies correlate highly (.7-.8) with intrinsic
motivations for reading. Because the reading tasks require
both strategies and motivation, performance on them entails
reading engagement, which is the joint operation of strategies and motivations in reading to learn. Such reading
The Journal of Educational Research
engagement is facilitated by integrated instruction, use of
abundant texts, and other characteristics such as autonomy
support and use of conceptual themes (Guthrie et al.. 1998;
Romance & Vitate, 1992). The present findings extend prior
results by showing generalizability of the integrated instruction and text resources from the individual classroom level
to the school level. Not only do those qualities typify classrooms, but they also typify schools that increase reading
achievement in the latter elementary grades.
The relatively negative effect of comprehension instruction on achievement change is not consistent with previous
classroom research. We expect that the negative effect in
this study was due to the fact that the comprehension
instruction was relatively isolated. The teachers' reports of
comprehension instruction did not include statements about
the substantive content context of the instruction. If the
instruction were isolated and not thoroughly explained as
widely transferable as recommended by many authors
(Dole, Duffy, Roehler, & Pearson, 1991; Duffy, 1993), it
may have been counterproductive.
Effects of instructional processes on reading achievement
in Grade 3 followed a very different pattern from Grade 5.
For reading achievement change in Grade 3, the reading
programs characterized with these instructional processes
did not significantly predict achievement change. That result
confirms our expectation on the basis of studies of current
instructional practice (Baumann et al., 1998) and of studies
focusing on outstanding teachers in the primary grades
(Pressley, Rankin, & Yokoi, 1996) and intermediate grades
(Pressley, Yokoi, Rankin, McDonald, & Minstretta, 1997).
We hypothesized that those characteristics as a set would
influence achievement change in the intermediate grades (5)
but not in the primary grades (3). This study contributes to
the existing literature by documenting that school-level
instructional characteristics of improving schools at the
intermediate level are very different from programs at the
primary level. It also illustrates the trends with a high-stakes
reading achievement test (Yen & Ferrara, 1997).
Characteristics of school reading programs were also associated with achievement change in several different content areas in Grade 5. School reading instruction increased
achievement in the content areas of science, mathematics,
and writing, as well as reading. Furthermore, the benefits
accrued even when prior achievement levels of students in
the schools had been statistically controlled. Benefits in
other content areas were also observed for books and
resources and curricular integration. In addition, basal
emphasis and direct comprehension instruction appeared
to be negatively associated with change for all subjectmatter areas.
There are at least three possible explanations for the finding that characteristics of reading instruction were associated with achievement change in nonreading areas. First, it is
possible that instruction that increases reading achievement
improves performance on the statewide performance
assessment (MSPAP) in across-subject matters because the
March/April 2000 [Voi. 93 (No. 4)]
performance assessment is reading intensive. All the
domains, including mathematics, reqt]ire extensive reading
comprehension. Consequently, students who rate higher in
reading comprehension are likely to excel on all tasks.
A second possible explanation is that instructional variables in reading such as an abundant use of books and resources provide expanded learning opportunities in the
other domains. When the reading program is enriched with
a range of trade books, authentic materials, and audio visual content, students may learn other domains as a result of
using those materials. A third possible explanation is that
there are associations between effective instruction in reading and effective instruction in another content domain such
as mathematics. That is, teachers who use effective strategies in reading, such as comprehension instruction, may
also use effective strategies in mathematics that enable students to gain mathematical concepts at a high level. These
possible interpretations should be explored further by examining instruction in reading concurrent with instruction and
achievement in one or more content areas of the curriculum.
In the present study, we raise questions about the types of
teacher knowledge and the use of that knowledge needed to
increase achievement. We also question the policy at the
state, district, and school levels regarding staff development. It is possible that instructional emphasis on integrated, coherent, effective curriculum can increase achievement
across domains. However, this possibility requires an experimental trial.
There are several limitations to this investigation that one
should note. The study was conducted in all schools, in
three districts where the majority of teachers were Caucasian, moderately experienced, and relatively highly educated. Urban schools did not volunteer to participate in the
study and were not included in the database. In addition, the
instructional analysis focused on intermediate grades, identifying effective dimensions of teaching. The analysis did
not focus on effective instruction in the primary grades, and
relatively few instructional processes were associated with
improving achievement at that level. However, in view of
the critical importance of reading achievement to school
success in the intermediate, middle school, and high school
levels, we believe that our findings warrant attention and
further investigation.
NOTE
The research reported herein was supported under the National Reading
Research Center project of the University of Maryland and the University
of Georgia, through a cooperative agreement (PR/Award No. 117A20007)
with the Office of Educational Research and Improvement, U.S. Department of Education. The findings and opinions expressed here do not necessarily reflect the position or policies of the National Reading Research
Center, the Office of Educational Research and Improvement, or the U.S.
Department of Education.
REFERENCES
Allen, S., & Reed. P. (1997). Talking about literature "'in-depth": Teachersupported group discussions in a fifth-grade classroom. In N. L
223
Karolides (Ed.), Reader response in elementary classroom: Quest and
discovery (pp. 53-68). Mahwah, NJ: Erlbaum.
"
Almasi. J. E (1995). The nature of fourth gradrrs' sociocognitive conflicts
in peer-led and teacher-led discussions of literature. Readh~g Research
Quarterts; 30. 314.-35 I.
Au, K. (1997). Ownership. literacy achievement, and students of diverse
cultural backgrounds. In J, T. Guthrie & A, Wigfleld (Eds.), Reading engagement: Motivating readers through integrated instruction (pp.
1-13). Newark. DE: International Reading Association.
Baumann. J. E (1984). The effectiveness of a direct instruction paradigm
for teaching main idea comprehension. Reading Research Quarterly 20.
93-115.
Baumann, J.. Hoffman. J., Moon. J. & Duffy-Hexter. A. M. (1998). Where
are teachers" voices in the phonics/whole language debate'? Results from
a survey of U.S. elementary teachers. Tile Reading Teacher. 51.
636-652.
Bean, T. W. (1995). Strategies for enhancing text comprehension in middle
school. Reading & Writing Quarterly, 11(2), 163-17l.
Berliner. D. C. (1986). In pursuit of the expert pedagogue. Educational
Researcher, 15. 5-13.
Bond. C. E.. Ross, S. M., Smith, L. J., & Nunnery., J. A. (1995). The effects
of the Sing. Spell. Read and Write program on reading achievement of
beginning readers. Reading Research & Instruction. 35(2). 122-141.
Brown, R.. Pressley, M., Van Meter, P., & Schuder, T. (1996). A quasiexperimental validation of transactional strategies instruction with lowachieving second-grade readers. Journal of Educational Psychologs; 88.
18-37.
Campbell, J. R., Voelkl, K. E., & Donahue. P. L. (1997). NAEP 1996 trends
in academic progress. Washington, DC: National Center for Education
Statistics.
Collins, C., (1993). Strategy instruction in a literature-based reading program. Elementary School Journal 94, 139-151.
Cunningham, A. E., & Stanovich, K. E. (1997). Early reading acquisition
and its relation to reading experience and ability 10 years later. Developmental Psychology, 33(6), 934-945.
Denner, P. E., McGinley, W. L, & Brown, E. (1989). Effects of story
impressions as a prereadingflwriting activity on story comprehension.
The Journal of Educational Research. 82, 320-326.
Doan, M. E, & Doan, R. L. (1998). An investigation of principals' and
teachers' perceptions of staff development needs in the elementary
school reading program. Journal of Instructional Psychology •5(2),
57-64.
Dole, J. A., Duffy, G. G., Roehler, L. R., & Pearson, P. D. (1991). Moving
from the old to the new: Research on reading comprehension instruction. Review of Educational Research. 61, 239-264.
Duffy, G. G. (1993). Teachers progress toward becoming expert strategy
teachers. Special issue: Strategies instruction. Elementary. School Journal 94. 109-120.
Feuer. M. J,, Holland, P. W., Green, B. E, Bertenthal, M. W., & Hemphill.
E C. (Eds.). (1998). Uncommon measures: Equivalence and linkage
among educational tests. Washington, DC: National Academy Press.
Fox. B. J. (1994). Storymates: A cross-age reading program. In E. G.
Sturtevant & W. M. Linek (Eds.), Pathways for literacy: Learners teach
and teachers learn: The sL~teenth yearbook of the College Reading
Association (pp. 259-267). Pittsburg, KS: College Reading Association.
Fullan, M. (1985). Change processes and strategies at the local level. Elementar3"School Journal 85(3), 391---421.
Gaskins, R. W., & Gaskins. 1. W. (19973. Creating readers who read for
meaning and love to read: The Benchmark School reading program. In
S. A. Stahl & D. A. Hayes (Eds.), Instructional models in readhtg (pp.
131-159). Mahwah, NJ: Ertbaum.
Goatley, V. J., Brock, C. H., & Raphael, T. E. (1995). Diverse learners participating in regular education "book clubs." Reading Research Quarterly, 30(3), 352-380.
Guthrie, J. T., Alao, S., & Rinehart. J. M. (1997). Engagement in reading
for young adolescents. Journal of Adolescent & Adult Literacy 40.
438--446.
Guthrie, J. T., Anderson. E., Alao, S., & Rinehart. J. (1999). Influences of
concept-oriented reading instruction on strategy use and conceptual
learning from text. Elementars.' School Journal, 99(~ ), 343-366.
Guthrie. J. T., Schafer. W. D., Wang, Y. Y., & Affierbach, E (1995). Relationships of instruction of reading: An exploration of social cognitive,
and instructional connections. Reading Research Quarterts, 30(IL
8-25.
Q
224
The Journal of Educational Research
Guthrie, J. T., Van Meter, P, Hancock, G. R., McCann, A., Anderson, E.,
& Alao, S. (1998). Does concept-oriented reading instruction increase
strategy-use and conceptual learning from text? Journal of Educational
Psychology 90(2), 261-278.
Guthrie, J. T.. Van Meter, P., McCann, A. D., Wigfield, A., Bennett, L..
Poundstone, C. C., Rice, M. E., Faibisch, E M., Hunt, B., & Mitchell,
A. M. (1996). Growth of literacy engagement: Changes in motivations
and strategies during concept-oriented reading instruction. Reading
Research Quarterly, 31, 306-332.
Hedges, L. V., & Olkin, 1. (1985). Statistical methods for meta-analysis.
Orlando, FL: Academic Press.
Kortez, C., Linn, R. L., Dunbar, S., & Shepard, L. ( 1991. April). The effects
of high-stakes testing on achievement: Preliminar3, findings about generalization across tests. Paper presented at the American Educational
Research Association, Chicago, IL.
Lapp, D., Flood, J., & Ranckbuhr, W. (1995). Using multiple text formats
to explore scientific phenomena in middle school classrooms. Reading
& Writing Quarterly, ••(2). 173-186.
Lundberg, I., & Linnakyta, P. (1993). Teaching reading around the world.
lEA: Grindeldruck, Hamburg, Germany.
Mandeville, G. K., & Anderson, L. W. (1987). The stability of school effectiveness indices across grade levels and subject areas. Journal of Educational Measurement, 24(3), 203-216.
Maryland State Department of Education. (1997). 1996 Maryland School
Performance Assessment Program--Technical Report. Annapolis: Author.
Maryland State Department of Education. (1998). Technical Report: 1997
Ma~. land School Performance Assessment Program. Baltimore.
Meloth, M. S., & Deering, P. D. (1994). Task talk and task awareness under
different cooperative learning conditions. American Educational
Research Journal 31(1), 138-165.
Miller, S. D., & Blumenthal, P. C. (1993). Characteristics of tasks used for
skill instruction in two basal reader series. Elementar3, School Journal,
94, 33--47.
Morrow, L. M., Pressley, M., & Smith, J. K. (1995). The effect of a literature-based program integrated into literac.v and science instruction on
achievement, use, and attitude toward literacy and science (Research
Report No. 37). Athens, GA: Universities of Georgia & Maryland,
National Reading Research Center.
O'Sullivan, C. Y., Reese, C. M., & Mazzeo, J. (1997). NAEP 1996 science
report card for the nation and the states. Washington, DC: National
Center for Education Statistics.
Payne, B. D., & Manning B. H. (1992). Basal reader instruction: Effects of
comprehension monitoring training on reading comprehension, strategy
use and attitude. Reading Research and Instruction, 32, 29-38.
Pressley, M. P. (1997). The cognitive science of reading. ContemporaD,
Educational Psychology 22(4), 247-260.
Pressley, M. P.. Rankin, J., & Yokoi, L. (1996). A survey of instructional
practices of primary teachers nominated as effective in promoting literacy. Elementary School Journal, 96, 363-384.
Pressley, M. P., Schuder, T., Bergman, J. L., & El-Dinar)', P. B. (1992).
Teachers in the students achieving independent learning program: A
researcher-educator collaborative interview study of transactional comprehension strategies instruction. Journal of Educational Psycholog3;
84, 231-246.
Pressley, M. E, Yokoi, L., Rankin, J., McDonald, R., & Minstretta, I.
(1997). A survey of the instructional practices of grade 5 teachers nominated as effective in promoting literacy. Scientific Studies of Reading, 1,
145-160.
Purkey, S. C., & Smith, M. (1983). Effective schools: A review. Elementary School Journal, 83, 428--452.
Reutzel. D. R., Sudweeks, R., & Hollingsworth, P. M. (1994). Issues in
reading instruction: The views and information sources of state-level
textbook adoption committee members. Reading Research & Instruction, 34(2), 149-171.
Reynolds, A. (1995). The knowledge base for beginning teachers: Education professionals' expectations versus research findings on learning to
teach. Elementary.' School Journal, 95(3), 199-221.
Romance, N. R., & Vitale, M. R. (1992). A curriculum strategy that
expands time for in-depth elementary science instruction by using science-based reading strategies: Effects of a year-long study in grade four.
Journal of Research in Science Teaching, 29, 545-554.
Santa, C. (1997). School change and literacy engagement. In J. T. Guthrie
& A. Wigfield (Eds.), Reading engagement: Motivating readers through
integrated instruction (pp. 218-235) Newark, DE: International Reading
Association.
Shanahan, T., & Lomax, R. G. (1986). An analysis and comparison of theoretical models of the reading-writing relationship. Journal of Educational Psycholog); 78(2), 116--123.
Slavin, R. E., Karweit, N. L., & Wasik, B. A. (1994). Preventing early
school failure: Research, polic3; and practice. Needham Heights, MA:
Allyn & Bacon.
Stevens, R. J., Madden, N. A., Slavin, R. E., & Famish, A. M. (1987, Fall).
Cooperative integrated reading and composition: Two field experiments.
Reading Research Quarterl3; 22, 433--.454.
Tanner, C., & Decotis, J. D. (1995). The effects of continuous-progress
nongraded primary school programs on student performance and attitudes toward learning. Journal of Research & Development in Education, 28(3), 135-t44.
Tucker, C. M., Chennault, S. A., Brady, B. A., Fraser, K. P. (1995). A parent, community, public schools, and university involved partnership
education program to examine and boost academic achievement and
adaptive functioning skills of African-American students. Journal of
Research & Development in Education, 28(3), 174-185.
Yen, W., & Ferrara, S. (1997). The Maryland School Performance Assessment Performance Program: Performance assessment with psychometric quality suitable for high stakes usage. Journal of Educational Psychological Measurement, 57(1 ), 60-84.
Walberg, H. J. (1986). Syntheses of research on teaching. In M. Wittrock
(Ed.), Handbook of research on teaching (3rd ed., pp. 214-230). New
York: Macmillan.
Wienert, E E., & Helmke, A. (1995). lnterclassroom differences in instructional quality and interindividual differences in cognitive development.
Educational Psychologist, 30, 15-20
Wilder, G. (1977). Five exemplary reading programs. In J. T. Guthrie (Ed.),
Cognition, curriculum, and comprehension (pp. 257-278). Newark, DE:
International Reading Association.
APPENDIX
Results of Six Principal Components Analyses for Revised Questionnaire
PC
Loading
.79992
.90308
.86176
.84717
.06669
.26064
.58365
.63436
BASAL PROGRAM EMPHASIS (o~ = .7925)
I use basals in my classroom.
I follow the teacher's guide for comprehension instruction in the basal I use.
I follow the sequence for teaching reading skills recommended in the basal program.
I rely heavily on the materials and resources contained in my basal workbook program.
During reading class students spend a significant amount of time reading and working individually.
COMPREHENSION STRATEGY INSTRUCTION (ct = .5028)
I help my students learn comprehension by making sure materials are matched to their reading
level.
I teach word identification through lots of practice with feedback.
My reading instruction progresses from simple lower level skills to complex higher order skills.
(Appendix continues)
,
March/April 2000 [3/ol. 93 (No. 4)]
Appendix continued
.63149
.47014
.58667
.68651
.57058
,66347
.71104
.78874
I encourage my students to recall their prior knowledge before they begin a new reading assignment.
I teach my students how and when to reread passages in a book to clarify their answers.
Before reading a new book, I introduce new or difficult words that I know will confuse or discourage my students if they aren't prepared for them.
WRITING (c~ = .7175)
My students are learning to use a graphic organizer to write a paragraph.
I teach my students to monitor their comprehension by asking themselves questions while they are
reading.
I teach my students how to summarize to improve their comprehension.
I teach my students to respond to writing prompts in literature and other content areas.
We spend a lot of time discussing and practicing how to read a writing prompt and how to organize a
response.
.48506
.68889
.24735
.54653
.66406
INTEGRATED INSTRUCTION (~ = .552 i)
We use the Writer's Workshop. My students read a book, select a topic they want to write about, and
publish their manuscript.
I pay just as much attention to English composition and grammar during social studies or science as
I do during language arts.
I teach my students how to take notes.
My reading and writing programs each have their own scope and sequence, independent of other
subjects.
We read expository texts in language arts at least once every 2 weeks.
The students in my class read long, challenging texts.
I teach language arts while teaching social studies and science.
1 send students to do research alone or in small groups in the media center without notice.
My students are learning about the writing process from planning to rough drafts to the final version.
.61801
.79272
.40126
.60092
.77208
COLLABORATION (~ = .6482)
I use cooperative learning often in my classroom.
Students in my class review and edit each other's work.
I teach my students how to take turns and solve problems etluitably when they work as a group.
We form small literature groups to facilitate discussion.
My students often read each other's compositions or reports.
.51480
.55430
.54150
.60263
.50459
BOOKS AND RESOURCES (c~ = .4010)
My students organize the classroom library into categories that make sense to them.
We have teacher assistance teams and I can talk and share with them whatever is on my mind.
Our media specialist collaborates with us on the development of literature-based units.
We select books for the classroom that are recommended by another school or school administrator.
We visit community resources that will enrich our understanding of what we are studying.
.53567
.22251
.74381
,06231
Note. Each group of items was analyzed separately with one factor extracted.
225