Protocol:
Graphic Organizer Use with Elementary
through Post-Secondary Students for
Reading Comprehension of Expository Text
Janet L. Proly, Jessica Rivers, Jamie B. Schwartz, Evgueni Borokhovski,
& Chad Nye
Date Submitted: July 4, 2010
BACKGROUND
A. OVERVIEW AND RESEARCH QUESTIONS
This review will investigate the use of graphic organizers and their role in
facilitating reading comprehension of expository text in students in kindergarten
through post-secondary school. The investigation includes randomized control
studies where graphic organizer use can be identified as the sole intervention and
the control condition uses a treatment as usual approach. The dependent variable
is the comprehension of expository reading material that is measured through
assessments (i.e., criterion referenced, standardized, norm referenced) following
the use of the text material. Specifically the research questions under
investigation are:
1)
What are the overall effects of using a graphic organizer intervention to
improve reading comprehension of expository text in kindergarten through
post-secondary levels of education?
2) Are there differential effects that relate to:
a) Graphic organizer attributes (i.e., use of linking language, computer based
vs. paper based)
b) Use of graphic organizer (i.e., before, during, or after presentation of
material)
c) Graphic organizer constructor (i.e., student, teacher)
d) Student classification (i.e., typical learner vs. learning disabled)
e) Educational level of participant (e.g. elementary, middle, high school,
post-secondary)
f) Intervention/instructional characteristics (i.e., frequency, intensity,
duration, training)
g) Text attributes (i.e., subject area, passage length)?
3)
What are the main knowledge gaps and research deficits in existing studies?
B. PROBLEM AND JUSTIFICATION FOR REVIEW
As students progress from elementary school to secondary and post-secondary
school, the demands of text-based reading increase. This process primarily
reflects a transition from narrative text-based learning to an emphasis on
learning from expository texts. Narrative text is heavily relied upon to teach
reading skills beginning in grades K-3 by providing a “story” context (Duke,
2000). Narrative text involves elements such as theme, setting, characters, plot,
conflict, and resolution and can be found in story books, picture books and
novels. Narrative texts are easier to relate to than expository text and are more
personal because they tell a story.
During or after Grade 4 is typically when students are introduced to expository
texts which present facts, opinions, processes, relationships and problemsolutions. Expository text, informational in nature and often including novel
technical vocabulary (Kim, Vaughn, Wanzek, & Shangjin Wei, 2004), is not
narrative or persuasive in nature, and is generally found in content area teaching
of subjects such as science, social studies, history, and geography. Students’
introduction to expository text also coincides with a decrease in reading
comprehension (Leach, Scarborough, & Rescorla, 2003). The emphasis of
narrative text in grades K-3 results in less exposure and practice with expository
text and may be one of the contributing factors to the “fourth-grade slump”
(Chall, Jacobs, & Baldwin, 1990). Additionally, from grade 4 through postsecondary school, there is an increasing emphasis on learning new information
through expository text, as well as an increasing accountability for students to
learn via independent reading, which can be problematic for many students.
Students who exhibit difficulties with constructing meaning from content-rich
expository text are at much greater risk for poor grades, which is one of the
strongest predictors of high school dropout rates (Alexander, Entwisle, & Horsey,
1997; Ekstrom, Goertz, Pollack, & Rock, 1986).
Unfortunately, educational research has not focused on reading comprehension
of expository text, even though reportedly only about one-third of U.S. third and
eighth graders are at proficient reading achievement levels (National Assessment
of Educational Progress (NAEP), 2007). Moreover, trends over the past twenty
years actually indicate a significant decrease in high school reading achievement
scores (Perie, Moran, & Lutkas, 2005). In order to positively influence the
statistics on student reading achievement, more attention towards educational
research examining the role teachers can play in facilitating reading
comprehension of expository text is warranted.
It is important to investigate strategies that are intended to help increase
comprehension of expository text. Expository texts are dense with information
and there are many different types of text structures for students to decipher. For
example, there are several formats of expository text found in science and social
studies books including: (1) procedural description, (2) enumeration, (3)
problem-solution, (4) sequence, (5) generalization, (6) compare-contrast, and (7)
2
The Campbell Collaboration | www.campbellcollaboration.org
classification (Weaver & Kintsch, 1991). Each expository text format serves a
specific purpose of presenting, structuring, relating, and/or organizing pieces of
information to enable its understanding and leading to meaningful learning.
One method to help facilitate the comprehension of expository text is the use of
graphic organizers. The strategy is designed to break down the text visually, thus,
increasing reading comprehension. Ausubel (1960; 1963) was the first to
hypothesize that advance organizers could be used to help learners expand their
knowledge of cognitive structure and, as a result, understand text better. Graphic
organizers are visual portrayals or illustrations that depict relationships among
the key concepts taken from the learning task (Hudson, Lignugaris-Kraft, &
Miller, 1993; Moore & Readence, 1984). Graphic organizers have been referred to
by different names over the years and examples include structured overviews,
flowcharts, timelines, tables, concept maps, semantic or syntactic feature
analysis, Venn diagrams, story maps, visual displays, framed outlines, cognitive
maps, mnemonic illustrations, and node-link maps.
Graphic organizers have been used and promoted as a strategy for teaching
expository text for several decades. A systematic review by Nesbit and Adesope
(2006) highlighted the publication trends of research on concept and knowledge
maps that showed an increase in publications in the mid-1980s. This increase in
graphic organizer publications produced research that explored different aspects
and types of graphic organizers (Guastello, Beasley, & Sinatra, 2000; Lipson,
1995), the populations that benefit from the use of graphic organizers
(McCrudden, Schraw, Lehman, & Poliquin, 2007; Troyer, 1994), the effectiveness
of teacher-developed versus student-created graphic organizers (Dunston &
Ridgeway, 1990; Foley, 1987), the use of graphic organizers in comparison with
other types of reading strategies (Darch & Gersten, 1986; Griffin, Simmons, &
Kameenui, 1991), and the use of graphic organizers with different types of
expository text (Alvermann, Boothby, & Wolfe, 1984; Rewey, Dansereau, & Peel,
1991).
Overall, the existing research mentioned above has shown that the use of graphic
organizers produces varying effects on student reading comprehension using
expository text. This review seeks to synthesize all of the available randomized
control studies, contained in both published and grey literature, to more closely
examine the influence of graphic organizer characteristics, text type and
difficulty, learner characteristics, and teacher involvement that positively
contribute to the effectiveness of the graphic organizer strategy. Furthermore, our
goal is to present an interpretation of the ways educators can most effectively
implement the successful elements and components contained within the
included studies. In this way, we hope to facilitate the transformation of
educational research to educational practice in an effort to improve students’
reading comprehension.
Previous Reviews on Graphic Organizers
Several systematic reviews and meta-analyses have been conducted investigating
the effectiveness of using graphic organizers to promote students’ comprehension
3
The Campbell Collaboration | www.campbellcollaboration.org
of expository text. These reviews have reported varying degrees of effectiveness
(Table A1) (Horton, McConney, Gallo, & Woods, 1993; Kim, Vaugh, Wanzek, &
Shangjin Wei, 2004; Moore & Readence, 1980; 1984; Nesbit & Adesope, 2006).
Moore and Readence (1980; 1984) conducted two meta-analyses evaluating the
effectiveness of graphic organizers for the use of learning from text. Moore and
Readence (1980) was unique in that it considered graphic organizers to be
synonymous with structured overviews. Moore & Readence (1980) defined
structured overviews as a graphic display of key passage vocabulary and their
relationship to new and known concepts. An overall mean effect size of 0.30 was
attained in the 1980 review and a 0.22 mean effect size was found in the 1984
meta-analysis. Moore & Readence (1980) concluded that graphic organizers were
most effective when used after the material was presented (post vs. advance
graphic organizers) and when vocabulary, rather than comprehension, was the
criterion variable. Moore and Readence (1984) also cautiously attributed greater
effect sizes to student involvement in the development of the graphic organizers.
Overall, these conclusions, based on these small effect sizes, must be interpreted
with caution because the overall mean effect sizes were not statistically
significant.
Horton, McConney, Gallo, and Woods (1993) conducted a meta-analysis of 19
studies that examined the effectiveness of concept mapping to improve student
achievement and attitude. Horton’s definition of a concept map originated from a
twelve year study conducted by Novak and Musonda (1991). Horton et al. (1993)
defined the concept map as portraying “hierarchy and relationships among
concepts.” This meta-analysis differed from the others in that it only included
studies in which the actual study took place in the classroom. Horton et al.’s
primary research questions focused on whether student or teacher prepared maps
were more effective and if gender influenced the map effectiveness. Horton et al.
found mean effect sizes ranging from .08 for the condition of teacher preparation
of key terms in the map to .88 for student overall map preparation in groups. The
mean of the student generated graphic organizer conditions was lower than the
mean of the teacher generated graphic organizer conditions with mean effect
sizes of .42 and .59, respectively, although only 3 studies were analyzed for the
teacher generated maps versus 14 studies for student generated maps. Unlike
Moore & Readence, Horton et al. did not consider the condition of map preparer
(teacher vs. student) to be a crucial factor in the effectiveness of concept mapping.
The limitations of this review included the poorly defined inclusion criteria (e.g.,
participant characteristics) and the limited search strategy (e.g., databases and
key words).
Griffin and Tulbert (1995) conducted a narrative review (2 meta-analyses, 2
research reviews, and 16 individual studies), assessing the impact of graphic
organizers on student comprehension and recall of expository text. Differences in
experimental design (e.g., simple vs. complex graphic organizer), instructional
process (e.g., student vs. teacher constructed graphic organizer), and study design
(e.g., absence of control group) make it difficult for broad statements to be made
concerning the effectiveness of graphic organizers. Griffin and Tulbert stated that
despite these research limitations, specific recommendations for classroom
4
The Campbell Collaboration | www.campbellcollaboration.org
practice could be made about graphic organizers. For example, post graphic
organizers are most effective when used in small-group activities and when
explicit teaching methods are used to implement the use of the graphic organizer
(e.g., modeling and guided practice).
Kim et al. (2004) was one of the few meta-analyses that examined the impact of
graphic organizers on reading comprehension with learning disabled students.
This meta-analysis summarized findings from 15 articles that produced 21
intervention studies from the years 1963-2001. Kim et al. defined graphic
organizers as visual displays that facilitated learning through arrow, lines or
spatial arrangement to depict structure and relationship. Kim et al.’s definition of
graphic organizers included semantic or syntactic feature analysis, Venn
diagrams, semantic maps, story maps, framed outlines and cognitive maps.
Overall, results of the Kim et al. meta-analysis revealed beneficial outcomes
across the 21 intervention studies, with mean effect sizes ranging from 0.40-1.79.
The meta-analysis showed positive effects when both the researcher and the
teacher implemented the graphic organizer intervention, with mean effect sizes of
0.96 and 1.05, respectively. The analysis by grade level (high school vs. middle
school) proved to not influence the effectiveness of the graphic organizer (1.18
and 1.08, respectively). There was no statistically significant difference found for
the graphic organizers on transfer tasks. The results of the meta-analysis revealed
information about variables that influence the effective use of graphic organizers,
but also highlighted problems in the research literature. For example, many of the
intervention outcomes were measured by researcher-developed tools as opposed
to standardized test measures. Only two of the studies used standardized
measures and neither of these studies revealed significant effect sizes. The
limitations of this review included its exclusion of post-secondary students, their
irregular search strategy (e.g. only searching 2 electronic databases for the years
1996-2001, and the fact that approximately 50% of the included studies were
conducted by the same author, with no accounting for the potential biasing of
results).
In 2006, Nesbit and Adesope conducted a meta-analysis that reviewed
experimental and quasi-experimental studies using concept maps and knowledge
maps to learn curriculum content for students in grade 4 through post-secondary
school. Concept and knowledge maps differ from graphic organizers because of
the use of labeled nodes and labeled or unlabeled links. Knowledge maps also use
a symbol system to denote concept relationships. Data for the analysis were
drawn from 55 studies retrieved through May 2005 (with no start date specified).
This review included problem-solving, cognitive, learning skills, or attitudes as
outcomes, making this one of the few meta-analyses that did not limit to a
reading comprehension outcome. This review analyzed several different
conditions including educational level, setting, subject, study duration, map
construction and map type. Results of the meta-analysis revealed greater effects
for 4th-8th graders (mean effect size 0.91) and postsecondary students (0.77) than
seen in the secondary students (0.17). A larger positive effect was also seen across
variables for students who constructed their concept maps versus those who
studied the concept maps. This review could be expanded by including all types of
graphic organizers and not limiting it to only concept or knowledge maps.
5
The Campbell Collaboration | www.campbellcollaboration.org
Table A1
Table of Past Meta-Analyses
A new systematic review is warranted for several reasons. First, new studies have
been published since Nesbit and Adesope et al.’s (2006) search ending in May
2005, which is the most recent graphic organizer review examining expository
text. More than three years of new information on a topic that has had
inconclusive results merits another look at the topic. Furthermore, more than
seven years have passed since Kim’s (2004) study with students with LD, whose
search ended in 2001.
No less importantly, the terminology used to describe the graphic organizers for
inclusion in some of the meta-analyses has been restrictive. For example, Nesbit
and Adesope et al.’s (2006) analyses included only studies that fit the narrow
definition of concept map, knowledge map or node-length map using these words
and word forms as their only keyword search terms. Furthermore, Horton’s 1993
study also placed an emphasis on concept maps by restricting key word search to
concept mapping, semantic mapping, and concept maps while in some
combinations education as a keyword was also added. Moore and Readence’s
(1980; 1984) studies used the term structured overview to define their study
focus, describing the term as a graphic display of key passage vocabulary and
their relationship to new and known concepts. Unfortunately, both of Moore and
Readence’s studies neglected to mention the search terms used. Kim’s (2004)
study did include a broad range of search terms for graphic organizer description,
but was limited to including only students with LD. This review will broaden the
scope of graphic organizer interventions by including all types of graphic
organizers.
Next, this review will expand the population examined in Nesbit and Adesope et
al.’s (2006) to include kindergarten through post-secondary school. Although
expository text is not as commonly seen in the early elementary grades as it is in
the later elementary grades and beyond, it is plausible that studies exist in the
6
The Campbell Collaboration | www.campbellcollaboration.org
younger grades using informational text material such as short passages or
informational basal readers. As a result, including all grade levels as a moderator
variable may shed light on differential effectiveness of graphic organizers for
varying learning levels.
In addition to the moderator variables addressed in previous meta-analyses (e.g.
grade, graphic organizer constructor, and content area), we will address
additional moderator variables. Areas that will be explored include, but are not
limited to: (a) length, duration and frequency of intervention, (b) fidelity of
implementation, (c) extent of intervener and student training on the intervention
method, (d) explicitness of graphic organizer construction process, (e) graphic
organizer attributes, and (f) technology use. The purpose of these analyses is to
determine under which conditions the intervention is more effective.
A comprehensive analysis of the effects of graphic organizers throughout the ages
will allow teachers of all students to better determine how graphic organizers can
be used to enhance reading comprehension of expository text. It is with this focus
on expository text and reading comprehension that we will further narrow the
topic. Lastly, we will examine the effects of graphic organizer use with both
regular education students and learning disabled (LD) students.
OBJECTIVES
The purpose of this review is to assess the effectiveness of graphic organizers in
improving reading comprehension of expository text for students in kindergarten
through post-secondary levels of education.
METHODOLOGY
A. STUDY INCLUSION CRITERIA
The review will employ two levels of the inclusion/exclusion procedure –
screening at the citation/abstract level and a thorough review of full-text
documents.
Inclusion Criteria at the Citation/Abstract Level
Studies at the citation/abstract level to advance to full-text review need to meet
the following standards:
o Address kindergarten (age 5) through post-secondary students
o Specify graphic organizer as the intervention method
o Specify expository text as the learning material
o Report measures of reading comprehension as study outcomes
o Utilize randomized control trials research designs.
Whenever the information reported in the abstract is not sufficient for making
confident decision, the study will be advanced to the full-text review level, at
7
The Campbell Collaboration | www.campbellcollaboration.org
which, in addition to the set of inclusion criteria above, attention will be paid to
statistical data necessary for effect size extraction and other essential information
typically absent in the abstracts.
Inclusion Criteria at the Full-Text Level
More specifically, studies will be included in this systematic review if they meet
the following inclusion criterion:
1. Study Participants
Studies will be included if the participants are in regular or special
education settings under the following conditions:
o Students grades K (age 5) -12 and college/university or
international equivalent
o Typically achieving or gifted students reading at or above grade
level
o Students defined as learning disabled, reading disabled, struggling
readers, reading difficulties including dyslexia, students at-risk for
reading difficulties in inclusive or self-contained classroom settings
o Students in school, clinical or experimental settings (i.e., university
clinic or one-on-one clinical intervention settings)
o Intervention is conducted in the native language of the learner (e.g.,
French students receive the intervention in French)
Studies will be excluded if the addressed population is:
o Non-students
o Second language learners in any language (i.e., intervention and
outcome measurement must be administered in primary language)
o Students who are (age 4 or below) described as in preschool settings
o Students in self-contained classrooms such as moderate to severe
disability groups (i.e., Trainably Mentally Disabled, Educably
Mentally Disabled, and Severely to Profoundly Mentally Disabled).
 The rationale for the exclusion of this subgroup is the
cognitive deficit present in these disability groups that
does not exist with students with learning disabilities
2. Nature of Intervention
Studies will be included if the intervention characteristics include:
o Use of a graphic organizer that coincides with reading materials
composed of expository text (i.e., non-narrative text depicting
concepts, relationships, processes, cause-effect, hierarchy, or
categorizing elements of the reading including but not limited to
main idea and supporting details).
8
The Campbell Collaboration | www.campbellcollaboration.org
3. Outcomes
Studies will be included when the outcome(s) measured reflect changes in
reading performance in immediate or delayed post test conditions such as:
o Expository text reading comprehension or
o Overall reading achievements
o Measurement of outcomes will be coded for standardized (i.e.,
Woodcock-Johnson III Tests of Achievement; Schrank, et al., 2001;
comprehension on Nelson-Denny standardized test of reading
skills; Brown, Fishco, & Hanna, 1993), criterion referenced (teacher
made, end of chapter test from text), or rating scale instruments.
Studies will be excluded that assess only:
o Problem-solving ability (i.e., such as in mathematics)
o Any other type of academic achievement
o Non-expository text
4. Study Design Features
Studies will be included in the final review as follows:
o Only RCTs will be included to provide a more comprehensive and
rigorous review of the causal impact of graphic organizers on the
measured outcomes. A preliminary search for appropriate studies
indicates that there is a sufficient number of high quality trials not
included in previous reviews.
Studies will be excluded if their design is quasi-experimental, preexperimental (single group pre- post test) or single subject.
Study characteristics with respect to other aspects of validity (e.g., Downs
& Black, 1998; Valentine & Cooper, 2008) will not be used as exclusion
criteria, but rather coded as methodological study features and addressed
in subsequent moderator variable analyses.
Publication Status: Published or unpublished.
Language of Instruction: Any language as long as the intervention is
delivered using the participants’ native language.
B. INFORMATION RETRIEVAL
1. Electronic Search Strategy:
For this systematic review, we will perform a broad and comprehensive electronic
search of the following data-bases from 1960 to present. This starting date is
chosen as Ausubel (1960) was the work that defined the first form of what, with
9
The Campbell Collaboration | www.campbellcollaboration.org
time, has become known as graphic organizers to help students learn. Electronic
databases to search in will include:









ERIC
PsycINFO
Dissertations and Theses: Full Text
Web of Science
British Education Index
Canadian Education Index
FRANCIS
Australian Education Index
Google Scholar (first 50 results)
The search terms that will be used in a variety of combinations with the
truncation symbol appropriate to the United States’ databases (ERIC, PsycINFO,
Web of Science and Dissertations & Theses & Google Scholar) are as follows:


graphic organizer* or concept map* or semantic map* or cognitive
map* or node-link map* or visual aid*
 AND
reading comprehension or reading achievement*
For the following international databases (British, Canadian & Australian
Education Index and FRANCIS, the following search terms, with appropriate
truncation, will be used in congruence with international indexing:

Graphic organizer* or concept map* or semantic map* or cognitive
map* or node-link map* or visual aid*
 AND
 Reading comprehension or reading achievement* or reading
improvement* or reading strateg* or reading ability*
2. Additional Sources for Information Retrieval:




We will search the reference lists of previously published meta-analyses
on the topic of graphic organizers.
We will contact the first authors of all included studies seeking any
unpublished studies or studies that are currently being conducted on
the topic.
We will search the 2 most recent issues of journals in which at least 3
preliminary included studies are published.
Non-English studies will be included if identified and retrieved from
the databases listed above by enlisting best efforts in English
translation and/or data extraction of relevant information.
C. ASSESSMENT OF METHODOLOGICAL QUALITY
10
The Campbell Collaboration | www.campbellcollaboration.org
We are aware of the varying degrees of methodological quality present in the
research literature. We will examine the study quality as it affects the outcomes of
our analysis and treatment effects. We will use the Downs and Black (1998)
checklist to assess the methodological quality of individual studies. Downs and
Black provide a 27-item assessment tool that includes a rating summary of
categories of: fidelity, attrition, design type, source, and unit of
assignment/analysis. The checklist will be completed by at least 2 reviewers for
all included studies. Any difference in rating between reviewers will be resolved
through discussion to reach a consensus in judgment for all items. In the event
the reviewers are unable to reach consensus, the decision will be made by a third
reviewer. Categories coded according to the checklist will be also used in
moderator variable analyses.
D. DATA EXTRACTION
A coding form will be developed to systematically organize relevant study
information and data. An example of the coding form is included in Appendix A2.
A companion coding book will be developed to operationalize key definitions
encountered in the included studies and to guide the review in extracting and
categorizing statistical data, as well as methodological, substantive and
demographic study features. Among its major categories there will be:
- Instructor equivalence
- Material equivalence
- Relevance and compatibility of settings for the experimental and control
conditions
- Type of statistical data reported
- Type and psychometric qualities of the assessment tools
- Category of the visualization tools/devices
- Fidelity of treatment implementation
- Instructor’s experience with intervention
- Learners educational level
- Class size
- Language of the instruction delivery
- Subject area
- Delivery method
For quality assurance purposes all included studies will be evaluated using
independent double data extraction/coding. Agreement rates of study inclusion at
the abstract and full text level will be reported as Cohen’s kappa index. Overall
inter-rater reliability of the coding form information will be determined by a third
independent party who will perform a random check of 20 percent of the
included studies’ coding forms. Coding discrepancies will be resolved to
consensus.
E. CALCULATING EFFECT SIZES
We will use statistical metric known as effect size to determine the strength of the
treatment effect. Due to differences in the presentation of the outcome data in
11
The Campbell Collaboration | www.campbellcollaboration.org
studies, various methods will be needed for calculating the effect size. In all cases,
we will use the Comprehensive Meta Analysis 2.0 (Borenstein, Hedges, Higgins,
& Rothstein, 2005) software and statistical conventions for effect size calculations
and data synthesis. We intend to use the following methods to calculate effect
size:
1. Standardized Mean Difference Statistic (Cohen’s d-index)
It is anticipated that the most common data presentation will include posttest
means and standard deviations and the calculation of the individual study effect
size(s) will be conducted using the standardized mean difference (d-index)
statistic (1):
di 
XExperiemental  XControl
sPooled
(1)
The standardized mean difference for the studies that provide post-treatment
outcomes. To calculate the d-index in studies that only use t, F, or p value
statistics, we will use conversion formulas such as those provided by Lipsey and
Wilson (2001) and Hedges, Shymansky, and Woodworth (1989).
2. Sample Size Correction
In order to correct for the potential bias of small sample sizes when calculating
effect sizes, all Cohen’s d estimates then will be converted into unbiased Hedge’s g
estimate (2):

3 
g   1
d
4N  9 

(2)
Using g and the standard error of g, the upper and lower boundaries of the 95th
confidence interval (CI) can be constructed to test the effect size statistical
significance.
F. DATA SYNTHESIS
1. Within Study Synthesis: Some studies may include more than one outcome
measure. Only data reflecting categories of outcomes specified among inclusion
criteria (i.e., reading comprehension, overall reading achievements) will be
extracted and analyzed. In case more than one measure addresses the same
outcome type within each category, individual effects will be extracted and then
either averaged (weighted by sample size when necessary); or, the most
representative measure will be selected to reflect treatment effect. As a result,
each independent comparison (results for treated participants against results for
the corresponding control group) will produce a single effect size in each category
of outcomes. Similarly, we will make sure that dependency issue does not arise
due to multiple representations – whenever the same group of participants is
12
The Campbell Collaboration | www.campbellcollaboration.org
used repeatedly (i.e., the same control group compared to two different treatment
groups), its sample size will be reduced proportionally (in this case, split in half)
in order to avoid its overrepresentation in the final dataset.
2. Across Study Synthesis: Effect sizes calculated within individual studies
will be aggregated into average point estimate – separately for each outcome
category – using random effects model (Lipsey & Wilson, 2001) and tested for
significance.
3. Sensitivity Analysis: To produce most accurate and unbiased aggregation
of the effect sizes, outlier analysis (also referred to as sensitivity analysis by
Hedges & Olkin, 1985) will be performed using a combination of approaches,
namely: a visual examination of a forest plot of the data and a “one study
removed” analysis. Sensitivity analysis seeks to determine if the removal of a
certain number of effect sizes from a distribution of effect sizes increases the fit of
the remaining effect sizes to a simple model of homogeneity without drastically
affecting substantive interpretation of the recalculated mean effect size. Identified
outliers will be removed from the dataset. We will also assess and report any
potential bias that may have contributed to any distortion of the resulting effect
sizes, which may include quality of the study, thoroughness of the reported data,
participant attrition, the size of the participant sample and other variables.
4. Heterogeneity Analysis: We will analyze the extent to which the results of
the different studies are consistently representative of the population in question
through analysis of homogeneity of findings. We will also calculate and report the
I2 statistic as indication of the proportion of true heterogeneity (i.e., above that
expected by sampling error) associated with each distribution of effect sizes. If
heterogeneity analysis indicates that observed variability cannot be explained by
sampling error alone, this variability may result from some systematic variation
in other characteristics of the studies. To further examine sources of this
variability, moderator variable analysis will be performed utilizing the mixed
effects model.
G. PUBLICATION BIAS
It is generally thought that unpublished studies result in inflated measures of the
intervention effect when compared published studies. To assess publication bias,
we will use the trim and fill procedure and visually inspect the resulting Funnel
Plot (Rothstein, Sutton, and Borenstein, 2005). Also, classic Fail-Safe N test
(Borenstein et al., 2005) will determine the number of null effect studies needed
to raise the p-value associated with the average effect above a specified level of .
H. INCOMPLETE REPORTING OF STUDY DATA
We are relying on publically available data to avoid publication bias. Therefore, if
outcome data have been omitted from a study, the study will be excluded.
I. POST HOC SUBGROUPS AND MODERATOR VARIABLE ANALYSIS
13
The Campbell Collaboration | www.campbellcollaboration.org
It may be of value to analyze how particular study features (or moderator
variables) influence the effect of the treatment. In order to determine to what
extent fluctuations in direction and magnitude of effect sizes result from
systematic variations in particular study characteristics, moderator variable
analyses will switch to the mixed effects model. We will analyze a restricted
number of categorical moderator variables, namely:
1. Graphic organizer attributes (i.e., use of linking language, computer based
vs. paper based) and type of graphic organizer (e.g., semantic feature
analysis vs. framed outline)
a. Rationale: There may be a differential impact on the intervention
dependent upon graphic organizer attributes and previous research
suggests effect differences based on the type of graphic organizer
2. Use of graphic organizer (e.g., before, during, or after presentation of
material)
a. Rationale: Logic model suggests that the sequence of presentation
may impact learning and retention of information
3. Graphic organizer constructor (i.e., student, teacher)
a. Rationale: Previous research suggests effect differences based on
constructor
4. Student classification (e.g., learning disabled vs. typical learner)
a. Rationale: Previous suggests differential effects of intervention
based on categorical classifications
5. Educational level of participant (e.g., elementary, middle, high school, post
secondary)
a. Rationale: There may be a differential impact on the intervention
associated with the educational level
6. Intervention/Instructional characteristics (i.e., frequency, intensity,
duration, training)
a. Rationale: There may be a differential impact on the intervention
dependent upon implementation characteristics
7. Text attributes (i.e., subject area, passage length)
a. Rationale: There may be differential effects of text attributes on
graphic organizer effectiveness
SOURCES OF SUPPORT
This review was supported by the Campbell Collaboration.
DECLARATIONS OF INTE REST
There are no known conflicts of interest.
REQU EST SUPPORT
n/a
14
The Campbell Collaboration | www.campbellcollaboration.org
AUTHOR(S) REVIEW TEA M
Lead reviewer:
Name: Janet L. Proly, M.A., CCC-SLP
Title: Doctoral Student and Speech-Language Pathologist
Affiliation: University of Central Florida
Address: 1589 Cherry Lake Way
City, State, Province or County: Lake Mary, FL
Postal Code: 32746
Country: USA
Phone: 407-804-1151
Email: [email protected] or [email protected]
Co-author(s):
Name: Jessica Rivers, B.S.
Title: Graduate Student
Affiliation: University of Central Florida
Address: 17408 Harry Jones Rd.
City, State, Province or County: Summerdale, AL
Postal Code: 36580
Country: USA
Phone: n/a
Mobile: 407-267-7159
Email: [email protected]
Co-author(s):
Name: Jamie B. Schwartz, Ph.D.
Title: Associate Professor
Affiliation: University of Central Florida
Address: P.O. Box 162215
City, State, Province or County: Orlando, FL
Postal Code: 32816
Country: USA
Phone: 407-823-4798
Mobile: n/a
Email: [email protected]
Co-author(s):
Name: Evgueni Borokhovski, Ph.D.
Title: Research Assistant Professor, Systematic Reviews Project Manager
Affiliation: Concordia University, Centre for the Study of Learning and
Performance (CSLP)
Address: 1455 De Maisonneuve Blvd. West, LB-581
City, State, Province or County: Montreal, Quebec
Postal Code: H3G 1M8
Country: Canada
Phone: (514) 848-2424 (5334)
Mobile: n/a
Email: [email protected]
15
The Campbell Collaboration | www.campbellcollaboration.org
Co-author(s):
Name: Chad Nye, Ph.D.
Title: Executive Director & Professor
Affiliation: University of Central Florida, Center for Autism & Related Disabilities
Address: 4000 Central Florida Blvd
City, State, Province or County: Orlando, FL
Postal Code: 32816
Country: USA
Phone: 407-823-0003
Mobile: n/a
Email: [email protected]
ROLES AND RESP ONSIBL IITIES
a) Content: Janet L. Proly, Ph.D. student with a focus in Language and
Literacy, Jessica Rivers, M.A. student in the field of Speech-Language
Pathology; Jamie Schwartz, Associate Professor of Language and Literacy
b) Systematic review methods: Chad Nye, Professor, Department of
Communication Sciences & Disorders
c) Information retrieval: Support/guidance needed. Janet Proly, Jessica
Rivers have had training in electronic database searching, and have access
to information specialists at UCF to support information retrieval.
d) Statistical analysis (meta-analysis): Evgueni Borokhovski, Concordia
University
TIMEFRAME
The review will be updated as per the Campbell guidelines for maintaining the
document.
16
The Campbell Collaboration | www.campbellcollaboration.org
REFERENCES
Aarnoutse, C., Van Leeuwe, J., Voeten, M., & Oud, H. (2001). Development of
decoding, reading comprehension, vocabulary and spelling during the
elementary school years. Reading and Writing: An Interdisciplinary
Journal, 14, 61-89.
Alexander, K., Entwisle, D., & Dauber, S. (2003). On the success of failure: A
reassessment of retention in the primary school grades. 2nd Edition.
New York: Cambridge University Press.
Alexander, K., Entwisle, D., & Horsey, C. (1997). From first grade forward: Early
foundations of high school dropout. Sociology of Education, 70, 87-107.
Alvermann, D. E., Boothby, P. R., & Wolfe, J. (1984). The effect of graphic
organizer instruction on fourth graders' comprehension of social studies
text. Journal of Social Studies Research, 8(1), 13-21.
Ausubel, D. P. (1960). The use of advance organizers in the learning and retention
of meaningful verbal material. Journal of Educational Psychology, 51,
267-272.
Ausubel, D. P. (1963). The psychology of meaningful verbal learning. New York:
Grune & Stratton.
Borenstein, M., Hedges, L., Higgins, J., & Rothstein, H. (2005). Comprehensive
meta-analysis. (Version 2). Englewood, NJ: Biostat.
Brown, J. I., Fishco, V. V., & Hanna, G. (1993). Nelson-Denny Reading Test:
Technical report forms G&H. Chicago: Riverside.
Chall, J. S., Jacobs, V. A., & Baldwin, L. E. (1990). The reading crisis: Why poor
children fall behind. Cambridge, MA: Harvard University Press.
Darch, C., & Gersten, R. (1986). Direction-setting activities in reading
comprehension: A comparison of two approaches. Learning Disability
Quarterly, 9(3), 235-43. Retrieved from
http://www.jstor.org/stable/1510469
Downs, S. H., & Black, N. (1998). The feasibility of creating a checklist for the
assessment of the methodological quality both of randomised and nonrandomised studies of health care interventions. Journal of Epidemiology
Community Health, 52, 377-384.
Duke, N. K. (2000). 3.6 minutes per day: The scarcity of informational texts in
first grade. Reading Research Quarterly, 35, 202-224.
Dunston, P. J., & Ridgeway, V. G. (1990). The effect of graphic organizers on
learning and remembering information from connected discourse. Forum
for Reading, 22(1), 15-23.
Ekstrom, B., Goertz, M., Pollack, J., & Rock, D. (1986). Who drops out and why?
Findings from a national study. College Teachers Record, 87(3), 356-373.
Foley, A. (1987). The effects of a mapping training program on the reading
comprehension of middle school students (Doctoral Dissertation,
University of Washington, 1986). Dissertation Abstracts International,
47(12), 4276.
Griffin, C., & Tulbert, B. (1995). The effect of graphic organizers on students'
comprehension and recall of expository text: A review of the research and
implications for practice. Reading and Writing Quarterly: Overcoming
Learning Difficulties, 11(1), 73-89.
17
The Campbell Collaboration | www.campbellcollaboration.org
Griffin, C. C., Simmons, D. C., & Kameenui, E. J. (1991). Investigating the
effectiveness of graphic organizer instruction on the comprehension and
recall of science context by students with learning disabilities. Reading,
Writing, and Learning Disabilities, 7, 355-376.
Gregg, M., & Sekeres, D. C. (2006). Supporting children’s reading of expository
text in the geography classroom. The Reading Teacher, 60 (2), 102-110.
Guastello, E. F., Beasley, T. M., & Sinatra, R. C. (2000). Concept mapping effects
on science content comprehension of low-achieving inner-city seventh
graders. Remedial and Special Education, 21(6), 356-365.
Hedges, L. V., & Olkin, I. (1985). Statistical methods for meta-analysis. Orlando,
FL: Academic Press.
Hedges, L. V., Shymansky, J. A., & Woodworth, G. (1989). A practical guide to
modern methods of meta-analysis. Washington, DC: National Science
Teachers Association.
Horton, P., McConney, A., Gallo, M., & Woods, A. (1993, January). An
investigation of the effectiveness of concept mapping as an instructional
tool. Science Education, 77(1), 95-111. doi:10.1002/sce.3730770107
Hudson, P., Lignugaris-Kraft, B., & Miller, T. (1993). Using content
enhancements to improve the performance of adolescents with learning
disabilities in content classes. Learning Disabilities Research and
Practice, 8, 106-126.
Kim, Ae-Hwa, Vaughn, S., Wanzek, J., & Shangjin Wei, J. (2004). Graphic
organizers and their effects on the reading comprehension of students with
LD: A synthesis of research. Journal of Learning Disabilities, 37(2), 105118. Retrieved from
http://web.ebscohost.com.ezproxy.lib.ucf.edu/ehost/pdfviewer/pdfviewer
?vid=6&hid=109&sid=e90d6ddf-97ea-4faf-b60feea286426d10%40sessionmgr111
Leach, J., Scarborough, H., & Rescorla, L. (2003). Late emerging reading
disabilities. Journal of Educational Psychology, 95, 211-224.
Lipsey, M. W., & Wilson, D. B. (2001). Practical meta-analysis. Thousand Oaks,
CA: Sage Publications.
Lipson, M. (1995). The effect of semantic mapping instruction on prose
comprehension of below-level college readers. Reading Research and
Instruction, 34(4), 367-378.
McCrudden, M. T., Schraw, G., Lehman, S., & Poliquin, A. (2007). The effect of
causal diagrams on text learning. Contemporary Educational Psychology,
32, 367-388.
Moore, D., & Readence, J. (1980). A meta-analysis of the effect of graphic
organizers on learning from text. In M. L. Kamil & A. J. Moe (Eds.),
Perspectives in reading research and instruction: Twenty-ninth yearbook
of the National Reading Conference (pp. 213-217). Washington, DC:
National Reading Conference.
Moore, D., & Readence, J. (1984, September). A quantitative and qualitative
review of graphic organizer research. Journal of Educational Research,
78(1), 11. Retrieved from
http://web.ebscohost.com.ezproxy.lib.ucf.edu/ehost/pdfviewer/pdfviewer
?vid=5&hid=109&sid=000c5177-e844-4804-addd2c07d3b481fb%40sessionmgr112
18
The Campbell Collaboration | www.campbellcollaboration.org
National Assessment of Education Progress (NAEP) (2007). The nation’s report
card: Reading 2007. Retrieved from http://nces.ed.gov/nationsreportcard
Nesbit, J., & Adesope, O. (2006). Learning with concept and knowledge maps: A
meta-analysis. Review of Educational Research, 76(3), 413-448.
Nippold, M. A. (2007). Later language development: School-aged children,
adolescents, and young adults, 4th Edition. Austin, TX: PRO-ED, Inc..
Novak, J. D. & Musonda, D. (1991). A twelve-year longitudinal study of science
concept learning. American Educational Research Journal, 28, 117-153.
Perie, M., Moran, R., & Lutkus, A. D. (2005). NAEP 2004 Trends in academic
progress: Three decades of student performance in reading and
mathematics (NCES 2005–464). U.S. Department of Education, Institute
of Education Sciences, National Center for Education Statistics.
Washington, DC: Government Printing Office.
Rewey, K., Dansereau, D., & Peel, J. (1991, July). Knowledge maps and
information processing strategies. Contemporary Educational
Psychology, 16(3), 203-214. doi:10.1016/0361-476X(91)90021-C
Rothstein, H. R., Sutton, A. J. & Borenstein, M. (2005). Publication bias in metaanalysis: Prevention, assessment and adjustments. England: Wiley.
Schrank, F. A., Becker, K. A., & Decker, S. (2001). Calculating
ability/achievement discrepancies between the Wechsler Intelligence
Scale for Children–3rd Edition and the Woodcock-Johnson III Tests of
Achievement (Woodcock-Johnson III Assessment Service Bulletin No. 4).
Itasca, IL: Riverside Publishing.
Snow, C. (2002). Reading for understanding: Toward and R&D program in
reading comprehension. Santa Monica, CA: RAND Corporation.
Troyer, S. J. (1994). The effects of three instructional conditions in text structure
on upper elementary students' reading comprehension and writing
performance. Paper presented at the annual meeting of the American
Educational Research Association. New Orleans, LA.
Valentine, J. C., & Cooper, H. (2008). A systematic and transparent approach for
assessing the methodological quality of intervention effectiveness
research: The study design and implementation assessment device (Study
DIAD). Psychological Methods, 13(2), 130-149.
Weaver, C. A., & Kintsch, W. (1991). Expository text. In R. Barr, M. Kamil, P.
Mosenthal, & P. D. Pearson (Eds.), Handbook of reading research (pp.
230-245). New York: Longman.
19
The Campbell Collaboration | www.campbellcollaboration.org
TABLES AND APPENDICIES
Appendix A2
Coding Form
Examining the Effects of Graphic Organizers on Reading Comprehension of
Expository Text in Elementary through College aged Students
1. Author:
______________________________________________________
2. Year of Publication: ______________________________________
3. Title:
________________________________________________________
________________________________________________________
4a. Article # ________
4b. Database: ___________________________
5a. Format:________________
5b. File name: ___________________
6. Did the study specify the use of Graphic Organizers/Concept maps/Cognitive
maps/Link-Node maps/Semantic maps? Yes No Inconclusive Not Sure
Comments:____________________________________________________
****If No....STOP!!! ***If Inconclusive or not sure...consult with other group
members
7. Did the study specify the use of Expository Text/Informational Text/or Content
Area Text in the treatment or materials used? Yes
No
Inconclusive
Not Sure
Comments: ____________________________________________________
**** If No….STOP!!! *** If Inconclusive or not sure...consult with other group
members
8. Did the study specify reading comprehension or reading achievement as an
outcome measure? Yes
No
Inconclusive
Not Sure
Comments: ____________________________________________________
**** If No….STOP!!! *** If Inconclusive or not sure...consult with other group
members
20
The Campbell Collaboration | www.campbellcollaboration.org
9. Did the study specify a popuation in kindergarten-postsecondary school?
Yes
No
Inconclusive
Not Sure
Comments: ____________________________________________________
**** If No….STOP!!! *** If Inconclusive or not sure...consult with other group
members
10. Is the study a randomized control trial? Quasi? Yes
No Inconclusive Not Sure
Comments: ____________________________________________________
**** If No….STOP!!! *** If Inconclusive or not sure...consult with other group
members
11. Is the study a single-subject design? Yes
No
Inconclusive
Not Sure
Comments: ____________________________________________________
**** If YES….STOP!!! *** If Inconclusive or not sure, consult with other group
members
Study Characteristics:
12. Source:
1-Journal Name:
________________________________________________________
2-Book or Book Chapter: _________________________________________________
3-Unpublished
4-Doctoral Dissertation
5-Master’s Thesis
6-Other
_______________________________________________________________
13. Stated Hypothesis/Research Question(s): (p.
)
1.____________________________________________________________
2.____________________________________________________________
14. Did Outcomes support the hypothesis: (answer Y/N/Inconclusive for each
question) (p.
)
1. Y N I Comments: ______________________________________________
2. Y N I Comments: ______________________________________________
21
The Campbell Collaboration | www.campbellcollaboration.org
15. Participant Characteristics: (p.
)
Table 1
FollowPre-Post
Group
Pre n
up
Attrition
Follow-
Attrition
%
up n
%
Post n
Comments
Treatment
Comparison
1
Comparison
2
Control
Table 2
Group
Group
Male
Female
Group
Mean
Mean
Mean
Grade
Percent
age
Age
Age
Level
Male
Comments
Treatment
Comparison
1
Comparison
2
Control
16. Socio-Economic Status: (p.
Upper
Middle
Lower
Mixed
)
Not Reported
Comments:____________________________________________________
22
The Campbell Collaboration | www.campbellcollaboration.org
17. Geographic Setting: (p.
Urban
Suburban
Rural
)
Mixed
Not Reported
Comments:____________________________________________________
18. Type of Institution: (p.
Public institution
Private institution
19. Race/Ethnicity: (p.
White/Caucasian
)
Not Reported
)
Hispanic/Latino
Other:_________
Mixed
African American
Asian
American Indian
Mixed
Not Reported
Comments:_____________________________________________________
20. Group Classification (p.
Learning disabled/disordered
Reading Disability- Not LD
Normal achieving students
)
Speech/Language disabled/disordered
Reading Difficulty/Struggling Reader
At-risk
Mix of
________&_________&________
21. Group Classroom Placement (p.
General education classroom
Mixed
Self-contained
Special education classroom
Other
22. Group Reading ability (p.
At-risk
)
)
Struggling reader (Reading below grade level
Reading disabled/disordered
Below
25th
Typical
percentile on reading scores
Test:_______________
Comments:_________________________________________________________
Intervention Characteristics: (p.
)
23. Type of Intervention:
_______________________________________________________
23
The Campbell Collaboration | www.campbellcollaboration.org
24. Type of Graphic Organizer used (as reported by author):
_____Semantic map
_____Cognitive map
_____Semantic Feature Analysis
_____Syntactic Feature Analysis
_____Cognitive Map with a mnemonic
_____Framed Outline
_____Concept map
_____Cognitive map w/o a mnemonic
_____Structured Overview
_____Other: _________________________________
_____NR
Additional descriptive information about graphic organizer:
__________________________________________________________________
__________________________________________________________________
25. Graphic organizer characteristics present: (as explicated in text of study or
demonstrated through pictorial representation in study):
(Check that which applies)
___ a. Nodes or shapes used to display ideas
___b. Nodes or shapes absent in display of ideas
___c. Table format used to display ideas
___d. Lines that connect ideas
___e. One directional arrows attached to lines connecting ideas
___f. Bi-directional arrows attached to lines connecting ideas
___g. Linking language (using text) connecting ideas
___h. Symbolic linking language (using symbols) connecting ideas
___i. Coding system identified for linking language connecting ideas (i.e., straight line with
diagonal line through it connecting 2 ideas means x)
___j. Coding system identified for representation of ideas (i.e., circles used for one type of
concept and squares used for another type of concept)
___k. Information flow uses a top-down approach (i.e., broader concepts appearing on top of
subordinate concepts)
___l. Information flow shows a web like design (i.e., major concept in the center with other ideas
branching outward)
Additional descriptive information about graphic organizer characteristics:
___________________________________________________________________
___________________________________________________________________
24
The Campbell Collaboration | www.campbellcollaboration.org
26. Was the text presented on a computer?
Text only
Graphic organizer only
Both
NR
26a. If computer based: Was hypertext used in text?
Yes
No
Other/describe:___________________________________
26b. If computer based: Were links to other sources used?
Yes
No
Other/describe:___________________________________
26c. If computer based and if links were used, what kind of links were
provided?
Links to pictures
Links to related websites
Other/describe: ___________
27. Was the graphic organizer created/presented on a computer or paper?:
_____Created on Computer
_______Created on Paper
_____Presented on Computer
_______Presented on Paper
_____NR
27a. If computer based: Was outcome measure presented via computer?
Yes
No
Details__________________________________
28. Was the graphic organizer created from a template or created independently?:
Template (from ________________)
Independently created
29. How much text does the graphic organizer contain in each
node/cell/concept/idea?:
One-word
Phrase
Sentence
NR
25
The Campbell Collaboration | www.campbellcollaboration.org
NR
30. How many different graphic organizers were used to depict the text material?
One
Two-Five
More than five
NR
31. Who constructed the Graphic Organizer?
Teacher/Researcher constructed
Student constructed
Partial Researcher/Teacher constructed
Mix
NR
32. Were the students given specific instruction on how to use the GO?
Y
N
Comments: _____________________________________________________
33. How much instruction on how to create the graphic organizer was given?
_____Low- Equal or less than 1 day;
frequency _________ duration _________ total minutes _______
_____Medium- More than 1 day to less than or equal to 1 school week;
frequency__________
duration__________ total minutes ______
_____High- More than 1 school week;
frequency__________
duration__________ total minutes ______
_____NR
34. When was the Graphic Organizer used?
_____Graphic Advance Organizer (before material presented)
_____Supplemental Graphic Organizer (during)
_____Graphic Post Organizer (after material presented)
_____Other:__________________
_____NR
35. Type of Expository Text Used:
Science
Social Studies
History
Psychology
Other:__________________________________
26
Biology
NR
The Campbell Collaboration | www.campbellcollaboration.org
Chemistry
36. Did the intervention post-tests use novel or familiar text?
Novel
Familiar
NR
37. Did the expository passage come from a textbook?
Yes
No
NR
38. What was the length of the expository passage?__________________
39. What was the readability level of the expository passage? __________
40. Was the readability level of the expository passage consistent with the age/level
of the participants?
Yes
No
NR
41. Total Length of Intervention Program:
______________________________________________________________
42. Frequency of Intervention:
______________________________________________________________
43. Number of Intervention Sessions:
______________________________________________________________
44. Length of Intervention Session:
______________________________________________________________
45. Who Delivered the Intervention:
______________________________________________________________
46. Was the instructor randomly assigned?
______________________________________________________________
47. Did the same instructor administer the intervention to all treatment groups?
______________________________________________________________
48. Did the same instructor administer the intervention to all treatment & control
groups?
27
The Campbell Collaboration | www.campbellcollaboration.org
______________________________________________________________
49. Where did the intervention take place? (e.g. classroom, clinical setting, postsecondary school)
______________________________________________________________
Design Characteristics: (p.
)
50. Fidelity:
Intervention implemented as described?
Yes
No
Unclear
51. Recruitment Pool
_____Referral
_____Score on standardized test _________
_____Existing Group
_____Wait List
_____Volunteer
_____Other
_____NR
52. Subject Assignment
_____Individual Random
_____Whole Group Random
_____Individual Matched-Random
_____Matched Non-Random
_____Non-Matched Non-Random
_____Other ___________
53. Was random assignment concealed from:
Researcher
no
Participant
no
yes
NR
Intervener
no
yes
NR
Assessor
no
yes
NR
yes
NR
NR
54. Blinding
Researcher
no
yes
Participant
no
yes
NR
1. Single blinding
Intervener
no
yes
NR
2. Double blinding
28
Was the blinding:
The Campbell Collaboration | www.campbellcollaboration.org
NR
Assessor
no
yes
NR
3. Triple blinding
55. Attrition _____________________ NR
56. Was this study funded by a grant? Y
N
NR
Name funding source/org: ____________
Outcome Measure: (p.
)
57. What was the format of the outcomes measured?
_____Forced choice (e.g. mulitple choice, true-false, fill-in-the-blank)
_____Writing (e.g. free recall, short answer)
_____Both
_____NR
56. POST-TEST: (p.
)
Norm
Group
Ref
Who
Criterion
Who
Rating
Who
Test
Admin.?
Ref Test
Admin.?
Scale
Admin.?
Treatment
Comparison
1
Comparison
2
Control
29
The Campbell Collaboration | www.campbellcollaboration.org
Comments
57. DATA EXTRACTION: (p.
Two Groups
1. Group:
)
2. Group:
Compared
Outcomes
Groups
Mean
SD
N at Posttest
N at Follow-up
d-Index
F value
Chi-square
t value
t value
U value
30
The Campbell Collaboration | www.campbellcollaboration.org