A Research-Based
Framework for
Houghton Mifflin Harcourt
GO Math!
Grades K–6
1
A Research-Based Framework for
Houghton Mifflin Harcourt
GO Math!
Contents Page
The Research Base: Write-In Student Editions
• Background........................................................................................... 5
• Strand 1: Writing to Learn..................................................................... 5
• Strand 2: Vocabulary.............................................................................. 7
• Strand 3: Scaffolding............................................................................. 9
• Strand 4: Metacognition...................................................................... 11
• Strand 5: Graphic Organizers............................................................... 13
• Bibliography........................................................................................ 16
Experimental Efficacy Studies
• Evaluation of the effectiveness of the concept development
and vocabulary development components......................................... 19
• Evaluation of the effectiveness of the Strategic Intervention
and Intensive Intervention materials on student’s mathematical
skills and strategy use........................................................................ 36
• Evaluation of Instructional Effectiveness of GO Math!...................... 56
GO Math! Studies
• Whitley School District (KY)................................................................. 58
• Kimper Public School: Pikeville, KY..................................................... 59
• Arthur T. Cummings Elementary: Winthrop, MA................................... 60
• Speake Public School: Danville, AL...................................................... 61
2
3
The Research Base
When judging a mathematics program, it is imperative to understand the
distinction between theories or expert opinion on how students learn and the
research base that provides evidence to support such theories and opinions. This document illustrates how the Houghton Mifflin Harcourt GO Math!
write-in Student Editions are based on the collective knowledge and expertise
of noted scholars and educators as well as numerous research studies from
various sources.
4
Background
The purpose of this report is to demonstrate clearly and explicitly the scientific research base Houghton Mifflin Harcourt utilized to develop
the Houghton Mifflin Harcourt GO Math! Student Editions. Five major research strands underpin the program: Writing to Learn, Vocabulary,
Scaffolding, Metacognition, and Graphic Organizers. These strands identify the key components of mathematics instruction identified by
recent research.
To help readers make the connections between the research strands and our Student Editions, the following sections are used within each strand:
• Defining the Strand. This section summarizes the terminology and provides an overview of the research related to the strand.
•Research that Guided the Development of Houghton Mifflin Harcourt GO Math! Student Editions.
This section identifies subtopics within each strand and provides excerpts from and summaries of relevant research
on each subtopic.
• From Research to Practice. This section explains how the research data is exemplified in Houghton Mifflin Harcourt GO Math!
Student Editions.
The combination of the major research recommendations and the related features of Houghton Mifflin Harcourt GO Math! Student Editions
should help readers better understand how Houghton Mifflin Harcourt GO Math! incorporates research into its instructional design.
A complete bibliography of all works cited is provided at the end of this section.
Strand 1: Writing to Learn
Defining the Strand
For most students, writing is a fundamental part of school. Nearly all students are asked to engage in some sort of writing on a daily basis, and the
writing they do is typically used to determine whether or not they know or understand something they have been taught. Producing pieces of writing
in order to demonstrate skills, knowledge, and understandings is a common and valuable purpose for classroom writing. There is another purpose for
writing in the classroom, however, that is equally as important – writing to learn.
Regardless of the content area, the very act of writing can help students to process new information, make sense of complex ideas, and connect to
their prior knowledge and experiences (Knipper & Duggan, 2006). According to Vygotsky (1962), such cognitive functions as analyzing and synthesizing develop more fully through writing engagement. Lance and Lance (2006), who use the term “exploratory writing” to refer to writing that has
as its goal idea investigation and discovery, contend that such writing encourages students to make sense of new ideas for which they do not yet
have a solid understanding. Research has shown that learners become more engaged in the learning process when they are asked to explain and
reflect on their thinking processes (Surbeck, 1994; Good & Whang, 1999; Hettich, 1976).
Often, in addition to or instead of writing, students choose to draw pictures in order to make sense of and reflect on new content. Research suggests
“that through drawing [students] are not only able to see what they are thinking, they are also able to play around with and transform their ideas”
(Brooks, 2009, p.319). Regardless of whether students represent their ideas in writing or through drawings, the very act of putting their ideas down
on paper can help students work through confusion and make sense of complex ideas, ultimately contributing to their academic success.
5
Research that Guided the Development of the Houghton Mifflin Harcourt GO Math!
Student Editions
Tiering and scaffolding: Two strategies for providing access to important mathematics
The purpose of this study was to determine whether students who wrote about their executive processes of problem-solving when solving math
problems achieved at higher levels in mathematics than a control group that did not engage in the same type of writing. Results indicate that the
students who wrote about their problem-solving processes made greater mathematical achievements than those who did not. The researchers
conclude that “writing about the executive processes of problem-solving and the problem-solving process in general may not only improve students’
problem-solving performance but may also help students more clearly understand the problem-solving process” (Williams, 2003, p. 187).
Writing in mathematics
Researchers have found that students’ conceptual understanding and problem-solving skills improve when they are encouraged to make sense of
mathematics by writing about… their mathematical thinking (Putnam, 2003).
Writing in mathematics
The results of this study suggest that students who often struggle to communicate their problem-solving processes and explain their mathematical
reasoning orally can more easily do so in writing (Baxter, Woodward, & Olson, 2005).
Writing in mathematics
“Writing in math class … provide[s] a way for students to reflect on their own learning and to explore, extend, and cement their ideas about the
mathematics they study” …it supports learning because it requires students to organize, clarify and reflect on their ideas.” (Burns, 2004, p. 30).
“Mathematics instruction should… teach [students] to monitor and reflect on their problem solving processes. Writing enhances… [this] skill,”
(Burns, 2004, p. 31).
Writing to learn mathematics
“…students are asked to ‘describe,’ ‘compare,’ ‘investigate,’ ‘explain.’ This kind of question requires an answer in written form. It encourages students to think about their thinking and to better understand that mathematics is more than a lot of short symbolic answers” (Russek, 1998, p. 40).
Writing and the ecology of learning
Writing develops thought processes useful in doing mathematics: abilities to define, classify, or summarize; methods of close, reactive reading; metacognition, an awareness of one’s own thinking and learning; and an awareness of attitudes toward mistakes and errors” (Connolly, 1989, p. 120).
Writing to Reduce Math Anxiety
“[Writing] is used to open doors of communication with students who may have math anxiety or who have ‘I hate math! Feelings…”
(Russek,1998, p. 36).
“Studies showed writing to have a positive impact predominately on mathematics anxiety, the acquisition of problem solving skills and the use of
cognitive and metacognitive processes” (Taylor & McDonald, 2007, p. 640).
“Another practical idea for addressing math anxiety is to use journal writing for students to express their understanding of mathematical concepts”
(Furner & Duffy, 2002, p. 70).
“Writing… helps many students become comfortable with mathematical terms and ideas” (Wadlington & Wadlington, 2008, p. 5–6).
6
From Research to Practice
The Houghton Mifflin Harcourt GO Math! Student Editions are designed to provide students with numerous opportunities to write about and
reflect on the processes they used to solve problems and make sense of new mathematical concepts. Throughout the Student Editions, students are
asked to write in response to prompts that ask them to engage in the following types of thinking and reflection:
Explain approaches to solving problems. Students are asked to describe the steps they went through in order to arrive at solutions to
problems. Doing so helps students identify and become more aware of their own processes, which will help them transfer those processes to
more complex problems they will encounter later on.
Reflect on information use. In order to help students think about the types of information with which they are provided to solve different
kinds of math problems, students are asked to consider how they used certain pieces of information to help them arrive at solutions. Writing
about and reflecting on information use can help students identify and clear up confusion and make better use of information in the future.
Draw pictures and diagrams to support problem-solving. Students are asked to represent their ideas and problem-solving processes
by drawing pictures or representing their thoughts on paper in other non-verbal ways. Doing so helps students see what they are thinking
and makes abstract ideas more concrete.
Because the Student Editions are write-in, students can write and draw in the same space in which they are making sense of and solving problems.
Avoiding the need to transfer ideas and responses to a separate piece of paper helps ensure that students’ thoughts will be uninterrupted. Furthermore, recording ideas in the same space that information is presented will help students come back to, make sense of, and benefit from their written
reflections in the future.
Strand 2: Vocabulary
Defining the Strand
Sometimes, what we say (or what we mean to say) is misunderstood by others. The words we choose are vital components of our communication
with others. Whether that communication is heard or spoken, read or written, or viewed or performed, the vocabulary we select to convey our
message is critical.
Broadly defined, vocabulary is knowledge of words and word meanings. It is important to note, however, that vocabulary does not solely consist
of knowing words and their meanings; vocabulary encompasses comprehending how words are used in oral and written formats. As Steven Stahl
states, “Vocabulary knowledge is knowledge; the knowledge of a word not only implies a definition, but also implies how that word fits into the
world” (Stahl, 2005).
Vocabulary knowledge is fundamental to learning in school and throughout life. In order to comprehend what is taught or encountered, students
must have access to the meanings of words so that they can understand what is being said or written. Because most of students’ success in school
and beyond depends upon their ability to read and write while showing understanding, there is a need to offer instruction that equips students with
the skills and strategies necessary for lifelong vocabulary development. Research shows that “By giving students explicit instruction in vocabulary,
teachers help them learn the meaning of new words and strengthen their independent skills of constructing the meaning of text” (Kamil et al.,
2008, p. 11).
7
Research that Guided the Development of the Houghton Mifflin Harcourt GO Math!
Student Editions
Vocabulary to Communicate Mathematically
“…students should have many opportunities to use language to communicate mathematical ideas…Opportunities to explain, conjecture and
defend one’s ideas orally and in writing about mathematics is an integral part of learning mathematics” (NCTM, 1989, p. 78).
Students need to know the meaning of mathematics vocabulary words—whether written or spoken—in order to understand and communicate
mathematical ideas. “…terms, phrases, and symbols are essential in communicating mathematical ideas; and becoming fluent with them is vital for
children’s mathematical learning” (Rubenstein & Thompson, 2002, p. 107).
“[Students] learn to use language to focus on and work through problems, to communicate ideas coherently and clearly…” (Martinez & Martinez,
2001, p. 5).
“The language of mathematics is an important component of our instruction…We teach through the medium of language. It is our major means
of communication” (Thompson & Rubenstein, 2000, p. 568).
Vocabulary to Increase Achievement in Mathematics
Research reveals that knowledge of mathematics vocabulary directly affects achievement in arithmetic, particularly problem solving. Earp notes,
“Reading comprehension and arithmetic comprehension tend to be positively related. Almost without exception instruction in vocabulary and/or
reading skills in arithmetic paid off in terms of greater achievement, especially in the area of problem solving (Earp, 1970, p. 531).
Research by Stahl and Fairbanks indicates that student achievement will increase by 33 percentile points when vocabulary instruction focuses on
specific words that are important to what students are learning (Stahl & Fairbanks, 1986).
“Enhancing students’ academic background knowledge…is a worthy goal of public education from a number of perspectives. In fact, given the
relationship between academic background knowledge and academic achievement, one can make the case that [vocabulary instruction] should be
at the top of any list of interventions intended to enhance student achievement” (Marzano, 2004, p. 4).
Vocabulary to Connect Concepts and Terminology
“Establishing connections between relationships of mathematical concepts and terminology is essential” (Renne, 2004, p. 258).
“The language of mathematics is an important component of our instruction…Students build understanding as they process ideas through
language” (Thompson & Rubenstein, 2000, p. 568).
As Usiskin’s research indicates, “If a student does not know how to read the mathematics…it is difficult to register the mathematics”
(Usiskin, 1996, p. 236).
From Research to Practice
The Houghton Mifflin Harcourt GO Math! Student Editions were designed to introduce students to the mathematical vocabulary necessary to
build on learning in mathematics. Throughout the Student Editions, students are presented with vocabulary terms relevant to the mathematics they
are learning.
8
The vocabulary is introduced and reinforced through teacher instruction and student practice and review:
Explain meanings of words and how they are used. In addition to teachers explaining vocabulary words and their specialized
meanings or how they might be represented with a sign or a symbol, students are offered reminders in the Student Editions about the
definitions of words. Students are also provided with context for the new vocabulary as related to the concept they are studying.
Practice vocabulary as related to the mathematical concepts. In order to help students understand and use the terminology, they
are presented with problems to practice the terms and concepts introduced.
Review meanings of words and how they are used. Prior to each new lesson, students are given a chance to show what they know
from previous study. Students are given opportunities to use the terms in various ways, such as written responses, flow maps, and fill-in-theblank sentences to further verify what they have learned about a concept.
By addressing vocabulary at both the teacher-level and the student-level, students have greater opportunity to connect vocabulary and concepts.
Teachers have the chance to address new vocabulary and to intercept student misconceptions, and students have the chance to practice and review
what they have learned. Because the worktexts are consumable, students can write notes about the meanings of new words in the places that make
sense to them. Students can also jot down symbols or graphics that help them to make sense of the new words.
Strand 3: Scaffolding
Defining the Strand
Many times learning a concept requires guidance in order to maintain and build on the knowledge that is acquired. When that concept is the
foundation for another concept, it is necessary to ensure that the transition between concepts is carefully supported. Similar to scaffolds used by
contractors to erect a structure, scaffolds are put in place to support students while gaining knowledge in school.
Scaffolding is an educational technique that involves providing support to students as they learn, and gradually decreasing the amount of support
provided until students are completing tasks independently. In scaffolding, students receive support as they reach competence and continue to
develop on their own—building on what they have learned. Vygotsky defined scaffolding as the “role of teachers and others in supporting the
learner’s development and providing support structures to get to that next stage or level” (Raymond, 2000, p. 176).
When scaffolding instruction, the types of scaffolds can vary but should consistently provide adequate support as needed. Scaffolds can be effective in many forms, including but not limited to, activating prior knowledge, modeling, questioning, or using cues or tools. “[Scaffolding] connotes a
custom-made support that can be easily disassembled when no longer needed. It also connotes a structure that allows for the accomplishment of
some goal that would otherwise be either unattainable or quite cumbersome to complete” (Stone, 1998, p. 344).
Research that Guided the Development of the Houghton Mifflin Harcourt GO Math!
Student Editions
Scaffolding to Deepen Mathematical Understanding
Research presented in Adding It Up accounts how scaffolding can be used to improve students’ problem solving. “By offering a subtle hint, posing
a similar problem, or asking for ideas from other students, [the teacher] provides some scaffolding to assist his students as they reason through the
grid problems… without reducing the complexity of the task at hand or specifying exactly how to proceed… thus affording the students an
opportunity to learn by considering and discussing solution strategies” (Kirkpatrick et al., 2001, p. 336). In a study conducted by Williams (2008),
teachers used scaffolding throughout mathematics learning which resulted in helping “…the class as a whole move toward deeper understanding
of the key concepts being studied” (Williams, 2008, p. 327).
9
As noted by Barton and Heidema (2002), one way to scaffold is to make sure students have a firm handle of concepts they need for future learning.
Making sure students have a solid grasp of prior knowledge is important to future knowledge acquisition. This relationship “…has a direct effect on
their acquiring new knowledge and skill. For example, the student who does not understand addition will be ill-equipped to learn multiplication…”
(Barton & Heidema, 2002, p. 4).
Baker, Schirner, and Hoffman (2006) share observations of classroom activities in which scaffolding is occurring. In their observations, “…scaffolding
is provided that will be the support, or foundation for later learning. When these students hear of the addition concept again, it will heighten their
readiness to formally begin a guided inquiry” (Baker, Schirner, & Hoffman, 2006, p. 20).
Scaffolding to Meet Individual Student Needs
In describing how her research shows that scaffolding meets varying student needs, Walker (2008) noted, “After demonstrations, teachers continue
to support or scaffold the new learning. They often provide continuous support and sustain learning by scaffolding students’ attempts. For example,
teachers can deal effectively with inappropriate student responses by using part of the response to probe reasoning” (Walker, 2008, p. 18-19).
This account demonstrates that scaffolding can be used to intervene with students on an individual level by using an incorrect response to
explore thinking.
Research presented in Adding It Up highlights that “Scaffolding…helps to maintain student engagement at a high level” (Kirkpatrick et al., 2001,
p. 336).
When presenting information about how scaffolding works to meet individual student needs, Barton and Heidema observe that “Discovering what
students already know about a topic helps teachers design instruction around the missing knowledge” (Barton & Heidema, 2002, p. 6).
Scaffolding to Build Confidence and Independence
Scaffolding is necessary support to enable students to become responsible for their own learning. As Hyde notes, “Scaffolding does not necessarily
make the problem easier, and the teacher does not do the work for students or show them how to do it. Like scaffolding along the side of a
building that enables the painters to safely work on the outside wall, the scaffolding does not do the work. It enables the person to do it”
(Hyde, 2006, p. 28).
Williams (2008) notes that “Scaffolding tasks allowed students to work independently at appropriately challenging levels, make sense of ideas, and
develop a sense of self-confidence in their mathematics knowledge and skills” (Williams, 2008, p. 329).
“Providing opportunities for… scaffolding enables students to have success at their level and also to be challenged to reach their developmental
potential” (Baker, Schirner & Hoffman, 2006, p. 21).
As highlighted in research by Anghileri, scaffolding in the form of reviewing concepts helps “… to refocus [students’] attention and give them
further opportunity to develop their own understanding rather than relying on that of the teacher” (Angileri, 2006, p. 41). Reviewing helps students
develop their own understanding of mathematics.
Larkin (2001) learned from interviewing and observing teachers who scaffolded instruction that their students became more independent
learners. “Scaffolding principles and techniques can guide teachers to assist students on any grade level to become more independent learners”
(Larkin, 2001, p. 34).
From Research to Practice
The Houghton Mifflin Harcourt GO Math! Student Editions were designed to provide students with ample guidance as they learn mathematical
concepts. Throughout scaffolds exist to help students solidify what they know in order to build on it. Scaffolds are in place to help students in the
following ways:
10
Build meaningful learning experiences. Students are given opportunities to solve problems that are relevant to the world around them.
Offering students meaningful contexts that brush their prior knowledge base is one way to support them as they learn new concepts.
Review and reflect on previous concepts before moving on. In order to prepare students for new learning, they are given a chance
to review the concepts and vocabulary they have learned previously. This scaffold allows for students to show what they know or reflect on
what they have not quite mastered yet, and responsibility for learning can shift to the student.
Complete problems in a graduated way. In order to help students as they practice on their own, graduated questioning is used as a
scaffold. Often students are asked how to solve a problem before they actually solve the problem, or students are given a less difficult
problem to solve before a more difficult one. This helps to organize their thinking as they work on their own.
Provide opportunities to model or show what they can do. In order to meet students’ different learning styles, the worktext includes
several ways to demonstrate learning. Students encounter problems that use manipulatives, graphics, pictures, line graphs, and many other
techniques to help support their thinking. These are scaffolds in place to guide students as needed as they practice independently.
Because the Student Editions are write-in, students are able to flip back and forth between pages in a lesson to look for relevant support to uncover
the answers they need. Students are given space to write and illustrate freely allowing them to explore their own thinking to solve problems as it
makes sense to them. Furthermore, the worktext follows a logical sequence by scaffolding before, during, and after each lesson in order to best
support learning.
Strand 4: Metacognition
Defining the Strand
Developing a range of thinking strategies needed to solve problems and knowing which strategy to choose are important developments for students.
Students need to be aware of what they can do well and what they need to work on. This encourages them to think about their thinking and
learning in order to better solve problems they encounter. Supporting metacognition in school fosters the development of good thinkers who are
successful problem-solvers and lifelong learners. Recognizing, developing, and improving the metacognitive capabilities of students is fundamental
to learning.
Metacognition is thinking about thinking—knowing “what we know” and “what we don’t know”—and how we can use that information. Some
basic metacognitive strategies include connecting new information to that previously learned, selecting thinking strategies purposefully, and
planning, monitoring, and evaluating thinking processes (Dirkes, 1985). Studies show that the use of metacognitive strategies increases learning.
These results suggest that supporting thinking strategies is useful and that independent learning will develop gradually (Scruggs, 1985).
Regardless of the content area in school, problem-solving and other activities provide opportunities for strengthening metacognitive strategies. It
is therefore important to focus student attention on how tasks are accomplished. Emphasizing content goals and process goals will help students
discover that understanding and applying thinking processes expands learning.
Research that Guided the Development of Houghton Mifflin Harcourt GO Math!
Metacognition to Build Mathematical Problem-Solving
“Competent problem solvers are efficient at keeping track of what they know and of how well or poorly their attempt to solve a problem is
proceeding. They continuously ask, ‘What am I doing?’ ‘Why am I doing it?’ ‘How will it help me?’” (Reys, Suydam, Lindquist, & Smith, 1998, p. 27).
11
“[W]hen students… reflect on their own thinking, it makes a significant impact on their ability to solve problems now and in the future”
(Roberts & Tayeh, 2006-07, p. 23
“Individual reflection or interaction with others (both teachers and peers) encourages students to communicate and explain their thinking”
(Reys, Suydam, Lindquist, & Smith, 1998, p. 32).
Metacognition to Increase Performance in Mathematics
Results from a study conducted by Lucangeli, Corndoli, and Tellarini (1998), show that supporting students’ metacognitive thinking and strategy use
has resulted in increased performance in mathematics. Researchers found that the group of students who received metacognitive support
outperformed peers who did receive the same training (Lucangeli, Corndoli, and Tellarini, 1998).
Research by Pogrow (1999) suggests that helping students understand their own metacognitive processes and strategies will help their academic
performance (Pogrow, 1999).
“Students learn more and better when they can take control of their learning by defining their goals and monitoring their progress… Effective
learners recognize the importance of reflecting on their thinking and learning from their mistakes” (NCTM, 2000, p. 21).
Metacognition to Improve Attitudes Toward Mathematics
Research shows that what students recognize and believe about themselves as students of mathematics not only affects their performance, but it
also affects the way they approach mathematics (Campione, Brown, & Connell,1998).
One study found students’ positive attitudes toward mathematics increased through the use of metacognitive approaches in learning. This approach
appears to provide a more advantageous learning atmosphere for students. Support of metacognitive strategies encourages students to be more
active and critical in their learning and thinking (Muin, Sumarmo, & Sabandar, 2006).
Research conducted by Maqsud (1998) examined the effects of metacognitive instruction on students’ mathematics achievement and their attitudes
toward mathematics. The experimental group used metacognitive strategies while learning, but the control group did not. The comparisons of pretest
and posttest measures revealed that the posttest scores for the experimental group were significantly higher than the control group in relation to
general ability, metacognitive awareness, attitude towards mathematics, and mathematics achievement (Maqsud, 1998, pp. 237-243).
From Research to Practice
The Houghton Mifflin Harcourt GO Math! Student Editions were designed to provide students with numerous opportunities to think about their
thinking and learning while solving problems and making sense of mathematical concepts. Throughout, students are asked to respond to prompts
that ask them to engage in the following types of planning, monitoring, and reflecting:
Plan how to solve a problem. Throughout the Student Editions, students are asked to think about the steps they need to go through
in order to solve problems. By planning how to solve a problem, students can determine what they to know, how they will use that
information, and what steps they will need to take.
Monitor success by periodic assessment. In order to help students remember what they have learned, they are given chances to
show what they know throughout. By providing these sections, students are able to assess their own strengths and
weaknesses regularly.
Monitor success by trying other ways. In Houghton Mifflin Harcourt GO Math!, students are prompted to solve a problem
“one way” first, and then they are asked to solve it “another way.” This enables students to gain a better, and more thorough,
12
understanding of mathematical concepts. They are learning to monitor what they know in order to apply it to different ways of
solving problems.
Reflect in a visual or written format. Students are asked to show what they know by drawing pictures or providing written explanations
of what they did and why. Visual and written responses challenge students to think about what they are doing – leading them to
contemplate what it is they are trying to solve, how they are actually going to solve it, and why they are going to solve in this way.
Because the Student Editions are write-in, students can plan directly on the pages as they solve problems. Allowing this flexibility, students can look
to previous lessons or within a lesson for support in planning to solve new material. Students can also write notes or use other strategies, such as
underlining, to think through problems. Furthermore, providing space for drawings and written responses, allows students to develop their metacognitive skill set by writing out what they are thinking. This also allows teachers to read students’ responses enabling them to support weaknesses and
build on successes.
Strand 5: Graphic Organizers
Defining the Strand
Sometimes in order to grasp a concept, it is helpful to think about it visually—using visual representations. Visual representations can be expressed
in many ways; graphic organizers are one way for students to arrange their ideas visually. Graphic organizers come in many varieties and have been
widely researched for their effectiveness in improving education outcomes for students.
Graphic organizers are illustrations used to organize and highlight key content and/or vocabulary (Lovitt, 1994). Graphic organizers are visual
representations that show relationships between facts, terms, and or ideas within a task or when learning a concept. Graphic organizers help
students develop a clearer understanding of concepts by enabling them to connect content in a meaningful way. Graphic organizers can be used for
a variety of purposes before, during, and after instruction, such as drawing out prior knowledge, organizing information, processing information, or
summarizing ideas.
“Graphic organizers are perhaps the most common way to help students generate nonlinguistic representations” (Marzano, Pickering, & Pollock,
2001, p. 75). Research indicates that use of graphic organizers is effective for helping students organize and remember content area information
(Horton, Lovitt, & Bergerud, 1990). Additional research indicates that using graphic organizers can be valuable in teaching students how to represent
problems in an illustrative format and how to determine the operation or operations necessary to solve a problem (Jitendra, 2002).
Research that Guided the Development of the Houghton Mifflin Harcourt GO Math!
Student Editions
Graphic Organizers to Sort Information
In research on using graphic organizers in mathematics, results suggest that “The organizer seemed to provide [students] with a framework that
gave them confidence in their ability to be successful and also required them to think through problem situations before beginning mathematical
calculations…Several factors accounted for the effectiveness of the graphic organizer. Of most significance was the fact that the organizer required
students to slow down and think through each problem. At first, students with more impulsive learning styles resisted the slower process, but they
accepted the organizer as they saw their improved performance and ability to successfully solve problems” (Braselton & Decker, 1994, pp. 280-281).
“Organizing data into a table helps children to discover a pattern and to identify information that is missing. It is an efficient way to classify data
and order large amounts of information or data, and it provides a record so that children need not retrace nonproductive paths or do computations
repeatedly to answer new questions” (Reys, Suydam, Lindquist, & Smith, 1998, p. 78).
13
Monroe & Pendergrass (1997) recommend the use of graphic organizers to “teach to the brain’s natural capacity for thinking and organizing
information” (Monroe & Pendergrass, 1997, p. 3).
Graphic Organizers to Support Varied Learning Styles
“Graphic organizers are effective differentiation strategies that affect student achievement in the classroom by providing a “big picture” view of
concepts and by helping to organize the relationships between concepts” (Shores & Chester, 2009, p. 71).
According to Hyerle, the use of graphic organizers can help learners who need support in making connections between what the assignment is
and how it will be completed. “These visual tools can do double duty by helping the teacher clarify a stated set of objectives and giving individual
students—especially students with special needs—a tool to complete the task” (Hyerle, 1996, p. 62).
Graphic Organizers to Connect Ideas
“Teachers can help students grasp embedded concepts as well as how other concepts are related by demonstrating these relationships with graphic
organizers” (Barton & Heidema, 2002, p. 20).
“[Graphic] organizers are especially helpful in representing abstract information in concrete forms” (Shores & Chester, 2009, p. 71).
“Students who are able to apply and translate among different representations of the same problem situation or of the same mathematical concept
will have at once a powerful, flexible set of tools for solving problems and a deeper appreciation of the consistency and beauty of mathematics”
(National Council of Teachers of Mathematics, 1990, p. 146).
“Visual brainstorming webs, task-specific organizers, and thinking-process maps thus provide a bridge between their own forms and the structures
that are embodied in the text but hidden in the guise of linear strings of words” (Hyerle, 1996, p. 15).
“[Words] on paper, arranged to represent an individual’s understanding of the relationship between words. Whereas conventions of sentence
structure make most writing linear in form, graphic organizers take their form from the presumed structure of relationships among ideas” (Clarke,
1991, p. 30).
In addition to helping students learn how to process, organize, and store new information, regular use of graphic organizers can increase
comprehension, retention, and recall of information (Jones, Palincsar, Ogle, & Carr, 1987).
Research by Braselton & Decker (1994) showed that “After engaging in independent practice with the graphic organizer, students showed marked
improvement in problem solving. This strategy was effective with students of all ability levels” (Braselton & Decker, 1994, p. 278).
From Research to Practice
The Houghton Mifflin Harcourt GO Math! Student Editions were designed to provide students with numerous opportunities to write about and
reflect on the processes they used to solve problems and make sense of new mathematical concepts. Throughout the Student Editions, students are
asked to write in response to prompts that ask them to engage in the following types of thinking and reflection:
Engage in powerful thinking. Before writing in the graphic organizers, students have to engage in powerful information processing and
higher order thinking Students are asked to recognize important information, make decisions about what to do, consolidate information,
show what they know, and solve problems.
Reflect on problems visually. Throughout, students are asked to use pictorial representations to solve problems. Solving
problems this way gives students a chance to show what they know and can do in a non-linguistic way which is an effective way to meet
the needs of diverse learners.
14
Show relationships among information. Students are asked to show what they know by drawing pictures or providing written
explanations of how to set up problems before actually solving them. Using graphic organizers in this way helps students slow down
their thinking in order to plan and sort out the information they have.
Extend understanding of important concepts. Using graphic organizers allows students to record information so that they do not
need to repeat steps or go back over information. All of the information they need is in the graphic organizer so they can build on what
they already know.
Because the Student Editions are write-in, the graphic organizers are ready-to-use for each lesson. Students can demonstrate their understanding of
key concepts visually and in writing as they move through the worktext – without language processing demands getting in the way. Furthermore,
the process of being able to write information directly on the graphic organizers will promote students’ active learning and creativity.
15
Bibliography
Anghileri, J. (2006). Scaffolding practices that enhance mathematics learning. Journal of Mathematics Teacher Education, 9, 33–52.
Baker, A., Schirner, K., & Hoffman, J. (2006). Multiage mathematics: Scaffolding young children’s mathematical learning. Teaching Children
Mathematics, 13(1), 19–21.
Baxter, J. A., Woodward, J., & Olson, D. (2005). Writing in mathematics: An alternative form of communication for academically low–achieving
students. Learning Disabilities Research & Practice, 20(2), 119–135.
Barton, M. L., & Heidema, C. (2002). Teaching reading in mathematics. Aurora, CO: Mid–continent Research for Education and Learning.
Bransford, J. D., Brown, A. L., & Cocking, R. R., Eds. (1999). How People Learn: Brain, Mind, Experience, and School. Washington, D.C.:
National Academy Press.
Braselton, S., & Decker, C. (1994). Using graphic organizers to improve the reading of mathematics. Reading Teacher, 48 (3), 276–81.
Brooks, M. (2009). Drawing, visualization, and young children’s exploration of “big ideas,” International Journal of Science Education, 3(1),
319–341.
Burns, M. (2004). Writing in math. Educational Leadership, 62(2), 30–33.
Campione, J. C., Brown, A. L. & Connell, M. L. (1988). Metacognition: On the importance of understanding what you are doing. In Randall I.
Charles & Edward A. Silver (Eds.), The Teaching and Assessing of Mathematical Problem Solving (pp. 93–114). Reston, Va.: NCTM.
Chang, K., Chen, I., & Sung, Y. (2002). The effect of concept mapping to enhance text comprehension and summarization. The Journal of
Experimental Education 71(1), 5–23.
Clarke, J. H. (1991). Patterns of Thinking. Needham Heights, Mass.: Allyn & Bacon.
Connolly, P. (1989). Writing and the ecology of learning. In P. Connolly & T. Vilardi (Eds.), Writing to learn mathematics and science (pp. 1–14).
New York: Teachers College Press.
Dirkes, M. A. (1988). Self-directed thinking in the curriculum. Roeper Review, 11(2), 92–94.
Earp, N. W. (1970). Observations on teaching reading in mathematics. Journal of Reading, 13, 529–33.
Furner, J. M., & Duffy, M. L. (2002). Equity for all students in the new millennium: Disabling math anxiety. Intervention in School and Clinic,
38(2), 67–74.
Gagnon, J. C., & Maccini, P. (2001). Preparing students with disabilities for algebra. Teaching Exceptional Children, 34(1), 8–15.
Good, J. M., & Whang, P. A. (1999). Making meaning in educational psychology with student response journals. Paper presented at the annual
meeting of the American Educational Research Association, Montreal, Canada.
Hartman, H. (2002). Scaffolding & Cooperative Learning. Human Learning and Instruction (pp. 23–69). New York: City College of City University
of New York.
Hettich, P. (1976). The journal: An autobiographical approach to learning. Teaching of Psychology, 3(1), 60–63.
Horton, S. V., Lovitt, T. C., & Bergerud, D. (1990). The effectiveness of graphic organizers for three classifications of secondary students in content
area classes. Journal of Learning Disabilities, 23, 12–22, 29.
Hyde, A. (2006). Comprehending math: Adapting reading strategies to teach mathematics K–6. Portsmouth, NH: Heinemann.
Hyerle, D. (1998). Visual Tools for Constructing Knowledge. Alexandria, Va.: Association for Supervision and Curriculum Development.
Jitendra, A. (2002). An exploratory study of schema-based word-problem-solving instruction for middle school students with learning disabilities:
An emphasis on conceptual and procedural knowledge. The Journal of Special Education, 36(1), 23–28.
Jones, B. F., Palincsar, A. S., Ogle, D. S., & Carr, E. G. (Eds.). (1987). Strategic teaching and learning: Cognitive instruction in the content areas.
Alexandria, Va. and Elmhurst, Il.: Association for Supervision and Curriculum Development andNorth Central Regional Educational Laboratory.
Kaméenui, E. J., Carnine, D. W., Dixon, R. C., Simmons, D. C., & Coyne, M. D. (2002). Effective teaching strategies that accommodate diverse learners
(2nd ed.). Upper Saddle River, NJ: Merrill Prentice Hall.
Kamil, M. L., Borman, G. D., Dole, J., Kral, C. C., Salinger, T., and Torgesen, J. (2008). Improving adolescent literacy: Effective classroom and
intervention practices: A Practice Guide (NCEE #2008–4027). Washington, DC: National Center for Education Evaluation and Regional
Assistance, Institute of Education Sciences, U.S. Department of Education. Retrieved from http://ies.ed.gov/ncee/wwc.
Kilpatrick, J., Swafford, J., & Findell, B., Eds. (2001). Mathematics LearningStudy Committee, National Research Council: Conclusions and
recommendations. In Adding It Up: Helping Children Learn Mathematics (pp. 407–432). Washington, D.C.: The National Academies Press.
16
Knipper, K. J., & Duggan, T. J. (2006). Writing to learn across the curriculum: Tools for comprehension in content area classrooms.
Reading Teacher, 59(5), 462–470.
Lance, D. M., & Lance, S. (2006). Writing-learn-assignments in content driven courses, Communication Disorders Quarterly, 28(1), 18–23.
Larkin, M. J. (2001). Providing support for student independence through scaffolded instruction. Teaching Exceptional Children, 34(1), 30–34.
Lovitt, S. V. (1994). Strategies for adapting science textbooks for youth with learning disabilities. Remedial and Special Education, 15(2), 105–116.
Lucangeli, D., Corndoli, C., & Tellarini, M. (1998). Metacognition and learning disabilities in mathematics. In T. E. Scruggs & M. A. Mastropieri (Eds.),
Advances in learning and behavioral disabilities (Vol. 12, pp. 219–244). Greenwich, CT: JAI Press.
Maccini, P., & Gagnon, J. (2005). Math graphic organizers for students with disabilities. American Institutes for Research, 1–10.
Maqsud, M. (1998). Effects of metacognitive instruction on mathematics achievement and attitude toward mathematics of low mathematics
achievers. Educational Research, 40(2), 237 – 243.
Martinez, J. G. R., & Martinez, N. C. (2001). Reading and writing to learn mathematics: A guide and a resource book. Boston: Allyn & Bacon.
Marzano, R. J. (2004). Building background knowledge for academic achievement: Research on what works in schools. Alexandria, Va.: Association
for Supervision and Curriculum Development.
Marzano, R. J., Pickering, D. J., & Pollock, J. E. (2001). Classroom Instruction that Works: research-based strategies for increasing student
achievement. Alexandria, Va.: Association for Supervision and Curriculum Development.
Monroe, E. E., & Pendergrass, M. R. (1997). Effects of mathematical vocabulary instruction on fourth grade students. Paper presented at 1997
Brigham Young University Public School Partnership Symposium on Education. ED 414 182.
Muin, A., Sumarmo, U., & Sabandar, J. (2006). Metacognitive approach to improve mathematics skills of high school students. International
Journal of Education, 1(1).
National Council of Teachers of Mathematics. (1989). Curriculum and Evaluation Standards for School Mathematics. Reston, Va.: NCTM.
National Council of Teachers of Mathematics. (1990). Curriculum and Evaluation Standards. Washington, D. C.: NCTM.
National Council of Teachers of Mathematics. (2000). Principles and Standards for School Mathematics. Reston, Va.: NCTM.
National Reading Panel (2000). Report of the National Reading Panel: Teaching children to read: An evidence-based assessment of the scientific
research literature on reading and its implications for reading instruction. Washington, D.C.: National Institute of Child Health and Human
Development, National Institutes of Health.
Pogrow, S. (1999). Systematically using powerful learning environments to accelerate the learning of disadvantaged students in grades 4–8.
In Charles M. Reigeluth (Ed.), Instructional Design Theories and Models, Volume II, A New Pardigm of Instructional Theory. Mahway, NJ:
Lawrence Erlbaum Associates.
Putnam, R. (2003). Commentary on four elementary math curricula. In S. Senk & D. Thompson (Eds.), Standards-oriented school mathematics
curricula: What does it say about student outcomes? (pp. 161–180). Mahwah, NJ: Erlbaum.
Raymond, E. (2000). Cognitive Characteristics. Learners with Mild Disabilities (pp. 169–201). Needham Heights, MA: Allyn & Bacon.
Renne, C. G. (2004). Is a rectangle a square? Developing mathematical vocabulary and conceptual understanding. Teaching Children
Mathematics, 258–263.
Reys, R. E., Suydam, M. N., Lindquist, M. M.,& Smith, N. L. (1998). Helping Children Learn Mathematics. Needham Heights, MA: Allyn & Bacon.
Roberts, S., & Tayeh, C. (2006–07). It’s the thought that counts: Reflecting on problem solving. Mathematics Teaching in the Middle School,
12(5), 232–237.
Rubenstein, R. N. & Thompson, D. R. (2002). Understanding and supporting children’s mathematical vocabulary development. Teaching Children
Mathematics. 107–112.
Russek, B. (1998). Writing to learn mathematics. The WAC Journal, 9, 36–45.
Schoenfeld, A. H. (1987). What’s all the fuss about metacognition? In A. H. Schoenfeld (Ed.), Cognitive science and mathematics education
(pp. 189–215). Hillsdale, NJ: Lawrence Erlbaum Associates.
Scruggs, T. E., Mastropieri, M. A., Monson, J., & Jorgenson, C. (1985). Maximizing what gifted students can learn: Recent findings of learning
strategy research. Gifted Child Quarterly, 29(4), 181–185. EJ 333 116.
Shores, C., & Chester, K. (2009). RTI for School Improvement. Thousand Oaks, CA.: Corwin Press & Council for Exceptional Children.
Stahl, S. A. (2005). Four problems with teaching word meanings (and what to do to make vocabulary an integral part of instruction). In E.H. Hiebert
and M.L. Kamil (eds.), Teaching and learning vocabulary: Bringing research to practice. Mahwah, NJ: Erlbaum.
17
Stahl, S. A., & Fairbanks, M. M. (1986). The effects of vocabulary instruction: A model-based meta-analysis. Review of Education Research, 56(1),
72–110.
Stone, C. A. (1998). The metaphor of scaffolding: Its utility for the field of learning disabilities. Journal of Learning Disabilities, 31, 344–364.
Surbeck, E. (1994). A glimpse through the door: Journal writing with preservice teachers. Childhood Education, 70(4), 232–235.
Taylor, J. A., & McDonald, C. (2007). Writing in groups as a tool for non-routine problem solving in first year university mathematics,
International Journal of Mathematical Education in Science and Technology, 38(5), 639–655.
Thompson, D. R., & Rubenstein, R. N. (2000). Learning mathematics vocabulary: Potential pitfalls and instructional strategies.
Mathematics Teacher, (93)7, 568–574.
Usiskin, Z. (1996). Mathematics as a Language. In Communication in Mathematics, K–12 and Beyond, 1996 Yearbook of the National Council
of Teachers of Mathematics (NCTM), edited by Portia C. Elliott and Margaret J. Kenney, pp. 231–43. Reston, Va.: NCTM.
Vygotsky, L. S. (1962). Thought and language, Cambridge, MA: MIT Press.
Wadlington, E., & Wadlington, P. L. (2008). Helping students with mathematical disabilities to succeed. Preventing school failure, 53(1), 2–7.
Walker, B. (2008). Adjusting instruction to meet students’ needs. (June 2008). Reading Today, 25(6), 18–19.
Williams, K. M. (2003). Writing about the problem-solving process to improve problem-solving performance. Mathematics Teacher, 96(3), 185–187.
Williams, L. (2008). Tiering and scaffolding: Two strategies for providing access to important mathematics. Teaching Children Mathematics. 14(6),
324–330.
18
Experimental Efficacy Studies
The efficacy of Houghton Mifflin Harcourt GO Math! has been well documented through
several recent studies. All of these studies, as well as future planned studies, meet the
various requirements of scientific research as set forth by the No Child Left Behind
legislation, including rigorous, systematics, and objective methods, as well as replicable and
valid results. This scientifically based research documents how Houghton Mifflin Harcourt
GO Math! helps students achieve in mathematics by improving understanding,
achievement and test scores.
Evaluation of the effectiveness of the concept development
and vocabulary development components
Experimental Efficacy
Concept Study and Vocabulary Study
This report describes the results of two efficacy studies to determine the effectiveness of the concept development and vocabulary approaches
evident in Houghton Mifflin Harcourt GO Math!.
Background Information
Houghton Mifflin Harcourt contracted with the Educational Research Institute of America (ERIA) to evaluate the efficacy of lesson
components in terms of their effectiveness in enhancing conceptual understanding and developing content-area vocabulary understanding as a
means to help students understand the mathematics being studied. The effectiveness of the program was evaluated by two separate studies, a
Concepts Study and a Vocabulary Study. The studies were conducted with Grade 3 students as Grade 3 is the mid-point of the grades for which the
program is available.
Research Questions
The following research questions guided the design of the two studies:
Are the concepts development component and vocabulary acquisition component instructionally effective in increasing the mathematics scores of:
• Grade 3 students?
• Lower achieving as well as higher achieving Grade 3 students?
• Grade 3 students who are receiving special services?
• Grade 3 students who are not English proficient as well as those who are English proficient?
• Grade 3 students who are identified as coming from families of low socio-economic status as well as those identified as coming from
families of high socio-economic status?
• Grade 3 students identified as minority students as well as those identified as non-minority students?
19
1. Pretest/posttest comparisons for all students in the sample.
2. Pretest/posttest comparisons for three achievement levels, based on students’
scores on the pretests.
Design and Procedures of the Study
3. Pretest/posttest comparisons for two groups based on whether students were
Research Design receiving special services such as special education.
ERIA followed an experimental
pretest/posttest design
for these studies.
sample
included
fiveon
schools
for the Concepts
and three different
4. Pretest/posttest
comparisons
forThe
two
groups
based
teachers’
ratings Study
of students
schools for the Vocabulary Study.
of the English
teachers volunteered
participate
in one orproficient.
the other of the two studies. Teachers received no specific
asAllbeing
proficientto or
non-English
training for the studies. For5.
theirPretest/posttest
efforts, teachers were
awarded
nominal
product
credits
theyon
could
use to purchase
materials from
comparisons for two groups that
based
teachers’
ratingsclassroom
of students
the Houghton Mifflin Harcourt
Publishers
catalog.
as being
of high
or low socio-economic status.
6. Pretest/posttest comparisons for minority and non-minority students.
Results
of the
Analysis:Concepts
Concepts Study
Results
of the
Analysis:
Study
Total Group
Analysis
Total Group
Analysis
Table 1 and Table
Figure 13provide
the results
for all of the
3 students
the study.
The results included
show statistically
(<.0001)
and Figure
1 provide
theGrade
results
for allincluded
of theinGrade
3 students
in thesignificant
study. The
improvementresults
from pretest
posttest indicating
that such a(<.0001)
change would
have occurredfrom
by chance
fewertothan
once outindicating
of 10,000 times if the study
showto statistically
significant
improvement
pretest
posttest
thatFigure
such1 shows
a change
wouldincrease.
have occurred by chance fewer than once out of 10,000 times if the
were repeated.
the percent
study were repeated. Figure 1 shows the percent increase.
Table 13
Paired t-test Comparison of Pretest/Posttest Scores
HSP MATH
Concepts
Concepts
Study Study
Total Group (N=104)
Pretest Scores
Posttest Scores
Analysis
Mean
Standard
Deviation
Mean
Standard
Deviation
t-test
Significance
49.1
26.0
68.9
23.8
9.584
<.0001
Cohen’s
d statistic
the strength
relationship
was .8.
A statistic
.8.8indicates
a large
effecta large effect
Cohen’s d statistic
for the
strength offor
relationship
was .8.of
A statistic
of .8 indicates
a large
effect size.ofThe
result, therefore,
indicates
size.scores
Theof.8allresult,
indicates
size for the gain
studentstherefore,
in the Concepts
Study. a large effect size for the gain scores of all students in the
Concepts Study.
Figure 1
Page 8
Comparison of Pretest and Posttest Percent Correct
Concepts
HSP MATH
Concepts
Study Study
Total Sample (N=104)
80%
70%
60%
50%
40%
30%
20%
10%
0%
69%
49%
Pretest Mean
Posttest Mean
Concepts Test
Achievement Level Group Analysis
20
In order to divide the students into achievement groups, the pretest scores of the 104 students
Achievement Level Group Analysis
In order to divide the students into achievement groups, the pretest scores of the 104 students were ranked from low to high and then divided into
three equal-sized groups. The low achievement group included 34 students and the average and high achieving groups included 35 students each.
After categorizing the students into these three categories, a paired comparison t-test analysis was conducted using the pretest/posttest scores.
Table 2 and Figure 2 provide the results for each of the three achievement groups. Table 2 shows that students in all three groups showed statistically significant (<.0001) improvement from pretest to posttest indicating that such a change would have occurred by chance fewer than once out of
10,000 times if the study were repeated.
Figure 2 shows the percent of raw score increase for the three achievement groups. By design, because of how the sorting for the student groups
in this achievement-level group analysis was done, the low and average groups started out with lower percentage correct scores. However, all three
groups showed significant improvement.
Table 2
4
Paired t-test Comparison of Pretest/Posttest Scores
HSP MATH
Concepts
Concepts
Study Study
Three Achievement Level Groups
(N=104)
Pretest Scores
Posttest Scores
Analysis
Group (N)
Mean
Standard
Deviation
Mean
Standard
Deviation
High
Achievement
(35)
77.7
8.9
88.3
9.0
6.671
<.0001
Average
Achievement
(35)
49.6
8.0
66.5
16.1
6.011
<.0001
Low
Achievement
(34)
18.4
12.0
50.9
26.2
7.097
<.0001
t-test
Significance
Cohen’s d statistic
for the
strength offor
relationship
for the of
highrelationship
achievement group
washigh
1.2, for
the average group
the statistic
wasfor
1.3, and for
Cohen’s
d statistic
the strength
for the
achievement
group
was 1.2,
thetheaverage
group
statistic
1.3,a and
theThe
lowresults,
group
the statistic
A effect
statistic
of all three
the low group
statistic was
1.6. the
A statistic
of .8 was
indicates
strongfor
effect.
therefore,
indicate was
a very1.6.
strong
size for
.8
indicates
a
strong
effect.
The
results,
therefore,
indicate
a
very
strong
effect
size
for
all
three
achievement groups.
achievement groups.
Figure 2
Comparison of Pretest and Posttest Percent Correct
HSP MATH Concepts Study
Three Achievement Level Groups
(N=104)
100%
80%
60%
88%
78%
21
67%
.8 indicates a strong effect. The results, therefore, indicate a very strong effect size for all three
achievement groups.
Figure 2
Comparison of Pretest and Posttest Percent Correct
Concepts
Study Study
HSP MATH
Concepts
Three Achievement Level Groups
(N=104)
100%
88%
80%
78%
67%
60%
51%
50%
40%
20%
18%
0%
Pretest Mean Score
Low Achievement
Posttest Mean Score
Average Achievement
High Achievement
Page 10
Special Services Group Analysis
Special Teachers
Serviceswere
Group
Analysis
asked to identify those students receiving special services such as special
Teachers were
asked to identify
those
receiving
special such
servicesservices.
such as special
education
as well asteachers
those not receiving
such services. Of
education
as well
as students
those not
receiving
Of the
104 students
categorized
the 104 students
teachers categorized
98 students,
44 as receiving
services
and receiving
54 as not receiving
such services.
Onlysix
six students were not
98 students,
44 as receiving
special
servicesspecial
and 54
as not
such services.
Only
categorized students
by teachers.were not categorized by teachers.
Table 5 and Figure 3 provide the results for each of these two groups. The students in both
Table 3 and groups
Figure 3 provide
thestatistically
results for eachsignificant
of these two (<.0001)
groups. The improvement
students in both groups
significant
(<.0001)
showed
from showed
preteststatistically
to posttest
indicating
improvementthat
fromsuch
pretest
to posttestwould
indicating
thatoccurred
such a change
have
occurred
by once
chanceout
fewer
once out
of 10,000
a change
have
by would
chance
fewer
than
ofthan
10,000
times
if thetimes if the study
were repeated.
study were repeated.
Figure 3 shows the percent of raw score increase for each of the groups. The special services
Figure 3 shows
the percent
raw with
score increase
for each
of thepercentage
groups. The special
services
groupHowever,
started out both
with somewhat
group
startedofout
somewhat
lower
correct
scores.
groups lower percentage
correct scores.
However, both
groups improved
the same
from pretesting
to posttesting.
improved
approximately
theapproximately
same amount
fromamount
pretesting
to posttesting.
Table 35
Paired t-test Comparison of Pretest/Posttest Scores
HSP MATH
Concepts
Concepts
Study Study
Special Services Groups
(N=98)
Pretest Scores
Posttest Scores
Analysis
Group (N)
Mean Standard
Deviation
Mean Standard
Deviation
t-test
Significance
Special Services (44)
46.2
30.1
66.1
30.5
5.592
<.0001
Non-Special Services
(54)
53.1
22.4
71.8
17.7
7.703
<.0001
Cohen’s d statistic for the strength of relationship for the special services group was .7 and for
the non-special services group the Cohen’s d statistic
22 was .9. A statistic of .8 indicates a large
effect size and a statistic of .5 indicates a medium effect size. The results, therefore, indicate a
Cohen’s d statistic for the strength of relationship for the special services group was .7 and for the non-special services group the Cohen’s d statistic
was .9. A statistic of .8 indicates a large effect size and a statistic of .5 indicates a medium effect size. The results, therefore, indicate a large effect
size for the non-special services students and a medium effect size for the special services group.
Figure 3
Comparison of Pretest and Posttest Percent Correct
Concepts
HSP MATH
Concepts
Study Study
Special Services Students and Non-Special Services Students
(N=98)
80%
72%
66%
70%
60%
50%
40%
53%
46%
30%
20%
10%
0%
Pretest Mean
Posttest Mean
Special Services
Non-Special Services
English Proficiency
Analysis
English Proficiency
Group Group
Analysis
Teachers were asked
to identify
those
students
who werethose
Englishstudents
proficient and
students
who proficient
were not English
Of the 104 students
Teachers
were
asked
to identify
whothose
were
English
andproficient.
those students
who 98
were
not English
proficient.
Of the
104as students
teachers
98 not
students,
54 by
asteachers.
teachers categorized
students,
54 as non-English
proficient
and 44
English proficient.
Onlycategorized
six students were
categorized
non-English proficient and 44 as English proficient. Only six students were not categorized by
teachers.
Table 4 and Figure
4 provide the results for each of these two groups. The students in both groups showed statistically significant (<.0001)
Table
6 and
Figure
4 provide
the results
each
these two
groups.
bothtimes if the study
improvement from
pretest
to posttest
indicating
that such
a changefor
would
haveofoccurred
by chance
fewerThe
than students
once out ofin
10,000
were repeated. groups showed statistically significant (<.0001) improvement from pretest to posttest indicating
that such a change would have occurred by chance fewer than once out of 10,000 times if the
study were repeated.
Figure 4 shows the percent of raw score increase for each of the English proficiency groups. The non-English proficient group started out with
4 shows
percent
of both
rawgroups
scoreimproved
increase
for each of
English
The
somewhat lowerFigure
percentage
correctthe
scores.
However,
approximately
thethe
same
amount proficiency
from pretestinggroups.
to posttesting.
non-English proficient group started out with somewhat lower percentage correct scores.
However, both groups improved approximately the same amount from pretesting to posttesting.
23
Group
(N)(N)
Group
Table
46 6
Table
Paired
t-test
Comparison
of
Scores
Paired t-test Comparison Pretest/Posttest
of Pretest/Posttest
Scores
Concepts
Study
HSP
MATH
Concepts
Study
HSP MATH Concepts Study
Two
English
Proficiency
Groups
Two
English
Proficiency
Groups
(N=98)
(N=98)
Pretest
Scores
Scores
Analysis
Pretest
Scores Posttest
Posttest
Scores
Analysis
Mean
Standard
Mean
Standard
t-test
Significance
Mean Standard Mean Standard t-test
Significance
Deviation
Deviation
Deviation
Deviation
English
Proficient
English
Proficient
(44)(44)
51.7
51.7
27.0
27.0
70.6
70.6
25.7
25.7
6.816
6.816
<.0001
<.0001
Non-English
Non-English
Proficient
(54)(54)
Proficient
47.9
47.9
25.3
25.3
67.7
67.7
22.5
22.5
6.264
6.264
<.0001
<.0001
Cohen’s
d statistic
for for
thethe
strength
of relationship
for for
thethe
English
proficient
group
waswas
.7 and
for for
Cohen’s
d statistic
strength
of relationship
English
proficient
group
.7 and
thethe
non-English
proficient
group
the
Cohen’s
d
statistic
was
.8.
A
statistic
of
.8
indicates
a
large
group
the Cohen’s
statistic
.8.the
A non-English
statistic ofproficient
.8 indicates
a large
Cohen’s d statistic fornon-English
the strength of proficient
relationship for
the English
proficient d
group
was .7was
and for
group the
Cohen’s d
effect
size
and
a
statistic
of
.5
indicates
a
medium
effect
size.
The
results,
therefore,
indicate
a a
effect
size
a statistic
.5 size
indicates
a medium
effect asize.
Theeffect
results,
therefore,
indicate
statistic was .8. A
statistic
of .8and
indicates
a large of
effect
and a statistic
of .5 indicates
medium
size. The
results, therefore,
indicate a large
large
effect
sizesize
for for
thethe
non-English
proficient
students
andand
a medium
effect
sizesize
for for
thethe
English
large
effect
non-English
proficient
students
a medium
effect
English
effect size forproficient
the non-English
proficient students and a medium effect size for the English proficient group.
group.
proficient group.
Figure
4 4
Figure
Comparison
of Pretest
andand
Posttest
Percent
Correct
Comparison
of Pretest
Posttest
Percent
Correct
Concepts
Study
HSP
MATH
Concepts
HSP MATH
Concepts
Study Study
English
Proficient
andand
Non-English
Proficient
Students
English
Proficient
Non-English
Proficient
Students
(N=98)
(N=98)
80%
80%
70%
70%
60%
60%
50%
50%
40%
40%
30%
30%
71%
71%
68%
68%
52%
52%
48%
48%
20%
20%
10%
10%
0%0%
Pretest
Mean
Pretest
Mean
Posttest
Mean
Posttest
Mean
English
Proficient
English
Proficient
Non-English
Proficient
Non-English
Proficient
Page
13 13
Page
24
Socio-Economic Group Analysis
Socio-Economic
Group Analysis
determine
if thehigher
program
favored students
higher over
socio-economic
those
who werestudents, teachers
To determineTo
if the
program favored
socio-economic
those who were students
categorizedover
as lower
socio-economic
categorized
as lower
socio-economic
students,
teachers
were
to categorize
students
as high Teachers
were asked to
categorize students
as high
or low socio-economic
status. The
scores of
theseasked
two groups
of students were
then compared.
or
low
socio-economic
status.
The
scores
of
these
two
groups
of
students
were
then
compared.
identified 57 students as low socio-economic status and 35 students as high socio-economic status. Twelve students were not categorized by
Teachers identified 57 students as low socio-economic status and 35 students as high socioteachers.
economic status. Twelve students were not categorized by teachers.
and Figure
5 provide
results
for each
of these
twogroups
groups.
Thestatistically
studentssignificant
in both (<.0001)
Table 5 and Table
Figure 57provide
the results
for each ofthe
these
two groups.
The students
in both
showed
groups showed statistically significant (<.0001) improvement from pretest to posttest indicating
improvement from pretest to posttest indicating that such a change would have occurred by chance fewer than once out of 10,000 times if the study
that such a change would have occurred by chance fewer than once out of 10,000 times if the
were repeated.
study were repeated.
Figure
5 shows
the
percent
for each
thelowsocio-economic
groups.
lowsomewhat
Figure 5 shows
the percent
of raw
score
increaseofforraw
eachscore
of the increase
socio-economic
groups.ofThe
socio-economic group
startedThe
out with
socio-economic group started out with somewhat lower percentage correct scores. However,
lower percentage correct scores. However, both groups improved approximately the same amount from pretesting to posttesting.
both groups improved approximately the same amount from pretesting to posttesting.
Table 57
Paired t-test Comparison of Pretest/Posttest Scores
HSP MATH
Concepts
Concepts
Study Study
Two Socio-Economic Groups
(N=92)
Pretest Scores
Posttest Scores
Analysis
Group (N)
Mean Standard
Deviation
Mean Standard
Deviation
t-test
Significance
Low Socio-Economic
Status (57)
48.1
26.3
67.9
25.5
8.998
<.0001
High Socio-Economic
Status (35)
53.7
27.6
72.9
23.1
4.417
<.0001
Cohen’s
d statistic
the strength
thewas
low
socio-economic
group was
.8 and
Cohen’s d statistic
for the
strength offor
relationship
for the of
lowrelationship
socio-economicfor
group
.8 and
for the high socio-economic
group
the Cohen’s d
for the high socio-economic group the Cohen’s d statistic was .8. A statistic of .8 indicates a
statistic was .8. A statistic of .8 indicates a large effect size. The results, therefore, indicate a large effect size for both the low and high
large effect size. The results, therefore, indicate a large effect size for both the low and high
socio-economic groups.
socio-economic groups.
Page 14
25
80%
70%
60%
80%
50%
70%
40%
60%
30%
50%
20%
40%
10%
30%
0%
20%
Figure 5
Comparison of Pretest and Posttest Percent Correct
Concepts
Study Study
HSP MATH
Concepts
Low and High Socio-Economic Students
Figure 5
(N=92)
Comparison of Pretest and Posttest Percent Correct
HSP MATH Concepts Study
73%
Low and High Socio-Economic Students
68%
(N=92)
54%
73%
68%
48%
54%
10%
48%
Pretest Mean
Posttest Mean
Low SES
0%
High SES
Pretest Mean
Minority and Non-Minority
Group Analysis
Posttest Mean
To determine if the program favored Low
non-minority
students
over those who were categorized as
SES
High SES
minority,
teachers wereGroup
asked toAnalysis
categorize students as minority or non-minority. The scores of
Minority and
Non-Minority
these
two
groups
of
students
were
thenthose
compared.
60 students
as minority
andfavored
Non-Minority
Analysis
To determineMinority
if the program
non-minority Group
students over
who were Teachers
categorized identified
as minority, teachers
were asked
to categorize students
and
36
students
as
non-minority.
Only
eight
students
were
not
categorized
by
teachers.
as minority orTo
non-minority.
Theifscores
of these two
groups ofnon-minority
students were then
compared.
Teachers
60 students
as minority
determine
the program
favored
students
over
thoseidentified
who were
categorized
as and 36
Table
8
and
Figure
6
provide
the
results
for
each
of
these
two
groups.
The
students
in
both
students as non-minority.
Only
eight
students
were
not
categorized
by
teachers.
minority, teachers were asked to categorize students as minority or non-minority. The scores of
groups
statistically
significant
(<.0001) improvement
from pretest
to posttest
indicating
these
twoshowed
groups of
students were
then compared.
Teachers identified
60 students
as minority
that
such
a
change
would
have
occurred
by
chance
fewer
than
once
out
of
10,000
times
the
and 36
students
as non-minority.
eight The
students
not
categorized
by teachers.
Table 6 and Figure
6 provide
the results
for each of theseOnly
two groups.
studentswere
in both
groups
showed statistically
significantif(<.0001)
study
were
repeated.
improvementTable
from pretest
posttest 6
indicating
that
such
a change
occurred
by chance
fewer
than
once outin
of both
10,000 times if the study
8 andtoFigure
provide
the
results
forwould
eachhave
of these
two
groups.
The
students
Figure
6
shows
the
percent
of
raw
score
increase
for
each
of
the
two
groups.
The
minority
and
were repeated.
groups showed statistically significant (<.0001) improvement from pretest to posttest indicating
non-minority
groups
had
almost
exactly
the
same
pretest
and
posttest
average
scores.
that such a change would have occurred by chance fewer than once out of 10,000 times if the
study
were repeated.
Figure 6 shows
the percent
of raw score increase for each of the two groups.
The minority
and non-minority groups had almost exactly the same
Table
8
Paired
t-test
Comparison
Pretest/Posttest
ScoresThe minority and
pretest and posttest
scores.
Figureaverage
6 shows
the percent
of raw
score
increase forofeach
of the two groups.
HSP
MATH
Concepts
Study
non-minority groups had almost exactly the same pretest and posttest average scores.
Two Socio-Economic Groups
Table
68
(N=96)
Paired t-test Comparison
of
Scores
Pretest Scores Pretest/Posttest
Posttest Scores
Analysis
HSP MATH
Concepts
Concepts
StudyStudy
Group (N)
MeanSocio-Economic
Standard
Mean
Standard t-test
Significance
Two
Groups
Minority
and
Non-Minority
Groups
Deviation
Deviation
(N=96)
Pretest
Posttest
Analysis<.0001
50.2 Scores
29.3
69.0 Scores
27.6
6.316
Minority Group (60)
Group
(N)
48.3 Standard
20.5
Non-Minority
Group (36) Mean
Deviation
Minority Group (60)
50.2
Non-Minority Group (36)
48.3
Mean
68.4 Standard
17.6
Deviation
29.3Page 15
69.0
20.5
68.4
Page 15
26
t-test
7.326 Significance
<.0001
27.6
6.316
<.0001
17.6
7.326
<.0001
Cohen’s d statistic for the strength of relationship for the minority group was .7 and for the non-
Cohen’s d statistic
for the strength
relationship
minoritywas
group1.1.
wasA.7 statistic
and for theofnon-minority
group
the Cohen’s
d statistic
wasa1.1. A
minority
group ofthe
Cohen’sfordthe
statistic
.8 indicates
a large
effect
size and
statistic
of .5
indicates
medium
size.
The results,
therefore,
largeaeffect
sizesize
forfor the
statistic of .8 indicates
a large
effect
size and aa statistic
of .5effect
indicates
a medium
effect size.
The results,indicate
therefore, aindicate
large effect
theand
non-minority
group
and
medium
effect size for the minority group.
non-minority group
a medium effect
size for
theaminority
group.
Figure 6
Comparison of Pretest and Posttest Percent Correct
Concepts
Study Study
Concepts
HSP MATH
Minority and Non-Minority
(N=96)
80%
69%
70%
60%
50%
40%
68%
48%
50%
30%
20%
10%
0%
Pretest Mean
Posttest Mean
Minority
Non-Minority
Page 16
27
ResultsResults
of theofAnalysis:
Vocabulary
the Analysis:
VocabularyStudy
Study
Total Group
Analysis
Total Group Analysis
of the
Analysis:
Vocabulary
Table 7 and Results
Figure 7 provide
the results
for all of the
Grade 3 students Study
included in the study. The results show statistically significant (<.0001)
Table 9 and Figure 7 provide the results for all of the Grade 3 students included in the study. The
improvementTotal
from pretest
to posttest indicating that such a change would have occurred by chance fewer than once out of 10,000 times if the study
Group
results
showAnalysis
statistically significant (<.0001) improvement from pretest to posttest indicating
were repeated. Figure 7 shows the percent increase.
that such
change7would
have
once out
of 10,000
times
if the
Table
9 anda Figure
provide
theoccurred
results forbyallchance
of thefewer
Gradethan
3 students
included
in the
study.
The
study
were
repeated.
Figure
7
shows
the
percent
increase.
results show statistically significant (<.0001) improvement from pretest to posttest indicating
that such a change would have occurred by chance
Tablefewer
79 than once out of 10,000 times if the
study were repeated. Figure
7 shows
the percent increase.
Paired
t-test Comparison
of Pretest/Posttest Scores
HSP MATH
Vocabulary
Vocabulary
Table 9 StudyStudy
Total
Group
(N=101)
Paired t-test Comparison
of Pretest/Posttest
Scores
HSP MATH Vocabulary Study
Pretest Scores
Posttest(N=101)
Scores
Analysis
Total Group
t-test
Mean
Standard
Mean
Standard
Significance
Pretest ScoresDeviation
Posttest Scores
Analysis
Deviation
t-test
Mean
Standard
Mean
Standard
Significance
61.1
24.3
76.0
22.0
8.375
<.0001
Deviation
Deviation
61.1
24.3
76.0
22.0
8.375
<.0001
Cohen’s d statistic
for the dstrength
of relationship
was .8. A of
statistic
of .8 indicates
largeAeffect
size. The
therefore,
indicates
a large effect
Cohen’s
statistic
for the strength
relationship
wasa .8.
statistic
of .8
.8result,
indicates
a large
effect
size for the gain
scores
of all
the Vocabulary
Study. a large effect size for the gain scores of all students in the
size.
The
.8students
result, intherefore,
indicates
Vocabulary
Study.for the strength of relationship was .8. A statistic of .8 indicates a large effect
Cohen’s
d statistic
size. The .8 result, therefore, indicates a large effect size for the gain scores of all students in the
Vocabulary Study.
Figure 7
Comparison of Pretest and Posttest Percent Correct
Vocabulary
Study
HSP MATH
Figure
7
Total Sample
(N=101)
Comparison of Pretest
and Posttest
Percent Correct
Vocabulary
HSP MATH
Vocabulary
Study Study
Total Sample (N=101)
76%
80%
70%
80%
60%
70%
50%
60%
40%
50%
30%
40%
20%
30%
10%
20%
0%
10%
0%
61%
76%
61%
Pretest Mean
Pretest Mean
Posttest Mean
Vocabulary Test
Vocabulary Test
Page 17
Page 17
28
Posttest Mean
were ranked from low to high and then divided into three equal-sized groups. The low
achievement group included 33 students and the average and high achieving groups included 34
students each. After categorizing the students into these three categories, a paired comparison ttest analysis
was
conducted
using the pretest/posttest scores.
Achievement
Level
Group
Analysis
Tablethe10
and Figure
8 provide
the results
forscores
eachof of
achievement
groups.
In order to divide
students
into achievement
groups,
the pretest
thethe
101three
students
were ranked from
low to Table
high and10
then divided into
shows
that
the
students
in
all
three
groups
showed
statistically
significant
(<.0001)
improvement
three equal-sized groups. The low achievement group included 33 students and the average and high achieving groups included 34 students each.
from the
pretest
to into
posttest
indicating
that
such comparison
a change t-test
would
havewas
occurred
chance
fewer than scores.
After categorizing
students
these three
categories,
a paired
analysis
conductedbyusing
the pretest/posttest
once out of 10,000 times if the study were repeated.
8 shows
percent
score
increasegroups.
for theTable
three
achievement
groups.
design,
Table 8 and Figure
Figure 8 provide
thethe
results
for eachofofraw
the three
achievement
8 shows
that the students
in allBy
three
groups showed
because
of
how
the
sorting
for
the
student
groups
in
this
achievement-level
group
analysis
wasfewer than
statistically significant (<.0001) improvement from pretest to posttest indicating that such a change would have occurred by chance
done,
the
low
and
average
groups
started
out
with
lower
percentage
correct
scores.
However,
all
once out of 10,000 times if the study were repeated.
three groups showed significant improvement.
Figure 8 shows the percent of raw score increase for the three achievement groups. By design, because of how the sorting for the student groups
in this achievement-level group analysis was done, the low and average groups started out with lower percentage correct scores. However, all three
groups showed significant improvement.
Table10
8
Table
Paired t-test Comparison of Pretest/Posttest Scores
HSP MATH
Vocabulary
Vocabulary
Study Study
Three Achievement Level Groups
(N=101)
Pretest Scores
Posttest Scores
Analysis
Group (N)
Mean
Standard
Deviation
Mean
Standard
Deviation
High
Achievement
(34)
84.6
3.1
89.9
7.7
4.047
<.0001
Average
Achievement
(34)
67.7
6.8
79.7
13.6
5.383
<.0001
Low
Achievement
(33)
30.0
12.8
58.1
26.8
7.292
<.0001
t-test
Significance
Cohen’s d statistic for the strength of relationship for the high achievement group was .9, for the
average group the statistic was 1.1, and for the low group the statistic was 1.3. A statistic of .8
Cohen’s d statistic for the strength of relationship for the high achievement group was .9, for the average group the statistic was 1.1, and for the
indicates a strong effect. The results, therefore, indicate a very strong effect size for all three
low group the statistic was 1.3. A statistic of .8 indicates a strong effect. The results, therefore, indicate a very strong effect size for all three
achievement groups.
achievement groups.
Page 18
29
100%
100%
80%
80%
60%
60%
40%
40%
20%
20%
0%
0%
Figure 8
Comparison of PretestFigure
and Posttest
Percent Correct
8
Vocabulary
Study
HSP
MATH
Vocabulary
Study
Comparison of Pretest and Posttest Percent Correct
Three
Achievement
Level Groups
HSP MATH
Vocabulary
Study
(N=101)
Three Achievement Level Groups
(N=101)
90%
80%
90%
80%
58%
58%
85%
85%
68%
68%
30%
30%
Pretest Mean Score
Posttest Mean Score
Mean ScoreAverage Achievement
Posttest Mean
Low Pretest
Achievement
HighScore
Achievement
Low Achievement
Average Achievement
High Achievement
SpecialSpecial
Services
Group
Analysis
Services
Group
Analysis
Teachers were asked to identify those students receiving special services such as special education as well as those not receiving such services.
Special
Group
Analysisthose students receiving special services such as special
TeachersServices
were asked
to identify
Of the 101 students, teachers categorized 44 as receiving special services and 57 as not receiving such services.
educationwere
as well
as those
not receiving
such services.
Ofspecial
the 101services
students,
teachers
categorized
Teachers
asked
to identify
those students
receiving
such
as special
44
as
receiving
special
services
and
57
as
not
receiving
such
services.
education as well as those not receiving such services. Of the 101 students, teachers categorized
Table 9 and Figure 9 provide the results for each of these two groups. The students in both groups showed statistically significant (<.0001)
44
as receiving
specialprovide
servicesthe
andresults
57 as for
noteach
receiving
such
services.
Table
11 andtoFigure
of these
two
groups.
Thethan
students
improvement
from pretest
posttest 9
indicating that
such a change
would have
occurred
by chance
fewer
once outin
of both
10,000 times if the
groups
showed
statistically
significant
(<.0001)
improvement
from
pretest
to
posttest
indicating
Table
11
and
Figure
9
provide
the
results
for
each
of
these
two
groups.
The
students
in
both
study were repeated.
that such
a change
would have
occurred(<.0001)
by chance
fewer than from
once pretest
out of 10,000
times
if the
groups
showed
statistically
significant
improvement
to posttest
indicating
study
were
repeated.
that such a change would have occurred by chance fewer than once out of 10,000 times if the
Figure 9 shows the percent of raw score increase for each of the special services groups.
study
repeated.
Figurewere
9 shows
the percent of raw score increase for each of the special services groups.
Figure 9 shows the percent of raw score increase
Tablefor
11each of the special services groups.
Paired t-test Comparison
of11
Table
9Pretest/Posttest Scores
Table
HSP
MATH
Vocabulary
Study Scores
Paired t-test Comparison of Pretest/Posttest
TwoMATH
Socio-Economic
Groups
HSP
Vocabulary
Study
Vocabulary
Study
(N=101)
Two Socio-Economic Groups
Pretest Scores(N=101)
Posttest Scores
Analysis
Pretest
Scores
Posttest
Scores
Group (N)
Mean
Standard
Mean
Standard
t-test Analysis
Significance
Deviation t-test
Group (N)
Mean Deviation
Standard Mean Standard
Significance
22.9
77.6 Deviation
21.6
5.001
<.0001
Special Services (44) 66.1 Deviation
Special Services
(44)
Non-Special
Services
(57)
Non-Special Services
(57)
66.1
57.2
57.2
22.9
24.9
24.9
77.6
74.9
74.9
21.6
22.3
22.3
5.001
6.827
6.827
<.0001
<.0001
<.0001
Page 19
Page
19 was .5 and for the non-special services group the Cohen’s d statistic
Cohen’s d statistic for the strength of relationship for the special services
group
was .7. A statistic of .8 indicates a large effect size and a statistic of .5 indicates a medium effect size. The results, therefore, indicate a medium effect
size for both groups.
30
effect size and a statistic of .5 indicates a medium effect size. The results, therefore, indicate a
medium effect size for both groups.
Figure 9
Comparison of Pretest and Posttest Percent Correct
HSP MATH
Vocabulary
Vocabulary
StudyStudy
Special Services Students and Non-Special Services Students
(N=101)
90%
78%
80%
70%
75%
66%
60%
57%
50%
40%
30%
20%
10%
0%
Pretest Mean
Posttest Mean
Special Services
Non-Special Services
English Proficiency Group Analysis
Teachers were
askedAnalysis
to identify those students who were English proficient and those students
English Proficiency
Group
who were not English proficient. Of the 101 students, teachers categorized 29 students as English
Teachers were asked to identify those students who were English proficient and those students who were not English proficient. Of the 101 students,
proficient and 72 students as non-English proficient.
teachers categorized 29 students as English proficient and 72 students as non-English proficient.
Table 12 and Figure 10 provide the results for each of these two groups. The students in both
groups showed statistically significant (<.0001) improvement from pretest to posttest indicating
Table 10 and Figure 10 provide the results for each of these two groups. The students in both groups showed statistically significant (<.0001)
that such a change would have occurred by chance fewer than once out of 10,000 times if the
improvement from pretest to posttest indicating that such a change would have occurred by chance fewer than once out of 10,000 times if the study
study were repeated.
were repeated.
Figure 10 shows the percent of raw score increase for each of the English proficiency groups.
The non-English proficient students started out with somewhat lower percentage correct scores.
Figure 10 shows the percent of raw score increase for each of the English proficiency groups. The non-English proficient students started out with
However, both groups improved approximately the same amount from pretesting to posttesting.
somewhat lower percentage correct scores. However, both groups improved approximately the same amount from pretesting to posttesting.
Table 10
12
Table
Paired t-test Comparison of Pretest/Posttest Scores
HSP MATH
Vocabulary
Vocabulary
Study Study
Two English Proficiency Groups
(N=101)
Pretest Scores
Posttest Scores
Analysis
Group (N)
Mean Standard Page
Mean
20 Standard
Deviation
Deviation
t-test
Significance
English Proficient
(29)
65.8
22.8
80.9
21.0
6.267
<.0001
Non-English
Proficient (72)
59.2
24.9
74.1
22.3
6.430
<.0001
Cohen’s d statistic for the strength of relationship for the English proficient group and the nonEnglish proficient group was .7. A statistic of .8 indicates a large effect size and a statistic of .5
31 indicate a medium effect size for both the
indicates a medium effect size. The results, therefore,
non-English proficient students and the English proficient group.
Non-English
Proficient (72)
59.2
24.9
74.1
22.3
6.430
<.0001
Cohen’s d statistic
for the strength
of relationship
for the English
proficient group
andthe
theEnglish
non-English
proficient group
.7. the
A statistic
Cohen’s
d statistic
for the strength
of relationship
for
proficient
groupwas
and
non-of .8
indicates a largeEnglish
effect sizeproficient
and a statistic
of .5 was
indicates
effect
The results, atherefore,
indicatesize
a medium
size for
group
.7. aAmedium
statistic
of size.
.8 indicates
large effect
and aeffect
statistic
ofboth
.5 the
indicates
a medium
effectproficient
size. The
results, therefore, indicate a medium effect size for both the
non-English proficient
students
and the English
group.
non-English proficient students and the English proficient group.
Figure 10
Comparison of Pretest and Posttest Percent Correct
Vocabulary
HSP MATH
Vocabulary
StudyStudy
English Proficient and Non-English Proficient Students
(N=101)
90%
81%
80%
70%
60%
50%
74%
66%
59%
40%
30%
20%
10%
0%
Pretest Mean
Posttest Mean
English Proficient
Non-English Proficient
Page 21
Socio-Economic Group Analysis
To determine if the program favored higher socio-economic students over those who were categorized as lower socio-economic students, teachers
were asked to categorize students as high or low socio-economic status. The scores of these two groups of students were then compared. Teachers
identified 75 students as low socio-economic status and 26 students as high socio-economic status.
Table 11 and Figure 11 provide the results for each of these two groups. The students in both groups showed statistically significant (<.0001)
improvement from pretest to posttest indicating that such a change would have occurred by chance fewer than once out of 10,000 times if the study
were repeated.
Figure 11 shows the percent of raw score increase for each of the socio-economic groups. The low socio-economic group started out with somewhat
lower percentage correct scores. However, both groups improved approximately the same amount from pretesting to posttesting.
32
low socio-economic group started out with somewhat lower percentage correct scores. However,
both groups improved approximately the same amount from pretesting to posttesting.
Table 11
13
Paired t-test Comparison of Pretest/Posttest Scores
HSP MATH
Vocabulary
Vocabulary
Study Study
Two Socio-Economic Groups
(N=101)
Pretest Scores
Posttest Scores
Analysis
Group (N)
Mean Standard
Deviation
Mean Standard
Deviation
t-test
Significance
Low Socio-Economic
Status (75)
57.4
24.1
73.3
22.3
7.045
<.0001
High Socio-Economic
Status (26)
71.6
22.3
84.0
19.6
5.068
<.0001
Cohen’s d statistic
for the
strength of
relationship
for theof
low
socio-economic
and for the high socio-economic
the Cohen’s d
Cohen’s
d statistic
for
the strength
relationship
forgroup
the was
low.7socio-economic
group wasgroup
.7 and
statistic wasfor
.6.the
A statistic
.5 indicates a medium
effectthe
size.Cohen’s
The results,d therefore,
effectof
size.5forindicates
both the low
high ofsocio-economic
group
statisticindicate
was .6.a medium
A statistic
a and high
socio-economic
groups.effect size. The results, therefore, indicate a medium effect size for both the low and
medium
high socio-economic groups.
Figure 11
Comparison of Pretest and Posttest Percent Correct
Vocabulary
Study Study
Vocabulary
HSP MATH
Low and High Socio-Economic Students
(N=101)
90%
80%
70%
60%
50%
84%
73%
72%
57%
40%
Page 22
30%
20%
10%
0%
Pretest Mean
Posttest Mean
Low SES
High SES
Minority and Non-Minority Group Analysis
To determine if the program favored non-minority students over those who were categorized as
Minorityminority,
and Non-Minority
Group
Analysis students as minority or non-minority. The scores of
teachers were asked
to categorize
To determine ifthese
the program
favored of
non-minority
over those
were categorized
as minority,
teachers
were asked
categorize students
two groups
studentsstudents
were then
to bewho
compared.
However,
teachers
identified
98tostudents
minorityThe
andscores
onlyofthree
students
non-minority.
such a small
number
of non-minority
as minority oras
non-minority.
these two
groups as
of students
were then With
to be compared.
However,
there were
questions regarding the
students,
it was
not possible
toTherefore,
conductitany
statistical
analysis
for statistical
this comparison.
identification of
minority and
non-minority
students.
was not
possible to
conduct any
analysis for this comparison.
CONCLUSIONS
The results of the various analyses were very 33
positive for both the Concepts Study and the
Vocabulary Study in demonstrating the effectiveness of increasing students’ knowledge and
CONCLUSIONS
The results of the various analyses were very positive for both the Concepts Study and the Vocabulary Study in demonstrating the effectiveness of
increasing students’ knowledge and understanding of math content, as represented by two sample chapters of the program. Table 12 shows that
every statistical analysis was significant at <.0001 and the effect size was medium or large for every comparison.
Table
Table 14
12
Summary of the
Results ofofthe
MATH program
Summary
theHSP
Results
For Two Experimental Tryouts of
Concept
Development
Vocabularyor
Development
Chapters Focusing
on Either
Conceptor
Development
Vocabulary Development
Concepts Study
Group Analyses
Statistically
Significant
Effect Size Using Cohen’s d Statistic
Total
Yes
Large
Achievement
Yes
All Groups: Large
Special Services
Yes
English Proficiency
Yes
Special Services Group: Medium
Non-Special Services Group: Large
English Proficient: Medium
Non-English Proficient: Large
Socio-Economic
Yes
Both Groups: Large
Minority
Yes
Non-Minority: Large
Minority: Medium
Total
Yes
Large
Achievement
Yes
All Groups: Large
Special Services
Yes
Both Groups: Medium
English Proficiency
Yes
Both Groups: Medium
Socio-Economic
Yes
Both Groups: Medium
Minority
N/A
N/A
Vocabulary Study
The results summarized in Table 14 support the conclusion that both the Concepts focus and the
Vocabulary
focus
successful
in increasing
understanding
and achievement
in in increasing
The results summarized
in Table
12 are
support
the conclusion
that both thestudent’s
Concepts focus
and the Vocabulary
focus are successful
mathematics. For both studies:
student’s understanding and achievement in mathematics. For both studies:
The average total scores for the pretest/posttest comparison increased statistically
significantly
for the total
group of
students
in bothsignificantly
studies and
thetotal
strengths
of the in both
• The average total scores
for the pretest/posttest
comparison
increased
statistically
for the
group of students
effect
were
high.
studies and the strengths of the effect were high.
•
•
The average scores also increased statistically significantly pretest to posttest for
•
The average scores also increased statistically significantly pretest to posttest for
groups statistically
based onsignificantly
pretest achievement
scores.
Allgroups
threebased
groups
in both
studies scores.
• The average scoresthree
also increased
pretest to posttest
for three
on pretest
achievement
a large
effect
All three groups inshowed
both studies
showed
a largesize
effectfor
sizethe
for gains.
the gains.
• The average scoresspecial
also increased
statistically
significantly
posttest for special
servicesstudents.
students asThe
well strength
as for non-special
services
students
as wellpretest
as forto non-special
services
of
thestrength
effect of
was
toshown
be medium
for the
special
group
services students. The
theshown
effect was
to be medium
for the
specialservices
services group
for for
bothboth
studiesstudies
and large for the
and
large
forConcepts
the non-special
services
in the
Concepts Study and medium in
non-special services
group
in the
Study and medium
in thegroup
Vocabulary
Study.
the Vocabulary Study.
34
Page 24
• The average scores also increased statistically significantly pretest to posttest for both English proficient and non-English proficient
students. The strength of the effect was shown to be large for the non-English proficient group in the Concepts Study and medium for
the English proficient group and for both groups in the Vocabulary Study.
• The average total scores also increased statistically significantly pretest to posttest for students identified as either low or high in terms
of economic advantage. The strength of the effect was large for both the high SES group and the low SES group in the Concepts Study.
In the Vocabulary Study, the effect size was medium for both SES groups.
• The average scores also increased statistically significantly pretest to posttest for minority and non-minority students in the Concepts Study.
In the Concepts Study the strength of the effect was shown to be large for the non-minority group and medium for the minority group.
This study sought to determine concept development and vocabulary approaches evident
in Houghton Mifflin Harcourt GO Math! are instructionally effective. The results of this
efficacy study provide a very positive response to that question.
35
Evaluation of the effectiveness of the Strategic Intervention
and Intensive Intervention materials on student’s mathematical
skills and strategy use
Experimental Efficacy
Strategic Intervention Study and Intensive Intervention Study
This report describes a control group/experimental group instructional efficacy study that was conducted to determine the impact of
Houghton Mifflin Harcourt GO Math! Strategic Intervention materials on students’ mathematical skills and strategy use.
Background Information
There has never been a greater need to ensure that the math programs that young students are using are optimally supporting them in developing
the mathematical skills and strategies required for success in high school, college, and in the workplace. Because of the importance of determining
the effectiveness of programs designed to support young children with mathematics instruction, Houghton Mifflin Harcourt contracted with the
Educational Research Institute of America (ERIA) to study the effectiveness of the Houghton Mifflin Harcourt GO Math! Strategic Intervention
materials. This report presents the findings from that study.
Design and Procedures of the Study
A quasi-experimental, control group pretest/posttest design was used for this study. Twelve grade 1 teachers from nine schools and ten grade 4
teachers from seven schools participated in the study. At both grades 1 and 4, six teachers participated in the experimental group. At grade 1,
six teachers participated in the control group and at grade 4, four teachers participated in the control group. While the assignment to either the
control group or the experimental group was not truly random, there was no known bias in the sampling and no known pattern to treatment group
assignment—teachers were grouped in the order in which they volunteered to participate, allowing for similar sample sizes in the experimental and
control groups.
At least one third of all students in each participating classroom in both the control group and the experimental group were working below grade
level in math. However, according to the classroom teachers, no students in either group were working more than two years below grade level in
math. The teachers in the experimental group were provided with directions and a schedule for participating in the study. These directions emphasized to teachers that they should teach the grade 1 or grade 4 chapter from their text and support students with Strategic Intervention materials.
According to the questionnaire results, at grade 1, an average of 54% of the students in each classroom were working up to two years below grade
level in math and therefore received instruction that was supplemented by the use of the strategic intervention materials. At grade 4, an average of
56% of the students in each classroom used the strategic intervention materials. No fewer than one third of the students in any classroom used the
strategic intervention materials. No classroom teacher used the strategic intervention materials with all students.
36
Executive Summary
Grade 1
Control Group/Experimental Group Posttest Analyses Researchers at ERIA conducted an Analysis of Variance (ANOVA) to determine if the differences
in posttest scores between the control group and the experimental group at grade 1 were significantly different. The total test included 34 items
(worth one point each) which was an adequate length to conduct an ANOVA. The .05 level of significance was used as the level at which differences
would be considered statistically significant. For these analyses, 82 students were included in the experimental group and 104 students were
included in the control group. In addition to the ANOVAs, effect-size analyses were computed for each of the comparisons. Cohen’s d statistic was
used to determine the effect size. This statistic provides an indication of the strength of the effect of the treatment regardless of the statistical
significance. Cohen’s d statistic is interpreted as follows:
.2 = small effect
.5 = medium effect
.8 = large effect
Table 10 presents the results of the ANOVA performed to determine if the difference in
posttest scores between the control group and the experimental group at grade 1 was
Table 1 presents the
results of thesignificant.
ANOVA performed
to determine
if the difference
posttest
betweenfor
the the
control
group and the
statistically
The average
percent
correct in
score
on scores
the posttest
control
experimental group
at grade
statistically
significant.
The average percent
score on athe
posttest forthat
the control
group
was1 was
74.1%
and for
the experimental
group correct
was 80.6%,
difference
was group was 74.1%
statistically
significant
at
the
.0001
level.
This
level
of
significance
indicates
that
such a indicates that
and for the experimental group was 80.6%, a difference that was statistically significant at the .0001 level. This level of significance
would by
have
occurred
chance
less than
once out
of 10,000
repetitions.
such a difference difference
would have occurred
chance
less thanby
once
out of 10,000
repetitions.
The effect
size was
medium. The
effect size was medium.
Table
Table10
1
ANOVA Results Comparing the Total Test Percent Correct Scores of the Control Group
and the Experimental Group on the Posttest
Grade 1
Number
of
Mean
Effect
Test
Group
Students Score
SD
F Test
Significance
Size
Posttest
Control
104
74.1% 14.5%
8.653
<.0001
.51
Posttest
Experimental
82
80.6% 15.2%
Figure
1 shows
percentage
of the
grade
control group
andscored
experimental
Figure 1 shows the
percentage
of thethe
grade
1 control group
students
and1experimental
groupstudents
students who
below 70%, from 70% to 89%,
group students who scored below 70%, from 70% to 89%, and 90% or higher on the
and 90% or higher on the posttest. Almost ten percent more students in the control group than in the experimental group received posttest scores
posttest. Almost ten percent more students in the control group than in the experimental
of below 70%. Conversely, nearly twenty percent more students in the experimental group than in the control group received posttest scores of 90%
group received posttest scores of below 70%. Conversely, nearly twenty percent more
or higher.
students in the experimental group than in the control group received posttest scores of
90% or higher.
Figure 1
Percentage of Control Group and Experimental Group Students
Scoring at Various Levels on the Posttest
Grade 1
37
group received posttest scores of below 70%. Conversely, nearly twenty percent more
students in the experimental group than in the control group received posttest scores of
90% or higher.
Figure 1
Percentage of Control Group and Experimental Group Students
Scoring at Various Levels on the Posttest
Grade 1
Figure 3 shows the percentage of students in the grade 1 strategic intervention group who
scored below 70%, 70% to 89%, and 90% or higher on the pretest and on the posttest. All
Figure 2 shows the percentage of students in the grade 1 strategic intervention group who scored below 70%, 70% to 89%, and 90% or higher on
students in the grade 1 strategic intervention group scored below 70% on the pretest.
the pretest and on the posttest. All students in the grade 1 strategic intervention group scored below 70% on the pretest. However, on the posttest,
However, on the
posttest,
63%higher.
of theInstitute
students
in the group scored 70% or higher.
of America
63% of the students in the 12
groupEducational
scored 70% orResearch
Figure
Figure 3
2
Percentage of Strategic Intervention Group Students
Scoring at Various Levels on the Pretest and Posttest
Grade 1
38
Figure 4 shows the percentage of students in the non-strategic intervention group who
Figure 3 shows the percentage of students in the non-strategic intervention group who scored below 70%, 70% to 89%, and 90% or higher on the
scored below 70%, 70% to 89%, and 90% or higher on the pretest and on the posttest.
pretest and on the posttest. The percentage of students in the non-strategic intervention group receiving a score of 90% or higher increased from
The percentage of students in the non-strategic intervention group receiving a score of
8% on the pretest to 47% on the posttest.
90% or higher increased from 8% on the pretest to 47% on the posttest.
Figure 34
Figure
Percentage of Non-Strategic Intervention Group Students
Scoring at Various Levels on the Pretest and Posttest
Grade 1
Control Group/Experimental Group Posttest Analyses
Grade 4 Researchers at ERIA conducted an Analysis of Variance (ANOVA) to determine if the differences in posttest scores between the control group and the
experimental group at grade 4 were significantly different. The total test included 32 items (worth one point each) which was an adequate length
to conduct an ANOVA. The .05 level of significance was used as the level at which differences would be considered statistically significant. For these
analyses, 101 students were included in the experimental group and 76 students were included in the control group. In addition to the ANOVAs,
effect-size analyses were computed for each of the comparisons. Cohen’s d statistic was used to determine the effect size. This statistic provides an
indication of the strength of the effect of the treatment regardless of the statistical significance.
Cohen’s d statistic is interpreted as follows:
.2 = small effect
.5 = medium effect
.8 = large effect
Table 2 presents the results of the ANOVA performed to determine if the difference in posttest scores between the control group and the experimental group at grade 4 was statistically significant. The average percent correct score on the posttest for the control group was 53.3% and for the
experimental group was 84.3%, a difference that was statistically significant at the .0001 level. This level of significance indicates that such a
difference would have occurred by chance less than once out of 10,000 repetitions. The effect size was large.
17 Educational Research Institute of America
39
difference would have occurred by chance less than once out of 10,000 repetitions. The
effect size was large.
Table
Table 13
2
ANOVA Results Comparing the Total Test Percent Correct Scores of the Control Group
and the Experimental Group on the Posttest
Grade 4
Number
of
Mean
Effect
Students Score
Size
Test
Group
SD
F Test
Significance
Posttest
Control
76
53.3% 13.3%
309.58
<.0001
2.59
Posttest
Experimental
101
84.3% 10.2%
Figure 5 shows the percentage of the grade 4 control group students and experimental
group students who scored below 70%, from 70% to 89%, and 90% or higher on the
posttest. In the control group, 90% of the students scored below 70% on the posttest and
Figure 4 shows the percentage of the grade 4 control group students and experimental group students who scored below 70%, from 70% to 89%,
none of them scored 90% or higher. In the experimental group, however, only 8% of the
and 90% or higher on the posttest. In the control group, 90% of the students scored below 70% on the posttest and none of them scored 90% or
students scored below 70% on the posttest and more than one-third of the group scored
higher. In the experimental group, however, only 8% of the students scored below 70% on the posttest and more than one-third of the group scored
90% or higher.
90% or higher.
Figure 45
Percentage of Control Group and Experimental Group Students Scoring at Various
Levels on the Posttest
Grade 4
18 Educational Research Institute of America
Total Experimental Group Pretest/Posttest Analyses
A paired comparison t-test was used to compare the pretest and posttest scores of the
grade 4 experimental group. The .05 level of significance was used as the level at which
increases would be considered statistically significant. A total of 101 students were
included in these analyses.
40
Total Experimental Group Pretest/Posttest Analyses
A paired Table
comparison
was usedthe
to compare
and posttest
scores of the grade
4 experimental
group.
.05 level of significance
14t-test
presents
resultstheofpretest
the paired
comparison
t-test
performed
to The
determine
if the was
used as the
level at whichbetween
increases would
be considered
significant.
A total
101 students
were included
in these
analyses.
difference
the pretest
andstatistically
the posttest
total
testofpercent
correct
scores
at grade
4
Table
14 presents
theaverage
results of
the paired
comparison
t-test performed
toon
determine
if the
was significant.
The
percent
correct
score increased
from 60.8%
the pretest
to
difference
between
the
pretest
and
the
posttest
total
test
percent
correct
scores
at
grade
4
Table 3 presents
the
results
of
the
paired
comparison
t-test
performed
to
determine
if
the
difference
between
the
pretest
and
the
posttest
84.3% on the posttest, a difference that was statistically significant at the .0001 level. total test
was
significant.
average
correct
score
on by
the
pretest
toposttest,
percent correct
grade
4The
was significant.
Thepercent
averagethat
percent
correct
scoreincreased
increased
fromfrom
60.8%60.8%
on the pretest
to 84.3%
on the
Thisscores
levelatof
significance
indicates
such
a change
would
have
occurred
chance
84.3%
onstatistically
the posttest,
a atdifference
that
significant
the .0001
level.
a difference
thatthan
was
the .0001
level.
Thiswas
level
significance
that such at
a change
would have
occurred by chance
less
once
outsignificant
of 10,000
repetitions.
Theofstatistically
effect
sizeindicates
was
large.
level
of significance
indicates
that such a change would have occurred by chance
less than This
once out
of 10,000
repetitions. The effect
size was large.
14 size was large.
less than once out of 10,000 repetitions. Table
The effect
Paired Comparison t-test Results Comparing the Experimental Group’s Pretest and
Table 14
3
Table
Posttest Total Test
Percent Correct Scores
Paired Comparison t-test Results Comparing
Grade 4the Experimental Group’s Pretest and
Posttest Total Test Percent Correct Scores
Number
Grade 4
of
Mean
Effect
Number
Test
Group
Students
Score
SD
t-test Significance
Size
of
Mean
Effect
Pretest Total Experimental
101
60.8% 9.5%
Test
Group
Students
Score
SD
t-test Significance
Size
38.389
<.0001
2.42
Posttest
Total
Experimental
101
84.3%
10.2%
Pretest Total Experimental
101
60.8% 9.5%
38.389
<.0001
2.42
Posttest Total Experimental
101
84.3% 10.2%
Figure 6 shows the percentage of the grade 4 experimental group students who scored
below 70%, from 70% to 89%, and 90% or higher on the pretest and on the posttest. The
Figure provide
6 shows athe
percentage
thedecline
grade 4from
experimental
students
who70%
scored
results
clear
picture ofofthe
pretest togroup
posttest
of below
correct
below 70%,
from
70%
to 89%,
andpretest
90% ortohigher
oninthe
pretest
and onofthe
posttest.
The
scores
and
the
large
increase
from
posttest
the
percentage
students
scoring
Figure 5 shows the percentage of the grade 4 experimental group students who scored below 70%, from 70% to 89%, and 90% or higher on the
results
provide
a clear picture of the decline from pretest to posttest of below 70% correct
90%
orposttest.
higher.
pretest and
on the
The results provide a clear picture of the decline from pretest to posttest of below 70% correct scores and the large
scores and the large increase from pretest to posttest in the percentage of students scoring
increase from pretest to posttest in the percentage of students scoring 90% or higher.
90% or higher.
Figure 65
Figure
Percentage of Experimental Group Students
Figure
6 Pretest and Posttest
Scoring at Various Levels
on the
Percentage of Experimental
Grade 4 Group Students
Scoring at Various Levels on the Pretest and Posttest
Grade 4
41
Figure 7 shows the percentage of grade 4 students in the strategic intervention group who
scored below 70%, 70% to 89%, and 90% or higher on the pretest and on the posttest. All
students in the grade 4 strategic intervention group scored below 70% on the pretest.
Figure 6 shows the percentage of grade 4 students in the strategic intervention group who scored below 70%, 70% to 89%, and 90% or higher on
However, on the posttest, 86% of the students in the group scored 70% or higher.
the pretest and on the posttest. All students in the grade 4 strategic intervention group scored below 70% on the pretest. However, on the posttest,
86% of the students in the group scored 70% or higher.
Figure 67
Percentage of Strategic Intervention Group Students
Scoring at Various Levels on the Pretest and Posttest
Grade 4
Figure 8 shows the percentage of grade 4 students in the non-strategic intervention group
scored
belowof 70%,
and 90%
or higher
pretest
and 70%
on the
Figurewho
7 shows
the percentage
grade 4 70%
studentstoin89%,
the non-strategic
intervention
groupon
whothe
scored
below 70%,
to 89%, and 90% or
posttest.
The
percentage
of
students
in
the
non-strategic
intervention
group
receiving
a or higher
higher on the pretest and on the posttest. The percentage of students in the non-strategic intervention group receiving a score of 90%
score
90%
higher
increased
from 34% on the pretest to 73% on the posttest.
increased
fromof
34%
on theorpretest
to 73%
on the posttest.
Figure 78
Figure
Percentage of Non-Strategic Intervention Group Students
Scoring at Various Levels on the Pretest and Posttest
Grade 4
22 Educational Research Institute of America
42
CONCLUSIONS
This study sought to determine the effect of the Houghton Mifflin Harcourt GO Math! Strategic Intervention materials on students’ math skills
and strategy use.
Table 20 summarizes the experimental group results showing the decreases from pretest
posttestthe
inexperimental
the percentages
ofshowing
students
scoringfrom
at pretest
lowertolevels
increases
fromscoring at lower
Table to
4 summarizes
group results
the decreases
posttest and
in the the
percentages
of students
to posttest
in the
percentages
of students
scoring
athigher
higher
levels.
levelspretest
and the increases
from pretest
to posttest
in the percentages
of students
scoring at
levels.
Table
Table 20
4
Summary of Changes from Pretest to Posttest of Percentage of
Grade 1 and Grade 4 Experimental Group Students Scoring at Various Levels
Student Sample
and Grade Level
Below 70%
70% to 89%
90% or Higher
Entire Sample
Grade 1
-34%
+9%
+25%
Grade 4
-77%
+39%
+38%
Strategic Intervention Group
Grade 1
-64%
+52%
+11%
Grade 4
-86%
+75%
+11%
Non-Strategic Intervention Group
Grade 1
--39%
+39%
Grade 4
--39%
+39%
The conclusion based on the data from reliable and valid assessments of the content
of the instructional materials administered to students at grades 1 and 4 who were
being taught from the HMH Florida Math program is that instruction using the
Theprogram
mathematical
skills and
strategy
use of skills
students
who were
up to two years
significantly
increased
students’
and strategy
useworking
in mathematics.
Furthermore,
skills and
strategy
use of students
who supplemented
were
below
grade levelthe
in mathematical
math and therefore
received
instruction
that was
by
working up to two years below grade level in math and therefore received
Houghton Mifflin Harcourt GO Math! Strategic Intervention increased in a way that
instruction that was supplemented by Houghton Mifflin Harcourt Florida Strategic
wasIntervention
statisticallyincreased
significant.
in a way that was statistically significant, as did the
knowledge and skills of those students who were taught from the HMH Florida
Math program only. Finally, students who were instructed using HMH Florida Math
program (including those whose instruction was supplemented by Houghton Mifflin
Harcourt Florida Strategic Intervention) increased their skills and strategy use in
mathematics significantly more than students in a control group.
43
Experimental Efficacy Intensive Intervention
This report describes a control group/experimental group instructional efficacy study that
was conducted to determine the impact of the Houghton Mifflin Harcourt GO Math!
Intensive Intervention Kit on students’ mathematical skills and strategy use.
Background
Response to Intervention (RTI) is a federally prescribed alternative to the widely discredited aptitude-achievement discrepancy approach to the
identification of students with learning disabilities. In 2004, when Congress reauthorized Individuals with Disabilities Education Act (IDEA), they
changed the law about identifying children with specific learning disabilities. IDEA now allows educators to use the RTI framework to identify
students with specific learning disabilities. RTI shifts the emphasis of the identification process toward providing support and intervention to
struggling students early on. RTI typically includes three “tiers” of instruction, with more intensive help provided if a child does not respond
at each tier. Most of the instruction is provided in general education, not in special education. Special education tends to be very expensive.
Eliminating or significantly reducing special education would release resources that could be redistributed in general education, serving many
more children (Hale, 2008).
RTI is a process that emphasizes how well students respond to changes in instruction. The essential elements of an RTI approach are: the provision
of scientific, research-based instruction and interventions in general education; monitoring and measurement of student progress in response to the
instruction and interventions; and use of these measures of student progress to shape instruction and make educational decisions (Klotz & Canter,
2006). RTI requires that instructional interventions be scientifically valid and systematically evaluated. Unfortunately, a major issue in today’s schools
is the widespread implementation of highly questionable, non-evidence-based instruction under the title RTI.
Because of the importance of determining the effectiveness of math programs in general and of programs designed to be implemented into the RTI
framework in particular, Houghton Mifflin Harcourt contracted with the Educational Research Institute of America (ERIA) to study the effectiveness of
the Houghton Mifflin Harcourt GO Math! Intensive Intervention Kit, one such RTI program. This report presents the findings from that study.
Research Questions The following research questions guided the design of the study:
• Is the Houghton Mifflin Harcourt GO Math! Intensive Intervention Kit instructionally effective in improving students’ mathematical
skills and strategy use?
• Do students whose math instruction is supported by the Houghton Mifflin Harcourt GO Math! Intensive Intervention Kit show improvements in their mathematical skills and strategy use that are above those shown by students in a control group?
Design and Procedures of the Study
A quasi-experimental, control group pretest/posttest design was used for this study. Eleven grade 2 teachers from seven schools and nine grade 5
teachers from nine schools participated in the study. At grade 2, six teachers participated in the experimental group and five teachers participated
in the control group. At grade 5, four teachers participated in the experimental group and five teachers participated in the control group. While the
assignment to either the control group or the experimental group was not truly random, there was no known bias in the sampling and no known
pattern to treatment group assignment—teachers were grouped in the order in which they volunteered to participate, allowing for similar sample
sizes in the experimental and control groups. The ten teachers participating in the experimental group used the Houghton Mifflin Harcourt
GO Math! Intensive Intervention Kit along with their primary math program. The teachers participating in the control group continued to use the
math programs that they had been using prior to their involvement in the study and did not use the Houghton Mifflin Harcourt GO Math!
Intensive Intervention Kit. The teachers in the experimental group were provided with directions and a schedule for participating in the study.
44
differences in posttest scores between the control group and the experimental group at
grade 2 were significantly different. The total test included 30 items (worth one point
each) which was an adequate length to conduct an ANOVA. The .05 level of significance
Executive
Summary
was used
as the level at which differences would be considered statistically significant.
GradeFor
2 these analyses, 68 students were included in the experimental group and 93 students
were
includedGroup
in the
control
group.
Control Group/Experimental
Posttest
Analyses
Researchers at ERIA conducted an Analysis of Variance (ANOVA) to determine if the differences
in posttest
betweentothethe
control
group and the
experimental group
at grade
2 werecomputed
significantly different.
The total
test included 30 items
Inscores
addition
ANOVAs,
effect-size
analyses
were
for each
of the
(worth one
point
each)
which
was
an
adequate
length
to
conduct
an
ANOVA.
The
.05
level
of
significance
was
used
as
the
level at which differences
comparisons. Cohen’s d statistic was used to determine the effect size. This statistic
would beprovides
considered statistically
significant.
these
analyses, 68
were included
the experimental
group and 93
an indication
ofForthe
strength
ofstudents
the effect
of theintreatment
regardless
ofstudents
the were included
in the control
group. In addition
to the ANOVAs,
effect-size
analyses were
for each
the comparisons. Cohen’s d statistic was used to
statistical
significance.
Cohen’s
d statistic
is computed
interpreted
asoffollows:
determine the effect size. This statistic provides an indication of the strength of the effect of the treatment regardless of the statistical significance.
= small
effectas follows:
Cohen’s d.2statistic
is interpreted
.5
=
medium
effect
.2 = small effect
.8 =effect
large effect
.5 = medium
.8 = largeTable
effect 11 presents the results of the ANOVA performed to determine if the difference in
posttest scores between the control group and the experimental group at grade 2 was
Table 1 presents
the resultssignificant.
of the ANOVA performed
to determine
if the difference
postteston
scores
the control
group
and the
statistically
The average
percent
correctinscore
thebetween
posttest
for the
control
experimental
groupwas
at grade
2 was and
statistically
significant.
The average percent
on theaposttest
for the control
group
79.3%
for the
experimental
groupcorrect
wasscore
89.8%,
difference
that group
was was 79.3% and
for the experimental
groupsignificant
was 89.8%, a difference
that waslevel.
statistically
significant
.0001 level. This indicates
level of significance
indicates
statistically
at the .0001
This
level atofthesignificance
that such
a that such
a difference
would
have
occurred
by
chance
less
than
once
out
of
10,000
repetitions.
The
effect
size
was
large.
difference would have occurred by chance less than once out of 10,000 repetitions. The
effect size was large.
Table
Table 11
1
ANOVA Results Comparing the Total Test Percent Correct Scores of the Control Group
and the Experimental Group on the Posttest
Grade 2
Number
of
Mean
Effect
Test
Group
Students Score
SD
F Test
Significance
Size
Posttest
Control
93
79.3% 11.4%
41.438
<.0001
1.06
Posttest
Experimental
68
89.8%
8.3%
11 Educational Research Institute of America
45
Figure 1 shows the percentage of the grade 2 control group students and experimental
Figurestudents
1 showswho
the percentage
of the
grade
control
groupand
students
andhigher
experimental
group
scored below
70%,
from2 70%
to 89%,
90% or
on the
group
who
scored
below
70%,
from
70%experimental
to 89%,
andgroup
90%
or
higher
oncontrol
the
posttest.
Over forty
percent
more
students
in experimental
the
than
in the
Figure 1 shows
thestudents
percentage
of the grade
2 control
group
students
and
group students
who scored
below
70%,
from 70% to 89%,
posttest.
forty
percent
more
students
experimental
thangroup
in the
control
posttest
scores
of
90%
or higher.
and 90%group
or higherreceived
onOver
the posttest.
Over
forty
percent
more
students
inin
thethe
experimental
group thangroup
in the control
received
posttest scores of
group
received
posttest
scores
of
90%
or
higher.
90% or higher.
Figure 1
Figure
1
Percentage of Control Group
and Experimental
Group Students
Percentage of
Control
Group and
Experimental
Group Students
Scoring
at Various
Levels
on the Posttest
Scoring at Various
Levels
on
the
Posttest
Grade 2
Grade 2
Experimental Group Pretest/Posttest Analyses
Experimental Group Pretest/Posttest Analyses
A paired comparison t-test was used to compare the pretest and posttest scores of the
A paired
comparison
t-test
to
compare
the pretestwas
andused
posttest
grade
2 experimental
group.was
Theused
.05
level
of significance
as thescores
level of
at the
which
Experimental
Group
Pretest/Posttest
Analyses
grade
2
experimental
group.
The
.05
level
of
significance
was
used
as
the
level
at
increases
would
be
considered
statistically
significant.
A
total
of
68
students
were
A paired comparison t-test was used to compare the pretest and posttest scores of the grade 2 experimental group. The .05 levelwhich
of significance was
increases
would
be
considered
statistically
significant.
A
total
of
68
students
were
included
in
these
analyses.
used as the level at which increases would be considered statistically significant. A total of 68 students were included in these analyses.
included in these analyses.
Table 12 presents the results of the paired comparison t-test performed to determine if the
Table 2 presents
of the paired
comparison
to
determine
thet-test
difference
between
theto
pretest
and
posttest
Tablethe
12results
presents
the
theperformed
paired
comparison
performed
determine
if the
difference
between
theresults
pretestoft-test
and
the
posttest
total iftest
percent
correct
scores
atthe
grade
2 total test
percent correct
scores
at between
grade 2The
was the
significant.
Thepercent
average
percent
correct
scoreincreased
increased
fromfrom
77.9%
on the scores
pretest
toat
89.8%
on the
difference
pretest
and the
posttest
total
test percent
correct
grade
2toposttest,
was
significant.
average
correct
score
77.9%
on the
pretest
a difference
thatsignificant.
was
significant
the .0001
level.
This
level of
significance
indicates
that
such77.9%
a change
would
occurred
was
The
average
percent
correct
score
increased
from
thehave
pretest
toby chance
89.8%
onstatistically
the posttest,
a atdifference
that
was
statistically
significant
at
the on
.0001
level.
less than This
once out
of
repetitions. The
effect
size was that
large. was
89.8%
on10,000
the
posttest,
a difference
significant
at the .0001
level.
level
of significance
indicates
suchstatistically
a change would
have occurred
by chance
This
level
of
significance
indicates
that
such
a
change
would
have
occurred
by
chance
less than once out of 10,000 repetitions. The effect size was large.
less than once out of 10,000 repetitions. The effect size was large.
Table
Table 12
2
Table 12the Experimental Group’s Pretest and
Paired Comparison t-test Results Comparing
Paired Comparison t-test
Results
Comparing
theCorrect
Experimental
Posttest Total Test Percent
Scores Group’s Pretest and
Posttest Total Test
Percent
Grade
2 Correct Scores
Number Grade 2
Number
of
Mean
Effect
of
Mean
Test
Group
Students
Score
SD
t-test Significance Effect
Size
Students
Score
SD
t-test Significance Size
Test
Group
Pretest
Total Experimental
68
77.9%
13.2%
8.851
<.0001
1.27
Pretest Total
Total Experimental
Experimental
68
77.9%
13.2%
Posttest
68
89.8%
8.3%
8.851
<.0001
1.27
Posttest Total Experimental
68
89.8%
8.3%
12 Educational Research Institute of America
12 Educational Research Institute of America
46
Figure 2 shows the percentage of the grade 2 experimental group students who scored
from of70%
to 89%,
and 90%
higher
thebelow
pretest
on the
posttest.
Figurebelow
2 shows 70%,
the percentage
the grade
2 experimental
groupor
students
whoon
scored
70%,and
from 70%
to 89%,
and 90%The
or higher on the
results
provide
a
clear
picture
of
the
decline
from
pretest
to
posttest
of
below
70%
correct
pretest and on the posttest. The results provide a clear picture of the decline from pretest to posttest of below 70% correct scores and the large
scores
and the
largein increase
from
pretest
to posttest
in the percentage of students scoring
increase
from pretest
to posttest
the percentage
of students
scoring
90% or higher.
90% or higher.
Figure 2
Percentage of Experimental Group Students
Scoring at Various Levels on Pretests and Posttests
Grade 2
Experimental Group Pretest/Posttest Analysis of Pretest Performance Groups The experimental group was divided into two equal groups of 34 students based on their pretest scores. Paired comparison t-tests were conducted
to determine if both groups made significant pretest to posttest gains.
Table 3 presents the results of the paired comparison t-test analysis of the student scores grouped by pretest performance. The average percent
correct score for the lower scoring group increased from 68.2% to 86.3% and the average percent correct score for the higher scoring group
increased from 87.6% to 93.3%. The difference for the lower scoring pretest group was statistically significant at the .001 level, indicating a change
that would have occurred by chance less than once out of 1,000 repetitions. The difference for the higher scoring pretest group was statistically
significant at the .0001 level. This level of significance indicates that such a change would have occurred by chance less than once out of 10,000
repetitions. The effect size was large for both groups.
13 Educational Research Institute of America
47
that would have occurred by chance less than once out of 1,000 repetitions. The
difference for the higher scoring pretest group was statistically significant at the .0001
level. This level of significance indicates that such a change would have occurred by
chance less than once out of 10,000 repetitions. The effect size was large for both groups.
Table
Table 13
3
Paired Comparison t-test Results Comparing the Pretest/Posttest Total Test Percent
Correct Scores for Students Grouped by Pretest Performance
Grade 2 Experimental Group
Number
of
Mean
Test
Students
Score
SD
t-test
Significance Effect Size
Lower Pretest Group
Pretest
34
68.2%
12.2%
8.640
<.001
1.66
Posttest
34
86.3%
9.3%
Higher Pretest Group
34
Pretest
87.6%
3.8%
6.821
<.0001
1.28
34
Posttest
93.3%
5.1%
Figure 3 shows the percentage of lower scoring students who scored below 70%, 70% to
89%, and 90% or higher on the pretest and on the posttest. While none of the students
Figure 3 shows the percentage of lower scoring students who scored below 70%, 70% to 89%, and 90% or higher on the pretest and on the
scored 90% or higher on the pretest, on the posttest, nearly half of the lower scoring
posttest. While none of the students scored 90% or higher on the pretest, on the posttest, nearly half of the lower scoring students scored 90%
students scored 90% or higher.
or higher.
Figure 3
Percentage of Lower Scoring Students
Scoring at Various Levels on the Pretest and Posttest
Grade 2 Experimental Group
14 Educational Research Institute of America
48
shows the
percentage
of higher
scoring
scored
70%,
Figure 4Figure
shows the4percentage
of higher
scoring students
who scored
belowstudents
70%, 70% who
to 89%,
and 90%below
or higher
on the70%
pretesttoand on the postandof90%
or inhigher
onscoring
the pretest
and on
the ofposttest.
Themore
percentage
offrom
students
test. The89%,
percentage
students
the higher
group receiving
a score
90% or higher
than doubled
pretest toinposttest, going
theonhigher
scoring
group
receiving a score of 90% or higher more than doubled from
from 35%
the pretest
to 85% on
the posttest.
pretest to posttest, going from 35% on the pretest to 85% on the posttest.
Figure 4
Percentage of Higher Scoring Students
Scoring at Various Levels on the Pretest and Posttest
Grade 2 Experimental Group
Control Group/Experimental Group Posttest Analyses
Grade 5 Researchers at ERIA conducted an Analysis of Variance (ANOVA) to determine if the differences in posttest scores between the control group and the
experimental group at grade 5 were significantly different. The total test included 30 items (worth one point each) which was an adequate length
to conduct an ANOVA. The .05 level of significance was used as the level at which differences would be considered statistically significant. For these
analyses, 80 students were included in the experimental group and 86 students were included in the control group.
In addition to the ANOVAs, effect-size analyses were computed for each of the comparisons. Cohen’s d statistic was used to determine the effect
size. This statistic provides an indication of the strength of the effect of the treatment regardless of the statistical significance.
Cohen’s d statistic is interpreted as follows:
.2 = small effect
.5 = medium effect
.8 = large effect
Table 4 presents the results of the ANOVA performed to determine if the difference in posttest scores between the control group and the experimental group at grade 5 was significant. The average percent correct score on the posttest for the control group was 66.5% and for the experimental
group was 81.3%, a difference that was statistically significant at the .0001 level. This level of significance indicates that such a difference would
have occurred by chance less than once out of 10,000 repetitions. The effect size was medium.
16 Educational Research Institute of America
49
significant at the .0001 level. This level of significance indicates that such a difference
would have occurred by chance less than once out of 10,000 repetitions. The effect size
was medium.
Table
Table14
4
ANOVA Results Comparing the Total Test Percent Correct Scores of the
Control Group and the Experimental Group on the Posttest
Grade 5
Number
of
Mean
Effect
Students Score
Size
Test
Group
SD
F Test
Significance
Posttest
Control
86
66.5% 25.0%
20.878
<.0001
.72
Posttest
Experimental
80
81.3% 14.9%
Figure 5 shows the percentage of the grade 5 control group students and experimental
group students who scored below 70%, from 70% to 89%, and 90% or higher on the
Figure 5 shows the percentage of the grade 5 control group students and experimental group students who scored below 70%, from 70% to 89%,
posttest. The percentage of students scoring 90% or higher on the posttest was almost
and 90% or higher on the posttest. The percentage of students scoring 90% or higher on the posttest was almost three times higher for the
three times higher for the experimental group than the control group.
experimental group than the control group.
Figure 5
Percentage of Control Group and Experimental Group Students
Scoring at Various Levels on the Posttest
Grade 5
17 Educational Research Institute of America
Experimental Group Pretest/Posttest Analyses
Experimental
Group Pretest/Posttest
A paired comparison
t-test was used Analyses
to compare the pretest and posttest scores of the
A pairedgrade
comparison
t-test was used togroup.
compareThe
the pretest
and posttest
scores of the grade
experimental
Theat
.05which
level of significance was
5 experimental
.05 level
of significance
was5 used
as thegroup.
level
used as the
level at which
increases
would be considered
statisticallysignificant.
significant. A total
80 students
includedwas
in these analyses.
increases
would
be considered
statistically
A oftotal
of 80 was
students
included in these analyses.
Table 5 presents the results of the paired comparison t-test performed to determine if the difference between the pretest and the posttest total test
percent correct scores at grade 5 was significant. The average percent correct score increased from 71.3% on the pretest to 81.3% on the posttest,
a difference that was statistically significant at the .0001 level. This level of significance indicates that such a change would have occurred by chance
less than once out of 10,000 repetitions. The effect size was medium.
50
less than
out of 10,000
repetitions.
The aeffect
sizewould
was medium.
This
levelonce
of significance
indicates
that such
change
have occurred by chance
less than once out of 10,000 repetitions. Table
The effect
15 size was medium.
Paired Comparison t-test Results Comparing
Table
Table15
5the Experimental Group’s Pretest and
Posttest
Total
Test
Percent
Scores Group’s Pretest and
Paired Comparison t-test Results Comparing theCorrect
Experimental
Grade
5 Correct Scores
Posttest Total Test
Percent
Number Grade 5
of
Mean
Effect
Number
Test
Group
Students
Score
SD
t-test Significance Effect
Size
of
Mean
Pretest Total Experimental
80
71.3%
Test
Group
Students
Score 16.9%
SD
t-test Significance
Size
8.814
<.0001
.63
Posttest Total Experimental
81.3% 16.9%
14.9%
Pretest
80
71.3%
8.814
<.0001
.63
Posttest Total Experimental
80
81.3% 14.9%
Figure 6 shows the percentage of the grade 5 experimental group students who scored
below 70%,
from
70%
to 89%,of
and
90%
or 5higher
on the pretest and
on thewho
posttest.
The
Figure
6 shows
percentage
the
grade
experimental
students
scored
Figure 6 shows
the percentage
of the
the grade
5 experimental
group
students
who scored below group
70%, from
70% to 89%,
and
90% or higher on the
results 70%,
provide
a clear
picture
ofand
the90%
decline
from pretest
to posttest
ofonbelow
70% correct
from
70%
to a89%,
or higher
on the
thecorrect
posttest.
pretest and below
on the posttest.
The
results
provide
clear picture
of the decline
from pretest
to pretest
posttest ofand
below
70%
scores The
and the large
scores
and
the
large
increase
from
pretest
to
posttest
in
the
percentage
of
students
scoring
results
provide
a
clear
picture
of
the
decline
from
pretest
to
posttest
of
below
70%
correct
increase from pretest to posttest in the percentage of students scoring 90% or higher.
90% orand
higher.
scores
the large increase from pretest to posttest in the percentage of students scoring
90% or higher.
Figure 6
Percentage of Experimental
Figure 6 Group Students
Scoring
at
Various
Levels
on the Group
PretestStudents
and Posttest
Percentage of Experimental
Grade
5 Pretest and Posttest
Scoring at Various Levels
on the
Grade 5
Experimental Group Pretest/Posttest Analysis of Pretest Performance Groups The experimental group was divided into two equal groups of 40 students based on their pretest scores. Paired comparison t-tests were conducted
19groups
Educational
Research
Institute
to determine if both
made significant
pretest to posttest
gains.of America
19 Educational Research Institute of America
Table 6 presents the results of the paired comparison t-test analysis of the student scores grouped by pretest performance. The average percent
correct score for the lower scoring group increased from 58.3% to 71.1% and the average percent correct score for the higher scoring group
increased from 84.3% to 91.6%. The difference for the lower scoring pretest group was statistically significant at the .001 level, indicating a change
that would have occurred by chance less than once out of 1,000 repetitions. The difference for the higher scoring pretest group was statistically
significant at the .0001 level. This level of significance indicates that such a change would have occurred by chance less than once out of 10,000
repetitions. The effect size was large for both groups.
51
level. This level of significance indicates that such a change would have occurred by
chance less than once out of 10,000 repetitions. The effect size was large for both groups.
Table 16
6
Table
Paired Comparison t-test Results Comparing the Pretest/Posttest Total Test Percent
Correct Scores for Students Grouped by Pretest Performance
Grade 5 Experimental Group
Number
of
Mean
Test
Students Score
SD
t-test
Significance
Effect Size
Lower Pretest Group
Pretest
40
58.3% 13.8%
6.626
<.001
.90
Posttest
40
71.1% 14.1%
Higher Pretest Group
40
Pretest
84.3%
6.7%
6.743
<.0001
1.12
40
Posttest
91.6%
6.2%
Figure 7 shows the percentage of grade 5 students in the lower scoring pretest group who
scored below 70%, 70% to 89%, and 90% or higher on the pretest and on the posttest.
Figure 7 shows the percentage of grade 5 students in the lower scoring pretest group who scored below 70%, 70% to 89%, and 90% or higher on
The percentage of lower scoring students scoring below 70% dropped by well over half
the pretest and on the posttest. The percentage of lower scoring students scoring below 70% dropped by well over half from pretest to posttest.
from pretest to posttest.
Figure 7
Percentage of Students in the Lower Scoring Pretest Group
Scoring at Various Levels on the Pretest and Posttest
Grade 5 Experimental Group
20 Educational Research Institute of America
52
Figure 8 shows the percentage of grade 5 students in the higher scoring pretest group who
Figure 8 scored
shows thebelow
percentage
of grade
5 students
in the
higher
scoring
pretest group
whopretest
scored below
70%posttest.
to 89%, and 90% or higher on
70%,
70%
to 89%,
and
90%
or higher
on the
and70%,
on the
the pretest
andpercentage
on the posttest.of
Thestudents
percentagein
of the
students
in thescoring
lower scoring
group
receiving a a
score
of 90%
higher
The
lower
group
receiving
score
of or
90%
orincreased
higherby more than
20% from
pretest to posttest.
increased
by more than 20% from pretest to posttest.
Figure 8
Percentage of Students in the Higher Scoring Pretest Group
Scoring at Various Levels on the Pretest and Posttest
Grade 5 Experimental Group
22 Educational Research Institute of America
53
This study sought to determine the effect of the Houghton Mifflin Harcourt Florida Math
Intensive
Intervention Kit on students’ math skills and strategy use.
Conclusions CONCLUSIONS
When comparing the experimental group with the control group at grade 2 and grade 5,
This study sought to determine the effect of the Houghton Mifflin Harcourt Florida Math
significant
were
found
between
the Intensive
two groups’
posttest
scores math skills and
This studystatistically
sought to determine
the effect differences
of the Houghton
Mifflin
Harcour
t GO Math!
Intervention
Kit on students’
Intensive
Intervention
Kit
on
students’
math
skills
and
strategy
use.
forWhen
the total
test,thewith
the experimental
groupgroup
receiving
scores
at both
grade differences
levels. were found
strategy use.
comparing
experimental
group with the control
at grade 2higher
and grade
5, statistically
significant
A
of thethe
results
is provided
inthe
Table
18.
table
indicates
whether
comparing
experimental
with
the The
control
group
at grade
2at and
grade
5, A summary of
between When
the summary
two groups’
posttest
scores
for the
total test,group
with
experimental
group
receiving
higher
scores
boththe
grade
levels.
differences
were
as well
asthethe
strength
ofsignificant
each
significant
were
found
between
the significant
two
groups’
posttest
scores
the resultsstatistically
is provided in Table
7. significant
The tabledifferences
indicates
whether
differences
were
as well
as thedifference.
strength
of each
significant difference.
for the total test, with the experimental group receiving higher scores at both grade levels.
A summary of the results is provided in Table 18. The table indicates whether the
Table
18
Table
7 of each significant difference.
differences were significant as well as the
strength
Summary of the Control Group/Experimental Group Posttest Score Analyses at
Grade 2 and Grade 5
Difference
Statistically
Table
18
Grade
Significant?
EffectAnalyses
Size
Summary of the Control Group/Experimental Group Posttest Score
at
Grade 2
Yes
Large
Grade 2 and Grade 5
Grade 5
Yes
Medium
Difference
Statistically
Grade
Size
When comparing the pretest to posttest Significant?
gains made by the experimentalEffect
group,
the group
Grade 2
Yes
Large
made statistically significant gains at both grade 2 and grade 5. In addition, significant
Grade
5 made by students who scored lower
Yes on the pretests as well as
Medium
gains were
those who
scored
higher
on
the
pretest.
A
summary
of
the
results
is
provided
in
Table
19 below.
The
When comparing the pretest to posttest gains made by the experimental group,
the group
When comparing the pretest to posttest gains made by the experimental group, the group made statistically significant gains at both grade 2 and
table
whether
the gains
were
significant
well
as the
strength
of each
made indicates
statistically
significant
gains
at both
grade 2as
and
grade
5. In
addition,
significant
grade 5. In addition, significant gains were made by students who scored lower on the pretests as well as those who scored higher on the pretest.
significant
gains were gain.
made by students who scored lower on the pretests as well as those who
A summary of the results is provided in Table 8 below. The table indicates whether the gains were significant as well as the strength of each
scored higher on the pretest. A summary of the results is provided in Table 19 below. The
significant gain.
table indicates whether the gains were significant
Table 19 as well as the strength of each
significant
gain. of the Experimental Group Pretest/Posttest Score Analyses at
Summary
Grade 2 and Grade 5
Table
Grade
Grade 5
Table219
8
Gain Statistically
Gain Score
Statistically
Summary of the Experimental
Group Pretest/Posttest
Analyses at
Group
Significant?
Effect
Size
Significant?
Effect Size
Grade 2 and Grade 5
Total Group
Yes Grade 2 Large
Yes Grade 5Medium
Lower Scorers on the
Gain Statistically
Gain Statistically
Yes
Large
Yes
Large
Pretest
Group
Significant?
Effect Size
Significant?
Effect Size
HigherGroup
Scorers on the
Total
Yes
Large
Yes
Medium
Yes
Large
Yes
Large
Pretest Scorers on the
Lower
Yes
Large
Yes
Large
Pretest
Higher Scorers on the
Yes
Large
Yes
Large
Pretest
24 Educational Research Institute of America
24 Educational Research Institute of America
54
Table 20 summarizes the experimental group results showing the decreases from pretest
Table 9 summarizes
the in
experimental
group resultsofshowing
the decreases
pretest tolevels
posttestand
in thethe
percentages
of students
to posttest
the percentages
students
scoringfrom
at lower
increases
from scoring at lower
levels and
the increases
from pretest
in the percentages
of students
scoring at
at higher
levels.
pretest
to posttest
in to
theposttest
percentages
of students
scoring
higher
levels.
Table
Figure209
Summary of Changes from Pretest to Posttest of Percentage of
Grade 2 and Grade 5 Experimental Group Students Scoring at Various Levels
Student Sample
and Grade Level
Below 70%
70% to 89%
90% or Higher
Entire Sample
Grade 2
-11%
-37%
+48%
Grade 5
-14%
-9%
+23%
Lower Scoring Pretest Group
Grade 2
-23%
-24%
+47%
Grade 5
-27%
+5%
+22%
Higher Scoring Pretest Group
Grade 2
--50%
+50%
Grade 5
--23%
+23%
The conclusion based on the data from reliable and valid assessments of the content
of the instructional materials administered to students who were being taught from
The conclusion
based
on the
data from
reliable
andIntensive
valuableIntervention
assessmentsKit
indicate
grade
the Houghton
Mifflin
Harcourt
Florida
Math
at grade
2 2 and
at gradewhose
5 is that
instruction
the program
increased
the
gradeand
5 students
instruction
wasusing
supported
by thesignificantly
Houghton Mifflin
Harcourt
GO Math!
mathematics skills and strategy use of these struggling students. Furthermore, grade
Intensive
Intervention
Kit increased
their skills and
usebyinthe
mathematics
significantly more
2 and
grade 5 students
whose instruction
wasstrategy
supported
Houghton Mifflin
Harcourt
Florida
Intensive
Kit increased their skills and strategy
than grade
2 and
gradeMath
5 students
in aIntervention
control group.
use in mathematics significantly more than grade 2 and grade 5 students in a control
group.
25 Educational Research Institute of America
55
Evaluation of Instructional Effectiveness of GO Math!
To assess the instructional effectiveness of the new Houghton Mifflin Harcourt GO Math! program, researchers from Educational Research
Institute of America (ERIA) assessed student math achievement in grades 3, 4, and 5 in ten different schools over the course of one semester.
A quasi-experimental, pretest/posttest design was employed to compare math performance among students using GO Math! (Experimental
assess
theanother
instructional
ofGroup).
the new Houghton Mifflin Harcourt GO Math!
Group) toTo
students
using
mathematicseffectiveness
program (Control
program, researchers from Educational Research Institute of America (ERIA) assessed student
math achievement in grades 3, 4, and 5 in ten different schools over the course of one semester.
The assessments for this study were developed by researchers at ERIA because the study only covered one semester of instruction; the reliability
A quasi-experimental, pretest/posttest design was employed to compare math performance
and validity evidence for the tests are reported in the study and indicate that the posttest had sound psychometric properties.
among students using GO Math! (Experimental Group) to students using another mathematics
program (Control Group).
All participating teachers either volunteered to participate in the study or were asked to participate by school administrators. An examination of
The assessments
this
wereschools
developed
researchers
at ERIA
study only
the demographic
characteristics for
of the
ten study
participating
indicatesbythey
are similar in terms
of thebecause
percentagethe
of students
enrolled in free/
covered
one
semester
of
instruction;
the
reliability
and
validity
evidence
for
the
tests
are
reduced lunch programs and other characteristics.
reported in the study and indicate that the posttest had sound psychometric properties.
All
participating
either
volunteered
to participate
inmade
the study
or gains
wereover
asked
to of the semester.
The results
showed
that both theteachers
Control group
classes
and the GO Math!
group classes
significant
the course
byforschool
administrators.
the demographic
characteristics
The effectparticipate
sizes were large
the Mathematics
total results.An
Theexamination
results also showofconsistent
evidence at grades
3, 4, and 5 thatof
thethe
GO Math!
ten
participating
schools
indicates
they
are
similar
in
terms
of
the
percentage
of
students
students made greater gains over the course of the semester than did the Control group students. Finally, the analysis clearly showed that the
enrolled in free/reduced lunch programs and other characteristics.
GO Math! program was equally effective with both higher and lower pretest scoring students.
The results showed that both the Control group classes and the Go Math! group classes made
significant gains over the course of the semester. The effect sizes were large for the Mathematics
total results. The results also show consistent evidence at both grades 3, 4, and 5 that the Go
Math! students made greater gains over the course of the semester than did the Control group
students. Finally, the analysis clearly showed that the Go Math! program was equally effective
with both higher and lower pretest scoring students.
Standard Score
Figure 1
Grade 3 Go Math! Group Students and Control Group Students Math Performance
340
330
320
310
300
290
280
270
260
250
327
303
278
277
Control Group
1 Educational Research Institute of America
56
Go Math!
Figure 2
Grade 4 Go Math! Group Students and Control Group Students Math Performance
340
329
Figure 2
Grade 4 Go Math! Group Students and Control Group Students Math Performance
Standard Score
Standard Score
320
310
340
300
330
290
320
280
310
270
300
260
290
250
280
270
301
329
277
301
264
277
264
Control Group
Go Math!
260
250
Control Figure
Group 3 Go Math!
Grade 5 Go Math! Group Students and Control Group Students Math Performance
340
Figure 3
325
330
Grade 5 Go Math! Group Students and Control Group Students Math Performance
320
307
310
340
300
325
330
290
281
320
307
280
271
310
270
300
260
290
281
250
280
271
Control Group
Go Math!
270
260
250
Standard Score
Standard Score
330
Control Group
57
Go Math!
Go Math! Studies
Whitley School District (KY)
District Demographics:
• Whitley School District is located in Kentucky and is comprised of 10 schools with a total
enrollment of 4,753 students.
• Student Ethnicities:
Caucasian 99%
• 70% Students eligible Free/Reduced Lunch
• 19% Students eligible Special Education Services
Measure: Kentucky Core Content Test (KCCT)
Period of Evaluation: 2010 (Baseline) to 2011
Summary: During the first year of using GO Math!, students at this high performing
school district that are deemed Proficient or Distinguished increased an average of 3%
across the three grade levels on the state’s KCCT.
Whitely School
District (KY)
Whitley School District
Whitley3,School
Grades
4, & 5District
KCCT
Grades 3, 4, &5 KCCT
Percent
At/Above Proficient
Percent At / Above Proficient
2010–2011
2010-2011
hics:
istrict is located Kentucky and is
schools with a total enrollment of 4,753
es:
9%
ible Free/Reduced Lunch
100%
90%
89%
91%
89%
91%
ible Special Education Services
Core Content Test (KCCT
n: 2010 (Baseline) to 2011.
he first year of using Go Math!, this
g school district witnessed an increase
udents Proficient or Distinguished on
T increased an average of 3% across
evels
80%
75%
+ 2 pts.
70%
70%
+ 2 pts.
+5 pts.
60%
Grade 3
Grade 4
2010 Percent
Grade 5
2011 Percent
Source of assessment data: http://www.education.ky.gov/KDE/
Administrative+Resources/Testing+and+Reporting+/Reports/Research+Data/
data: httphttp://http://www.education.ky.gov/KDE/Administrative+Resources/Testing+and+Reporting+/Reports/Research+Data/
Source of demographic data: MDR
ic data: MDR
58
Kimper Public School
Pikeville, KY
• District Demographics:
• This rural school serves 180 students in grades K–8 with a professional staff that
includes 14 teachers, as well as 14 support personnel and administrators.
• Student Ethnicities:
Caucasian 100%
• 75% Students eligible Free/Reduced Lunch
• 13% Students eligible Special Education Services
Measure: Kentucky Core Content Test (KCCT)
Period of Evaluation: 2010 (Baseline) to 2011
Summary: In only one year of using GO Math!, the percentage of students deemed
Proficient or Distinguished on the state’s KCCT increased an average of 9%.
Kimper Public School
Kimper Elementary
Pikeville, KY Grades
Kimper
3, Elementary
4, & 5 KCCT
ics:
erves 180 students in grades K-8 with
f that includes 14 teachers, with 14
and administrators.
es:
00%
ible Free/Reduced Lunch
Grades 3, 4, &5 KCCT
Percent
At/Above Proficient
Percent At / Above Proficient
2010–2011
2010-2011
88%
90%
81%
80%
71%
70%
ible Special Education Services
Core Content Test (KCCT)
n: 2010 (Baseline) to 2011.
ne year of using Go Math!, the
udents Proficient or Distinguished on
T increased an average of 9% .
60%
60%
64% 63%
+ 10 pts.
+ 18 pts.
50%
40%
Grade 3
Grade 4
2010 Percent
Grade 5
2011 Percent
Source of assessment data: http://www.education.ky.gov/KDE/
Administrative+Resources/Testing+and+Reporting+/Reports/Research+Data/
data: httphttp://http://www.education.ky.gov/KDE/Administrative+Resources/Testing+and+Reporting+/Reports/Research+Data/
Source of demographic data: MDR
c data: MDR
59
Arthur T. Cummings Elementary
Winthrop, MA
District Demographics:
• This suburban school serves 464 students in grade 3–5 with a professional staff that
includes 20 teachers, with 40 support personnel and administrators.
• Student Ethnicities:
Caucasian 89%
Hispanic 7%
Other 4%
• 32% Students eligible Free/Reduced Lunch
• 18% Students eligible Special Education Services
Measure: Massachusetts Comprehensive Assessments (MCAS)
Period of Evaluation: 2010 (Baseline) to 2011
Summary: In only one year of using GO Math!, the percentage of students deemed
rthur T. Cummings
Elementary
all three grade levels.
Winthrop, MA
Proficient or Advanced on the state’s MCAS increased an average of over 6% across
hics:
ool serves 464 students in grade 3-5
al staff that includes 20 teachers, with
nel and administrators.
es:
9%
ible Free/Reduced Lunch
Arthur
Cummings
Arthur
Cummings Elementary
Elementary
Grades
MCAS
Grades3,3,4,
4, &
&55MCAS
Percent
At
/
Above
Proficient
Percent At/Above Proficient
2010-2011
2010–2011
70%
60%
n: 2010 (Baseline) to 2011.
ne year of using Go Math!, the
udents Proficient or Advanced on the
ncreased an average of over 6% across
evels.
56%
50%
ible Special Education Services
setts Comprehensive Assessments
60%
40%
46%
+ 4 pts.
37%
29%
30%
39%
+ 7 pts.
+ 8 pts.
20%
Grade 3
Grade 4
2010 Percent
Grade 5
2011 Percent
Source
of assessment
http://profiles.doe.mass.edu/
Source
of assessment
data:data:
httphttp://profiles.doe.mass.edu/
Source
of demographic
data: data:
MDRMDR
Source
of demographic
60
Speake Public School
Danville, AL
District Demographics:
• This rural school serves 275 students in grades K–8 with a professional staff that
includes 36 teachers, with 22 support personnel and administrators.
• Student Ethnicities:
Caucasian 63%
American Indian/Alaska Native 28%
African American 5%
Hispanic 4%
• 67% Students eligible Free/Reduced Lunch
• 10% Students eligible Special Education Services
Measure: Alabama’s Reading and Mathematics Test (ARMT)
Period of Evaluation: 2010 (Baseline) to 2011
Speake
Public School
Meets or Exceeds on the state’s ARMT increased an average of over 7%.
Danville, AL
Summary: In only one year of using GO Math!, the percentage of students deemed
hics:
serves 275 students in grades K-8 with
ff that includes 36 teachers, with 22
l and administrators.
es:
3%
dian/Alaska Native 28%
erican 5%
%
Speake
SpeakePublic
Public School
School
Grades
ARMT
Grades3,3,4,
4, &
&55ARMT
PercentAt/Above
At / Above Meets
Meets
Percent
2010-2011
2010–2011
100%
90%
88%
91%
85%
80%
ible Free/Reduced Lunch
ible Special Education Services
70%
s Reading and Mathematics Test
n: 2010 (Baseline) to 2011.
one year of using Go Math!, the
udents Meets or Exceeds on the state’s
d an average of over 7%
67% 67%
+ 2 pts.
60%
67%
+ 18 pts.
50%
Grade 3
Grade 4
2010 Percent
Grade 5
2011 Percent
Source
assessment
http://profiles.doe.mass.edu/
Source
of of
assessment
data:data:
httphttp://profiles.doe.mass.edu/
Source
demographic
Source
of of
demographic
data:data:
MDRMDR
61
Notes
___________________________________________________________________________
___________________________________________________________________________
___________________________________________________________________________
___________________________________________________________________________
___________________________________________________________________________
___________________________________________________________________________
___________________________________________________________________________
___________________________________________________________________________
___________________________________________________________________________
___________________________________________________________________________
___________________________________________________________________________
___________________________________________________________________________
___________________________________________________________________________
___________________________________________________________________________
___________________________________________________________________________
___________________________________________________________________________
___________________________________________________________________________
___________________________________________________________________________
___________________________________________________________________________
___________________________________________________________________________
___________________________________________________________________________
___________________________________________________________________________
___________________________________________________________________________
62
Notes
___________________________________________________________________________
___________________________________________________________________________
___________________________________________________________________________
___________________________________________________________________________
___________________________________________________________________________
___________________________________________________________________________
___________________________________________________________________________
___________________________________________________________________________
___________________________________________________________________________
___________________________________________________________________________
___________________________________________________________________________
___________________________________________________________________________
___________________________________________________________________________
___________________________________________________________________________
___________________________________________________________________________
___________________________________________________________________________
___________________________________________________________________________
___________________________________________________________________________
___________________________________________________________________________
___________________________________________________________________________
___________________________________________________________________________
___________________________________________________________________________
___________________________________________________________________________
63
© Houghton Mifflin Harcourt Publishing Company. All rights reserved. Printed in the U.S.A. 03/12 MS11231
64
800.225.5425 • hmheducation.com