Evaluation of Xtreme Reading
Implementation
Sean Murphy
North East Leadership Academy
Cohort 1
Spring 2012, NCSU
Problem of Practice
STRATEGIES
OBJECTIVES
A: Train teachers on the
13 instructional stages
used year long as core
instructional strategies
Teachers know and use
research-based
strategies effectively
B: Train teachers on
research based literacy
strategies through Xtreme
Reading Program
C: Train teachers to use
Possible Selves strategy
Teachers differentiate
instruction
Students collaborate
during class
Students use literacy
strategies proficiently
D: Provide on-going
support for Xtreme
reading teachers and
students
ULTIMATE GOAL
Teachers know
how to reach
struggling
learners
At-risk students
are more
confident and
successful
readers.
STRATEGIES
A: The 13 instructional
stages provided an
instructional routine of
pedagogic practice
(the “how”).
B: Xtreme Reading
included 8 research-based
literacy strategies
explicitly taught to
students (the “what”).
Evaluation Questions
Data
How many hours were spent training
teachers on 13 stages of instruction?
Training agendas
How many hours were spent
supporting teachers on the 13 stages
of instruction?
Support logs
Did the teachers think the session was
of high quality?
Professional Development
Questionnaire
On how many strategies were
teachers trained?
Training agendas
How many professional development
training sessions were given?
Support logs
To what extent did teachers think the
trainings were useful for their instruction?
Professional Development
Questionnaire
C: Possible Selves
was a motivational
strategy.
For the purpose of the project,
results were insignificant .
D: On-going support for
Xtreme Reading teachers
and students included
coaching, planning
assistance and modeling.
How many hours of support were
provided for the strategies?
Likert Scale
How supported did teachers feel
during the process?
Focus Group
Objectives
1: Teachers can explicitly
teach literacy strategies
to students (the “what”).
Evaluation Questions
Data
How confident do teachers feel using
research-based strategies?
How often do teachers use researchbased strategies?
Professional Development
Questionnaire
Teacher Self-Report Survey
2: Teachers differentiate
instruction, as laid out in
the 13 Instructional Stages
(the “how”).
3: Students collaborate
during class, as laid out in
the 13 instructional stages
(the “how”).
4: Students use literacy
strategies proficiently
How often do teachers differentiate
instruction?
Learning Strategies Walk
Through Rubric
Retrospective Pre/Post
Student Survey
Pre-Post Assessments for
Each Strategy
How often do students collaborate during
class?
How many students used the
strategies proficiently?
Results
A: The 13 instructional stages provided an instructional routine of pedagogically practice (the “how”)
• According to training agendas, teachers received approximately 4
hours of training on the 13 stages of instruction at the beginning of
the year.
• According to the support logs, teachers received 7 hours of support
specifically regarding teacher practices (13 stages of instruction).
Teachers did not receive support for 13 stages of instruction (as a
whole model) after initial training. However, there was specific
support for differentiation and student collaboration. For the
former teaching strategy, I provided 3 hours of support (mainly
through directive support and modeling) and curriculum coaches
provided .5 hours. Teachers received 1 hour of support on student
collaboration from myself and 2.5 hours of support from coaches.
Results
B: Xtreme Reading included 8 research-based literacy strategies explicitly taught to students (the “what”)
• Teachers were trained on a total of 3 strategies during the
first semester of implementation, conducted over 4 training
sessions. Teachers received a total of 12 hours of training
each during the Fall Semester. Teachers were trained on 2
strategies from January through March in 3 training
sessions, totaling 8 hours for each teacher.
• Over the course of the semester teachers were trained on
literacy strategies. Their reaction was mixed. During the
first session, 1 out of 3 teachers strongly agreed that
training was useful for their instruction, 2/3 disagreed. For
the second session, 1/3 strongly agreed and 2/3 agreed.
With the third session, all teachers strongly disagreed that
it was useful. All agreed the fourth session was useful.
Results
Results
C: Possible Selves was a motivational strategy.
• Questionnaires showed that 3/3 teachers felt
confident and prepared to teach the Possible
Selves strategy after training.
• According to interviews, teachers were
introduced and discussed how to use Possible
Selves strategies in the classroom. Sharing of
Possible Selves lessons was facilitated by
administration during weekly PLC sessions. Only 1
lesson was shared, student artifacts were
collected but not reviewed.
Results
D: On-going support for Xtreme reading teachers and students included coaching, planning assistance and modeling.
•
Support logs showed administration provided 26 hours of support either individually or with
Xtreme teachers as a PLC, over the course of the school year. Support came in the form of coaching,
modeling and facilitating planning. University of Kansas coaches provided 6 hours. Support included
coaching and facilitating planning.
•
According to self-report likert scales, 3 out of 3 teachers strongly agreed that administrators
prepared them for implementing strategies. They also strongly agreed that administration supplied
adequate on-going support to effectively teach strategies. However, the feelings about curriculum
coaches were less enthusiastic. 2 teachers agreed that the coaches prepared them for
implementing strategies and 1 disagreed. Regarding the on-going support from curriculum coaches,
1 teacher felt neutral about the level of support and 2 felt inadequately supported.
•
In a 30-minute focus group, teachers voiced negative feelings toward support 18 times and positive
feelings toward support 6 times. It was stated that more on-going support and coaching was
needed after initial training. Specifically, teachers felt “neglected,” “abandoned,” and “used” by the
curriculum coaches after training. They generally agreed that administration provided on-going
support (4 of 6 positive comments were directed toward administration).
Results
1: Teachers can explicitly teach literacy strategies to students (the “what”)
• Based on survey data, 66% of teachers they strongly
agreed that they were confident teaching three out of
four of the strategies they received training. They were
least confident teaching the Word ID strategy. In
comments, teachers said this training was ineffective
because it was conducted virtually through Face Time
link.
• 26 copies of the Learning Strategies Walk Through
rubric showed that teachers were using research-based
strategies on 12 occasions; 4 in January, 6 in February,
2 in March.
Results
2: Teachers differentiate instruction, as laid out in the 13 Instructional Stages (the “how”)
• According to 26 copies of the Learning
Strategies Walk Through Rubric, teachers
differentiated instruction 18 times; 2 in
January, 6 in February and 10 in March.
Differentiation consisted of purposeful
organization of group work, leveled texts, task
variety and individual support.
Results
3: Students collaborate during class, as laid out in the 13 instructional stages (the “how”)
• According to a retrospective pre/post survey, Xtreme Reading
students said the number of times they collaborated in the
classroom increased from the first days of school to the middle of
spring. Specifically, in response to a prompt asking, “how often did
you collaborate the beginning of the school year,” 33% responded
“about once a week”, 29% responded “once or twice a week” and
14% collaborated “everyday.” In the same survey, a prompt asked
student, “how often do you collaborate now?” 20% of students
marked that they collaborated “once a week,” 42% reported “once
or twice a week” and 25% said they collaborated everyday.
• In 26 copies of the Learning Strategies Walk Through Rubric taken
from December-March, students were collaborating 18 times.
Collaboration consisted of student paired-reading 8 times and
collaboration on projects 10 times. 2 in January, 6 in February, 10 in
March.
Results
Results
Recommendations: Strategies
•
Provide More Spiraled Support for 13 Stages of Instruction: One major lesson learned, in reflection, was the
necessity for clarifying the need for instructional change. One of the reasons Xtreme Reading was chosen as an
intervention program was that it required differentiated and collaborative work. At the time of initial training it
was assumed that this was already a routine, to some extent, in the classrooms. Thus, teachers did not have
ample training on effective pedagogy, as presented in the 13 instructional stages routine. Informal and formal
observations showed that South Johnston teachers relied heavily on direct instruction and assessment. This was
true both before and after the program, proving the need to measure the amount of differentiated instruction and
collaborative work. In preparing for next year’s implementation, much consideration and focused support should
be given on instructional change.
•
Offer more support. Three major changes should be made in supporting methods: 1) Frontload support, 2) Offer
more ongoing support, 3) Make support more directive. First, after consulting with the KU coaches, it is clear that
amount of support should be more frontloaded next year, with consistent visits and updates during the first
quarter. Second, while the amount of support might decrease in the second semester, open and regular dialogue
should persist throughout the implementation (including analysis of data collected). The third point will be
discussed in more detail below. At this point, though, it is worth mentioning that support should be more rooted
in the materials and include specific follow-up about how instruction can be improved.
•
Manage the Transition. In hindsight, it is clear that the transition from should have been planned and managed
more effectively. Many of the teachers made it clear in support sessions that they were unfamiliar with specific
strategies. They voiced apprehension with teaching reading strategies explicitly and had a hard time letting go of
traditional instructional practices (predominately lecture and test). This could be expected, but plans should have
including pathways forward, including methods for celebrating accomplishments and acknowledging areas for
improvement.
Recommendations: Objectives
•
Utilize the program’s resources. This was alluded to above in the “offer more directive support”
bullet. The Xtreme Reading program has a vast amount of resources, including lesson plans, pacing
outlines, and formative assessments that can inform instruction. Yet, almost all of the support we
offered (especially in the beginning of the year) did not put these materials to use. On a regular
basis, administration and/or instructional coaches should sit down with the teachers and the
materials. This can be used to ground conversation. Teachers should reflect on how closely their
planning and pedagogy follow the guidelines. Of course, teachers should have the autonomy to
make modifications; however, they must know what the standard looks like and try adhering to the
program as it is designed.
•
Make Goals Clearer and More Prominent: In retrospect, the outcomes for the program should’ve
been framed in both teacher and student terms. We were trying to change instructional habits as
much as content. However, this is a more delicate process, as former teaching habits were already
in place, demanding attention to the transition. What’s more, the entire instructional cycle should
have been emphasized, so that teachers were constantly reminded how differentiated teaching and
collaborative learning fit into the framework for teaching. Once the goals were stated more clearly,
systems should be in place to regularly monitor progress. The walk through form could have been
used more consistently with teachers, anchoring coaching conversations.
Evaluation Reflections
• Don’t ask so many questions. To be embarrassingly honest, I didn’t
mean for this PowerPoint to be this long. However, by asking so
many questions, I left myself with little choice. I remember Dr. Corn
saying, “Keep it simple, you’ll have to collect data on everything you
ask.” Well, I guess we learn from experience.
• Stop and smell the data. I waited too long to really start analyzing
the data I was collecting. If I had started earlier, I would’ve noticed a
pattern (content not changing, pedagogy changing) earlier on. This
could have shaped student interventions and teacher support.
• Sense of efficacy from understanding the process. In general, I didn’t
have the slightest idea about how to conduct an program
evaluation. This projects helped me take the long-view. By seeing
how the entire cycle is completed, I’ll be more prepared every step
of the way, from planning to reflection.