Hello my name is Kristin Nafziger, Project Director of the Program
Enhancement & Quality Assurance contract for the Texas ACE
Program. Welcome to the independent evaluation guide webinar.
In this webinar I will give an overview and highlight a few aspects of
the new Independent evaluation guidance from the Texas education
agency about the Texas ACE program.
Slide 2
The purpose of the independent evaluation is ultimately to inform
improvements in Texas ACE programs term to term and year to year
to get better outcomes for the students and families we serve. For the
past two years, TEA has requested that grantees submit a summative
evaluation report by the end of the grant period. This guidance
ultimately did not result in grantees submitting information to TEA
about the ongoing implementation of their program and the changes
they were making to improve over time.
In order to improve the utility of the evaluation activities and ultimately
get credit for all the hard work you do to implement these grants, we
have developed a new set of guidance for evaluation that focuses on
the use of a logic model.
The objective is to conduct the evaluation at the center level, address
specific Center level research questions, complete interim reports
that reflect on the implementation of programming, and finally use the
most rigorous research methodology available to make statements
about the outcomes that our programs achieve.
Slide 3
I really want to stress that it is not our goal to make more work for
Grantees. Again it is not our intent to make more work for you! The
goal is to get the work you are already doing with your evaluators as
part of gathering the ACE leadership team including the
project director the family engagement specialist, the site
coordinators; and engaging with the evaluator to document aspects of
implementing your program. What we hear from grantees is that this
is the evaluation work that is already going on.
So an additional goal is to have you working with your evaluator to
create more meaningful reports on the progress of all your hard work
and those interim reports will be really important when get to the
aspect of the evaluation is on the summative outcomes that your
program has.
Slide 4
For those of you who are familiar with the previous guidance you will
see that the first section around identifying and selecting an evaluator
remained the same. We left this in because it contains practical tips
and a process to identify an evaluator, which is especially important
for project directors who have not conducted independent evaluations
before.
Next it presents an overall statewide logic model for the program,
then outlines a generic logic model for a grantee and a Center.
It next outlines specific evaluation questions that TEA would like to
have grantees work with their evaluators to address. Notice that we
highlighted work with your evaluator; these are not reports, not
something that we expect the project director site coordinators to
complete without support from the evaluator.
Next, the guidance provides an outline of the expected evaluation
activities and cost guidelines. Which I hope you will notice remain the
same from the previous guidance. And it also describes each
deliverable that will be due to the Texas Education Agency.
The final thing it does is to provide a status update on the agencies
expectations for reporting on program outcomes.
Slide 5
First the guidance presents the logic model for the statewide
program; this should look familiar to many of you as it is been in
previous evaluation and performance metric webinars. You can see
highlighted in green where the role that the grantees and centers play
in the ace program while the other sections describe TEA's support
services.
Slide 6
When we get to the logic model for an ACE grantee we can get into a
lot more detail about the program. You'll also notice there is a theory
of action presented for the program. Basically the program
requirements are included in this theory of action-- that students in
need, spending additional time in well-structured and aligned after
school activities, taught by highly-qualified personnel, focused on the
four components will yield improvement in the academic performance,
attendance, behavior, and promotion and graduation rates of
students.
This is really the theory of action for the overall statewide program,
the federal program with a few exceptions, the grantee, and each
center.
You will also notice something specific to the grantee role is that the
role at the grantee level is to allocate differentiated resources and
support each center that is served by the grant so that the theory of
action can be implemented with fidelity at each center.
While it would be nice to have one logic model for every fiscal agent
in the ACE program the differences between an elementary center or
a middle school campus are so great, the resources so varied, and
the outputs so different based on student and family needs; to
conduct a useful evaluation you need to have a logic model for each
center.
Slide 7
At a High-level, each logic model for a center includes a description
of the available resources, key implementation practices which are
based on research, the set of activities that are scheduled at that
center and the outputs or the participation which is the planned
amount of time that you're going to have students and families
engaged in those activities. And finally, if the resources are in place
and the key practices are being implemented and the activities are
scheduled and students and families are receiving the services and
participating in the program then you might expect a change in the
intermediate outcomes and ultimately the impact that all students
graduate ready for college and career.
I encourage you to refer to the logic model training webinar that is on
MyTexasACE.org for more detailed information about the center logic
model and how to complete one for each center.
Slide 8
So now we've talked about the logic model and let's talk about the
first deliverable that will be due as part of the evaluation process.
By November 14 we would like to have each ACE grantee complete a
logic model for each center as well as modify and sign the template
that's in Appendix 2 for the evaluator agreement. We anticipate that
all of these deliverables would be able to be created with the help of
your evaluator over about 1 to 2 hours worth of meetings. We expect
the logic models to be based on information that is readily available to
you and based on your day-to-day work so it’s something that you're
very familiar with and should not be heavy left to complete
The purpose for the evaluator agreement is to simply assure TEA that
the project director and evaluator have read the expectations for the
evaluation activity and both agree to conduct these evaluation
activities.
For Deliverable 1, each project director will submit one email that will
include a logic model for each center along with the signed evaluator
agreement to the help desk. The logic models can be in the Microsoft
word template that is provided, or they could be in a PDF converted
format, could be in PowerPoint, or another format that is readable by
TEA to review the work of the grantee.
Slide nine
Next, the guidance outlines specific center level program evaluation
questions that TEA would like to have addressed across two interim
reports.
At a high-level there are key questions for each of the component of
the logic model. For example, in the resources component, grantees
and centers should ask themselves (in collaboration with their
evaluator) do we have the right resources to implement the program?
how do we know that? What evidence exists that suggests whether
we do or not?
The guidance goes on to outline a minimum set of questions the
grantees are expected to address for each component of each
center’s logic model.
Slide 10
For the first interim report due to TEA, grantees and centers will focus
on addressing component two of the evaluation questions. These
focus on the key implementation practices. Edvance is developing an
online survey to make the submission of the information as easy as
possible. This will be due January 27th of 2014.
Slide 11
For the second interim report due to TEA, grantees and centers will
focus on addressing components 1, 3 and 4 of the evaluation
questions. These components cover resources and outputs including
the scheduled activities and the actual participation in those activities.
Grantees will work closely with their evaluators to reflect on the
evidence that parts of their program are working and what parts could
be improved.
Slide 12
With regard to the evaluation activities that are outlined on page 14,
these have been slightly modified based on several interviews and
focus groups with project directors and evaluators about the work that
is typically conducted right now with evaluators. We’ve also added a
focus on the deliverables that will be due to TEA this year.
Again, notice that the cost guidelines remain the same, at a maximum
of $5000 per center.
Slide 13
In order to make causal statements about the impact that ACE
programs have on student outcomes, programs would have to
randomly assign students to the program and that’s never going to
happen.
The next best way to make statements that our program makes a
difference for students, is to use a quasi-experimental research
design. This involves grantees identifying a matched comparison
group of non-ACE participants to regular ACE participants.
TEA acknowledges that ACE grantees have varying access to data,
expertise, and resources in order to gather the match comparison
group data and conduct the analysis. Therefore, TEA and Edvance
are working together to create a matched comparison group for each
center. We are still exploring the feasibility of gathering this data
directly from the agency and providing it to each center. We will know
much more by the end of this year and will provide detailed guidance
about analyzing and reporting on program outcomes to each grantee.
Our hope is that we will be able to deliver a matched comparison
group for every center, we understand that there will be some
impacts based on timing and PEIMS reporting, but that is exactly
what we are studying right now.
Slide 14
We realize that this guidance is different from what you had at the
beginning of the grant year. We wish that we had developed this
guidance earlier in the year even before the program started. But
after we reviewed last years' evaluations and spoke to many project
directors and evaluators, we understood that the guidance from TEA
had to be significantly different in order to increase the usefulness of
the evaluation activity.
After conducting several focus groups with evaluators and with
project directors and sharing this information with the new cycle 8
grantees we realize that this work will be different that what was
asked for before, but it should not be extra. What we really learned
was that the summative report we were asking for before was extra
work because it was disconnected to the ongoing conversations you
were having with your having with your evaluator.
You will have access to training on completing the logic model; that is
available now. Training on completing the interim report one and
interim report two will be coming this fall as well as further guidance
on analyzing and reporting on program outcomes.
We recognize that there is still some uncertainty about how the
summative evaluations will be conducted this year but please
remember there is no penalty for following this guidance and we
believe that this logic model approach will give us an opportunity to
more fully tell our story. The story of how our ACE programs make a
tremendous difference for their students and their families.
Thank you so much for your attention and as always if you have any
questions contact your technical assistance consultant (your TAC)
email the help desk and know that we will be having several Town
Hall webinars for Q&A sessions as well in the near future.
Thanks again. Happy Evaluating.