An Evaluation Primer: A Plan for Measuring
Program Quality and Outcomes in After School
and Summer Programs
by Brooke Culclasure, Ph.D.
May 3, 2016
Introductions
• Presenter is Dr. Brooke Culclasure, research director
in the Riley Institute at Furman University and
principal investigator of numerous studies across S.C.
• Introductions of session attendees/program
affiliation/description of current evaluation activities
Format of Presentation
• Discuss why evaluation is important
• Discuss why quality is important, related to
evaluation
• Discus past outcomes evaluations and their link to
quality
• Discuss available tools to measure quality
• Discuss QSA and how it has been used
• Breakout groups
Why is evaluation important?
• Provides us with useful information to help us
know our strengths and weaknesses, how to
improve our program, and better serve our
students
• Helps us highlight successes and be able to
tell our success stories
• Provides us with solid data to help us
compete for resources
Why is quality important?
• In after school and summer learning programs,
research has shown that only high-quality programs
produce positive outcomes for youth
• Many studies have looked at test scores, engagement,
and other factors…BUT programs rated as low quality
do not show impact
• Studies by scholars such as Vandell, Noam and others
all focus on quality as key in determining outcomes
How do we assess program quality?
• Becomes obvious, then, that we must focus on quality
• Numerous tools available, some are self-assessment
and some are conducted by external evaluators
• There are 3-4 widely used tools, we will go into detail
about one today
• Talk first about why self-assessment is a good choice
How do we assess program quality?
Why Self-Assessment?
• It helps program providers to understand overall
program quality and identify steps needed for
improvement
• It is a low-stakes process
• It can focus on program needs and priorities while
keeping areas outside the focus from slipping
• Low cost
How do we measure quality?
Four widely used tools
• Youth Program Quality Assessment (PQA)
• A Program Assessment System Training (APAS): Only
assessment that helps afterschool programs link quality
to youth outcomes!
• Dimensions of Success Observation Tool (DOS): STEM
focused
• Quality Self Assessment (QSA)
Quality Tools
What is the QSA Tool? (NY State Network)
• The QSA Tool is organized around 10 elements of quality
• It uses a set of indicators to help program providers
determine their strengths and areas in need of
improvement
• It is structured to lead to an action plan for increasing
program quality
• The tool was developed by experienced program and TA
providers to be relevant and reasonable
What Makes a Quality Program?
Ten Essential Elements of a Quality Afterschool Program
•
•
•
•
•
•
•
•
•
•
Environment and Climate
Administration and Organization
Relationships
Staffing and Professional Development
Programming and Activities
Linkages Between Day and Afterschool
Youth Participation and Engagement
Parent, Family, and Community Partnerships
Program Sustainability and Growth
Measuring Outcomes and Evaluation
Source: NY State Network for Youth Success
QSA Tool vs. Evaluation
QSA Tool:
Formal Evaluation:
• provides structure for
dialogues around outcomes
and quality.
• takes a more formal approach
to assessing programs to
determine what their
outcomes are.
• strengthens communication
between stakeholders.
• often uses data gathered for
accountability and
compliance.
• helps build a collective vision.
• uses data gathered to inform
practice and build capacity.
• is often facilitated by program
staff and/or other
stakeholders.
• has higher stakes and less gray
areas.
• is often facilitated by an
external observer.
Cycle of Improvement
Breakout Groups
• Divide into breakout groups
• Reconvene for discussion/reporting back
• Questions: Please contact Brooke Culclasure, 864294-3236, [email protected]