St t i Data
Strategic
D t Use
U
for
f
Afterschool and Summer Programs
Hartford Foundation for Public Giving
Evaluation Toolkit
Anita M. Baker, Ed.D.
Evaluation Services
Things to Ponder
What are you trying to
achieve?
When are data available?
From what source?
How can you use data
internally and externally?
1
Baker: www.evaluationservices.co
Evaluation Toolkit
What are Qualitative Data?
Qualitative data - come from surveys, interviews,
observations and sometimes record reviews
reviews. They
consist of:

descriptions of situations, events, people, interactions, and
observed behaviors;

direct quotations and ratings from people about their
experiences, attitudes, beliefs, thoughts or assessments;

excerpts or entire passages from documents, correspondence,
records, case histories, field notes.
Collecting
g and analyzing
y gq
qualitative data p
permit study
y of
selected issues in depth and detail and help to answer
the “why questions.”
! Qualitative data are just as valid as quantitative data!
2
Baker: www.evaluationservices.co
Evaluation Toolkit
What are Quantitative Data?
Q
Quantitative
data - come mostly
y from
surveys, program records (e.g., tests,
attendance), and standardized observation
instruments.


To obtain quantitative data it is necessary to be
able to categorize the objects of interest in ways
that permit assignment of numerical values and
counting.
counting
Depiction of quantitative data in tables and figures
is common and relatively easy.
! Quantitative data alone, never tell the whole story !
3
Baker: www.evaluationservices.co
Evaluation Toolkit
Strategic Use of Data =
Using Data to Influence Change
TARGET AUDIENCES
USE DATA TO:
Assess conditions

Identify service gaps and/or
unmet needs


C
Community
it R
Residents
id t

Parents, Students

School Staff, Officials

Monitor trends

Funders

Encourage and inform dialogue

Partner Agencies
(including staff)

Mobilize stakeholders/target
g audiences

Policy Makers (District

Support long-term/strategic planning

Answer key evaluation questions
4
Baker: www.evaluationservices.co
officials, School Board
members)

L i l
Legislators
Evaluation Toolkit
Data Sources and Data Collection Strategies
for Out
Out-of-School
of School Time (OST) Programs
Programs*
DATA SOURCES
DATA COLLECTION

Surveys

Surveys

School Data

Interviews

Program Data

Observations

Record Reviews
* Afterschool and Summer programs
5
Baker: www.evaluationservices.co
Evaluation Toolkit
Surveys Can Be Good Sources of
Available Data for OST Programs

RESPONDENTS
Students

FOCUSES
Satisfaction

T h
Teachers

Involvement

Parents

Recognition
g

Others

Feedback/Assessment
6
Baker: www.evaluationservices.co
Evaluation Toolkit
School Data: Common Categories
7

Descriptive

Attendance

Disciplinary

P
Progress

Standardized Achievement

Terminal Measures
Baker: www.evaluationservices.co
Evaluation Toolkit
School Data: Categories
g
and Specifics
p
Category
8
Specific Measures
DESCRIPTIVE
Grade, Age, Race/ethnicity, Gender, Economic Status
ED. DESCRIPTIVE
IEP, ELL, Over-age for Grade
ATTENDANCE
Average Daily Attendance, Total Number of Absences,
Excused and Unexcused Absences, Truancy/LTA, Punctuality
DISCIPLINARY
Number of incidents, referrals, in school and out-of-school
suspension, expulsion
PROGRESS
GPA, satisfactory performance in subjects, credit
accumulation, promotion
STANDARDIZED
ACHIEVEMENT
Raw Scores, Normative Scores/Ranks, Threshold Scores
TERMINAL MEASURES
Graduation, Drop Out, College Acceptance
(
(retention/graduation)
i / d i )
(at/or above proficiency)
Baker: www.evaluationservices.co
Evaluation Toolkit
School Data: Important Considerations
 Confidentiality,
Confidentiality FERPA
 Access
 Definitions
D fi i i
 Roll
Over
 Availability
9
Baker: www.evaluationservices.co
Evaluation Toolkit
Program Data: Common Categories

Program Descriptions (e.g., categorical –
academic,
d i enrichment,
i h
recreation;
i change
h
in
i offerings,
ff i
level
l l
of service)
10

Participation (multiple Units of Analysis

Program
g
Outcomes (e.g., points, skills test results)

CM Outcomes (e.g., # of referrals)
e.g. month,
day, group, participant; multiple measures, e.g., daily
attendance)
Baker: www.evaluationservices.co
Evaluation Toolkit
Program Matrix 2010
2010-11
11
Academic Support
pp
Tutoring
Test Prep
Truancy Prevention
Fall
Spring




PYD/Lif kill
PYD/Lifeskills

Trips and Special Events
Arts & Culture
Visual Arts
Computer Graphics
Songwriting and Performing






Sports
Basketball
Tennis

Science & Technology

Language
11



Baker: www.evaluationservices.co
Evaluation Toolkit

Planning For
Strategic Data Use
12
Baker: www.evaluationservices.co
Evaluation Toolkit
Data Use Planning: Considerations

Determine whether you can or should collect or present any
of the following.
g Be Specific!
p
Can you
y combine info?

Stakeholder Feedback

Program Participation Findings

S h lD
School
Data ((and
d other)
h ) Findings
Fi di

Figure out who is going to collect and analyze any of the
above.
above

Determine the best formats (e.g., oral reports, stand alone
reports,
epo ts, co
comprehensive
p e e s ve reports)
epo ts) for
o ppresenting
ese t g findings
gs to
specific audiences.

THINK ABOUT FOLLOW-UP– once yyou make somethingg
public, an action is usually expected.
13
Baker: www.evaluationservices.co
Evaluation Toolkit
Important Data
Data-Related
Related Terms

Data can exist in a variety of forms



Records: Numbers or text on pieces of paper
Digital/computer: Bits and bytes stored electronically
Memory: Perceptions, observations or facts stored in a person’s mind

Qualitative, Quantitative

Primary v. Secondary Data

Variables (Items)

Unit of Analysis
y

Duplicated v. Unduplicated

Unit Record (Client
(Client-level)
level) v. Aggregated
14
Baker: www.evaluationservices.co
Evaluation Toolkit
Data Needed to Support Strategic Use
Ideally, we want data that are:




Available on an ongoing and/or recurring basis (so that
we can measure change over time)
Relevant
Easily understood
Easily collectible and accessible




D t mustt often
Data
ft b
be cleaned
l
d tto b
be useable
bl
Cost-effective
Complete
p
and accurate
Capable of being added to over time


15
Must be kept up-to-date
Data storage requirements needed for the long-term
long term
Baker: www.evaluationservices.co
Evaluation Toolkit
Increasing Strategic Use of Data
at an Organizational Level
Plan for and direct data use.
1.
Know when required reports are scheduled
scheduled, who contributes,
contributes
what is contributed, and when, and who intended users are
Develop different reports/communiqués for specific stakeholders


2.
Incorporate evaluative thinking into data collection,
management and use.
3.
Assess staff needs and proficiency.
4.
D l / i t i ttechnology
Develop/maintain
h l
plans.
l



16
Hardware and software
TA/Training
Access and Integration
Baker: www.evaluationservices.co
Evaluation Toolkit
Increasing Strategic Use of Data
at an Organizational Level (Con
(Con’t.)
t.)
Collect and disaggregate data.
5.
Age, gender,
A
d program experience, etc.
Multiple perspectives on data interpretation
Professional and “community”
community perspectives



Help ensure consumers of data are trained in how to
use and interpret the data.
data
6.

7
7.
17
Provide and promote forums for interpreting and discussing data
C d t evaluations.
Conduct
l ti
Baker: www.evaluationservices.co
Evaluation Toolkit
Tips for Analyzing
Quantitative Data
18
Baker: www.evaluationservices.co
Evaluation Toolkit
Analyzing Quantitative Data:
A Few Important Terms
Terms*
• Case: individual record (e.g., 1 participant, 1 day, 1 activity)
• Demographics:
D
hi descriptive
d
i i characteristics
h
i i ((e.g., gender)
d )
• Disaggregate: to separate or group information (e.g., to
look at data for males separately from females) –
conducting crosstabs is a strategy for disaggregating data.
• Partition(v): another term that means disaggregate.
• Unit of Analysis: the major entity of the analysis – i.e.,
the what or the whom is being studied (e.g., participants,
groups activities)
groups,
• Variable: something that changes (e.g., number of hours
of attendance)
*common usage
19
Baker: www.evaluationservices.co
Evaluation Toolkit
A l i
Analyzing
Q
Quantitative
tit ti
Data
D t
Important Things to Look at or Summarize
Example Questions You
Could Answer
What to Do
What That Means
C l l
Calculate
Frequencies
F
i
C
Count
how
h many there
h are off something.
hi
Count how often something (e.g., a
response) occurs.
Calculate Total and/or
V lid Percentages
Valid
P
t
Frequency/total *100
How many participants
H
i i
were in
i
each group?
What were the demographics of
participants?
How many answered “Yes”
Yes to
Question 2?
What proportion of participants
met intensity targets?
What proportion of all those who
answered question 2, said “Yes.”
20
Baker: www.evaluationservices.co
Evaluation Toolkit
A l i
Analyzing
Q
Quantitative
tit ti
Data
D t
Important Things to Look at or Summarize
Example Questions You
Could Answer
What to Do
What That Means
Determine
Central Tendencies
Calculate the average (mean),
(mean) or
identify the median (middle) or mode
(most common value).
What is the average number of
hours participants attend?
Avg. =
What is the most common
numbers of days attended in a
week? (mode)
Sum of Values
Total Number of Values
Total # of hours
Total # of people with hours
21
Baker: www.evaluationservices.co
Evaluation Toolkit
A l i
Analyzing
Q
Quantitative
tit ti
Data
D t
Important Things to Look at or Summarize
Example Questions You
Could Answer
What That Means
What to do
Determine Distributions Determine the minimum value, the
What was the least amount of
attendance for the group? What
was the most?
maximum, and/or how the data are
grouped
(e.g, high, medium, or low values,
quartiles, percentiles, etc.).
Cross-Tabulations
(pivot tables are crosstabs)
22
How many participants fall into
low, medium, and high intensity
groups?
Relationship between 2 or more
variables (also called contingency
analyses, can include significance tests
such as chi-square analyses)
Baker: www.evaluationservices.co
Are there relationships between
participant characteristics and
outcome changes?
Evaluation Toolkit
EXAMPLES of
Analyzed Data
23
Baker: www.evaluationservices.co
Evaluation Toolkit
SOAR Afterschool Program*
Teacher Survey Results
School 1
N= 24
School 2
N=14
School 3
N=14
Know their school has SOAR After School Program
100%
93%
100%
Can ID SOAR Leaders
78%
83%
100%
Feel somewhat/very confident describing activities
34%
85%
57%
Have a role in SOAR
30%
67%
50%
Have been involved with SOAR
35%
58%
57%
Describe progress as somewhat/very noticeable
57%
67%
58%
Feel SOAR efforts result in positive outcomes
36%
75%
43%
Teacher Awareness: % who . . .
Teacher Involvement: % who . . .
Progress and Impacts: % who . . .
* SOAR is a fictitious program.
24
Baker: www.evaluationservices.co
Evaluation Toolkit
Parent Responses to School Climate Survey
% of Parents Who Agree
School provides opportunities for
parents to learn
50%
Families are invited to participate
65%
I feel welcome at this School
75%
0% 10% 20% 30% 40% 50% 60% 70% 80%
* Results fabricated to exemplify presentation.
25
Baker: www.evaluationservices.co
Evaluation Toolkit
School Data Example
Alger Middle School and Matching School F, Percentage of
St d t with
Students
ith 16+ Days
D
Ab
Absent,
t 2005
2005-2008
2008
45.0%
Alger Middle School
Matching School F
40.0%
34.3%
Perccentage of Studdents
35.0%
29.5%
30.0%
25.0%
28.6%
24 1%
24.1%
18.9%
20.0%
16.2%
15.0%
10.0%
5.0%
0 0%
0.0%
2005-2006
26
2006-2007
Baker: www.evaluationservices.co
2007-2008
Evaluation Toolkit
SOAR Program Attendance
2007 2008 vs
2007-2008
s 2008-2009
2008 2009
Schools with SOAR*
07-08 08-09 CHANGE
Enrolled
A
Average
T l
Total
Hours
964
1140
78.6
146.5
+86%
* *Target: 1000 students
students, 100 extra hours on average
* SOAR is a fictitious program.
27
Baker: www.evaluationservices.co
Evaluation Toolkit
Afterschool Attendance Intensity:
SOAR Program Data 2008
2008-09
09
SPRING SEMESTER, 2008-09
Round
R
d2S
Schools
h l
(2008-09)
n=1140
Round
R
d1S
Schools
h l
(2005-06)
n=915
146.5 hrs
166.9 hrs
Low (1 - 45)
45%
30%
Mid (46 - 99)
17%
17%
High (100 - 144)
11%
19%
TOP High(145+)
28%
35%
39%
54%
Average Attendance OST
Total Hours 
TARGET: 50% HIGH ATTENDANCE
(100+)
* SOAR is a fictitious program.
28
Baker: www.evaluationservices.co
Evaluation Toolkit
School Test Results by Program Attendance
SOAR Initiative 2008-09
Table 1: Percent of Students With Proficient State Test Scores
in Reading, Math, Writing (Round 1 Schools Only)
High
Intensity**
Lower Intensity
n=445
n=695
READING
77%
47%
MATH
53%
45%
WRITING
42%
29%
**High intensity = 100+hours
* SOAR is a fictitious program.
29
Baker: www.evaluationservices.co
Evaluation Toolkit