Washington 21st CCLC Evaluation
2015 Data Collection Activities
Neil Naftzger
Principal Researcher
February 2015
Copyright © 20XX American Institutes for Research. All rights reserved.
Topics
Evaluation update
Leading indicator surveys
Youth surveys
2
Evaluation Update
 Evaluation Activities in 2015
•
•
•
•
•
•
Leading indicator surveys (February to March 2015)
Youth surveys (April to May 2015)
Data for federal reporting (timeline uncertain)
Case study site selection and recruitment (Summer 2015)
Leading indicator reports and planning with data (Fall 2015)
Case study site visits begin (Fall 2015)
3
Leading Indicator Surveys
 Used to populate the leading indicator reports
•
•
•
•
Data obtained from federal reporting systems
Surveys
PQA data
Youth outcome data
 Purpose is to inform quality improvement efforts
 Leading indicator surveys collect data from two groups:
• Site coordinators
• Staff working directly with youth in the delivery of programming
4
Leading Indicator Survey Timeline
 Emails to project directors containing a link to the survey
management system and a username and password will
be sent during the week of February 16th
 Complete surveys from site coordinators and afterschool
program staff are due March 31st
5
Survey Administration Process
• Once logged into the Project Director Survey Management
page:
•
Ensure the centers listed for a given grant are correct and send any corrections
needed to [email protected]
•
Under the name of each center, a survey Web link unique to each center is
provided
•
This link needs to be forwarded to the site coordinator for that center
•
When the link is clicked on, the respondent will be asked to create a survey
account and then be prompted to take the survey in question
•
Site coordinators and staff will utilize the same link to create a survey account
and take the survey for their role in the program
•
You can return to the Project Director Survey Management page to view the
completion status of surveys and send email reminders
Staff Selection
• In addition to the site coordinator, ideally, a minimum of 8 staff per site should
be identified per center
• Staff selected to the survey:
 Need to spend the majority of their time working directly with youth by leading
activities
 Can be paid or a volunteer
 Can work for the grantee directly or for a partner organization
 Can be working in the program or recently ended their participation in the program
(e.g., led activities in a previous session that was active at some point in 2014-15)
 Should work (or did work) in the program at least once a week
• If there is less than 8 staff at a given center that meet these criteria, then all
staff at the center meeting these criteria should be asked to take the survey
Project Director Survey
Management Page
Center Coordinator and
Staff Login Page
Staff asked to
complete
student
reports will
need to first
create an
account
Creating a New Survey Account
Completion
of this field
will
determine if
staff take
the site
coordinator
or staff
survey
Clicking the create user button will take the user to either
the staff survey or site coordinator management page
Monitoring Survey Completion
• Project directors can see both:
•
Who has completed a survey account
•
Whether or not they have completed their survey from the Project Director Survey
Management page
• Automated reminders can be sent directly from the Project
Director Survey Management Page
• Project directors are encouraged to monitor the survey process
frequently
Youth Survey
12
Youth Survey Pilot
 Piloted a revised version of the Student Engagement,
Motivation, & Beliefs Survey developed by the Youth
Development Executives of King County
 Pilot involved 38 centers funded by 21st CCLC
 A total of 1,199 surveys were collected from 21st CCLC
participants enrolled in grades 3-10
13
Youth Survey Pilot
 Based on pilot results, the survey was revised to improve its
functioning
 Current version comprised of the following scales (54 items):
• Academic identity
• Mindsets
• Self-management
• School belonging
• Interpersonal skills
• Retrospective program impact on academic behaviors
• Retrospective program impact on self-management
• Program belonging and engagement
14
Revised Self-Management Scale
15
Revised Self-Management Scale
Self Management
100
Percent of Respondents
90
80
70
60
50
40
30
20
10
0
Not at all true
Somewhat true
Mostly true
Completely true
16
Youth Survey Next Steps
 Assessing if youth survey scores are correlated with school-related
outcomes
• State assessment results in reading and mathematics
• School-day attendance
• Disciplinary incidents
 Examining how these relationships may vary by grade level
 Additional revisions may be made to the survey based on results from
these analyses
17
Youth Survey Next Steps
 All centers serving youth in grades 4-12 will be asked to collect youth
survey data in the spring of 2015
 This data collection replaces the teacher survey (programs can
continue to collect these data if they find this information useful for
local evaluation purposes, but OSPI will not collect these data)
 Preference to collect data online
• Let us know if this is going to be an issue by sending an email to [email protected]
18
Youth Survey Next Steps
 Survey will involve a sample of participating youth in targeted grade
levels that meet certain program participation levels
 The statewide student identifier will be collected in relation to youth
taking the survey or some similar method will be employed to connect
survey data with the state data warehouses
 A draft of the survey will be available by March 1st, along with
additional information on data collection procedures
 Online collection of youth survey data will begin on April 1 and
continue through May 29
19
Youth Survey Next Steps
 A separate Webinar will be scheduled to review the survey interface
 For this cycle, we are interested in collecting responses from youth on
all survey scales
 Providing programs with an option of selecting scales relevant to their
programs in the future is under consideration
 Results from the youth survey will be made available in the leading
indicator reports
20
Neil Naftzger
630-649-6616
[email protected]
1120 East Diehl Road, Suite 200
Naperville, IL 60563-1486
General Information: 630-649-6500
www.air.org
21