YOUTH OUTCOMES SURVEY (YOS)
Examples
April 2011
Contents
Contents...................................................................................................................................... 1
Consent Forms ............................................................................................................................ 2
Example Language .................................................................................................................. 2
Telling Your Story ........................................................................................................................ 3
General Language ................................................................................................................... 4
National– OJJDP 2011-2012 (various OES – RR, SOR, and YOS) ............................................ 4
National – Altria 2011 (YOS adoption, baseline data) ............................................................ 4
National – Goizueta (combined measures of OES) ................................................................. 5
Example – extracted from “Establishing YOS Full Report to NLC” ......................................... 6
Agency Example – Accountabilities & Goals........................................................................... 7
Agency Example – Juvenile Justice ......................................................................................... 8
Logic Models ............................................................................................................................. 10
Agency Examples – for United Way ...................................................................................... 10
Submissions to this Document ................................................................................................. 13
.
1
YOUTH OUTCOMES SURVEY (YOS)
Examples
April 2011
Consent Forms
Consent Forms for All Participants
Your agency should collect consents from all match parties for collecting evaluation data,
including the YOS surveys.
For those children/parents already in the enrollment process, a parental/guardian consent
form must be completed – use of a “passive” consent form would be appropriate. (This
process entails sending out the “passive” consent form to parents of enrolled children –
asking them to notify the agency if they do NOT want to participate in the study/complete
surveys. No action is needed if parents give permission).
If your agency’s current child/parent application already contains appropriate language
requiring the consent of the parent/guardian to have them and their child(ren) participate in
agency evaluation services (ie. surveys, collection of report cards, etc), then no additional
steps are needed.
For all future participants, update your child application form to include parental/guardian
consent for evaluation surveys. Below are 2 common examples.
Example Language
(excerpt - sample 1)
I hereby request Big Brothers Big Sisters of XXX, Inc., a non-profit xxxx corporation, to make
available the services of BIG BROTHERS BIG SISTERS to my child, and, if possible, assign
him/her to a screened volunteer. I give my consent for myself and my child to participate in
all assessment and evaluation services, to cooperate and assist in all planning activities, and
to receive all services for my child as deemed necessary by Big Brothers Big Sisters XXX
(“Agency”). I consent to my child completing a questionnaire containing questions about
school, home life, and personal interests. I hereby knowingly release and hold harmless Big
Brothers Big Sisters of XXX and their nominees, assignees and designees from liability for any
damages incurred due to any injuries sustained by my child or myself as a result of my child’s
participation in any Agency sponsored activity or in being transported thereto. This
authorization shall be effective and continually in force, to the extent permitted by law, from
the date of this authorization until revoked by the parent/guardian with written notice, or
Successor Authorization, provided by BBBS, is executed.
(excerpt - sample 2)
I give permission:
1) for my youth to participate in the FULL AGENCY NAME Program
2) for the school to provide social and academic information about my youth to FULL
AGENCY NAME (e.g. report cards, attendance reports, behavior reports)
3) to have me and my youth complete a questionnaire containing questions about peer
relationships, feelings about school, grades, educational expectations, parental
relationships and attitudes toward risky behaviors
4) to have my youth talk with a FULL AGENCY NAME staff person about personal safety.
2
YOUTH OUTCOMES SURVEY (YOS)
Examples
April 2011
Telling Your Story
Agencies are encouraged to be creative and thorough in developing their case for support
using the concepts covered in this document. Details regarding research citations can be
found under Research, in the YOS section of Agency Connection. Also, agencies might want
to refer the latest FAQ’s regarding theory and data interpretation.
Define Your
Strategic
Outcome
Areas
Select YOS
Measures
Based on an
Understanding
of Trends
Collect &
Analyze Data
(compliance is
key)
Tell Your
Story
Remember, there are several ways you can interpret your data:
1. The percent of children either showing improvements and/or staying the same for a
specific outcome category or sub-category;
2. The percentage of positive outcomes. This measure uses a more general approach
to demonstrate general level of impact on youth, rather than by specific outcome
area; and,
3. Outcomes measures indicating statistically significant change (large sample size
req’d)
Other points to keep in mind:
 Never commit to a certain percentage increase (watch language that says improve
“by”) from baseline to follow-up.
 There is no “right” score; however, desired scores should always be in the upper half
of the scale
 There will continue to be a percentage of youth that don’t stay the same or show
improvement in a particular scale. This is particularly true when children score high
at baseline. Try sifting through your data to determine the children that need to
show improvement. For example, “out of those youth needing to show
improvement in their attitudes towards X risk, X% of them actually did.”
 Do not lead with statistical significance
 Mix it up! Use SOR, retention, etc. and continue to educate your funder about the
importance of quality. If possible, collect hard data, such as school grades.
3
YOUTH OUTCOMES SURVEY (YOS)
Examples
April 2011
General Language
Example of funder interested in specific outcome measures; uses % of youth
showing improvement or staying the same
“Within each of the outcome categories of _______, _______, and __________,
X% of our youth will demonstrate improvement.”
“X%-X% of youth who complete a minimum of a full relationship period, i.e.,
12 months for CB and one full school year for SB, will maintain or show
improvement in ___________.”
Example of funder interested in overall positive change, not necessarily a particular
area; uses % of youth showing improvement
“Of all the youth served in _____agency, X% will demonstrate improvement in X
number of positive outcomes.”
Example of funder interested in changes in high level, strategic outcome areas for
large sample size; uses statistical significance (not recommended for agencies at
this time)
“Within each of our 3 strategic outcome areas, we will have statistically
significant change in at least one associated outcome measure.”
Supplemented by: “Within each of our 3 strategic outcome areas, X% of
children will demonstrate improvement.”
National– OJJDP 2011-2012 (various OES – RR, SOR, and YOS)
Example of funding language for juvenile justice, combination of measures, YOS
using percentage of youth showing improvement in an area
“In addition, less than 15% of the youth matched can be arrested during the
sub-grantee cycle; at least 70% of the youth matched must demonstrate
positive change in at least two of the six areas related to risky behaviors,
assessed through the Youth Outcome Survey (YOS); and, at least 75% of the
youth matched must achieve positive scores on Strength of Relationship
(SoR) survey conducted at 3 months post match and again at the 12 month
anniversary of each match as reported in AIM.”
National – Altria 2011 (YOS adoption, baseline data)
This example focuses on programmatic/organizational impacts by using % adoption
and study of baseline data.
4
YOUTH OUTCOMES SURVEY (YOS)
Examples
April 2011
1. 96% compliance of the baseline survey by grantee agencies;
2. % of youth who personally disapprove of youth tobacco use through study of
baseline data collected in YOS throughout the year;
3. % of youth who personally disapprove of youth engagement in risky behaviors
through study of baseline data collected in YOS throughout the year.
National – Goizueta (combined measures of OES)
This example highlights the combination of measures available to funders due to
BBBS’s investment in the Outcomes Evaluation System, supplemented by hard data
collection.
Match Quality (Length +Strength=Outcomes)
Research shows that positive outcomes for mentored youth are directly related to
the length of the relationship and the strength of the relationship.1 Beneficial
impacts on a child are the result of strong connections with the mentor, forged over
time. Simply put: length + strength = outcomes. Big Brothers Big Sisters thus uses a
length + strength = outcomes framework to drive results for mentored youth; it has
developed measures of length, strength and outcomes to track those results. Each
youth in the program will be measured for success within this framework. Academic
data will also be collected from the schools to strengthen the significance of the
statistics, help inform agency partnerships, and substantiate BBBS’ role in closing the
academic achievement gap.
Each of the following metrics of length, strength, outcomes, and academic data will
be collected in AIM, BBBS’ proprietary information management database (with the
exception of hard school data which will be collected manually).
Length
Twelve-Month Retention Rate. BBBS uses a 12-month retention rate as its primary
indicator of match length. Twelve-month retention rate is the percentage of
matches that are still open after 12-months. The 12-month match retention rate will
be tracked by all pilot agencies with the goal to demonstrated steady improvement
at each agency over the course of the grant.
Strength
Strength of Relationship (SOR): The Strength of Relationship (SOR) measure is a tool
that will help agencies bring about positive outcomes for children. Designed by
mentoring research experts, the measure consists of 10 questions for mentees and
15 questions for mentors that help BBBS determine the strength of the mentoring
relationship. The survey will be given at the 3-month point and annually thereafter.
1
Dubois, David L., Holloway, Bruce E., Valentine, Jeffrey C., and Cooper, Harris. 2002. Effectiveness of Mentoring Programs for
Youth: A Meta-Analytic Review. American Journal of Community Psychology 30(2), 157-197.
5
YOUTH OUTCOMES SURVEY (YOS)
Examples
April 2011
Outcomes
Youth Outcome Survey (YOS): Using a pre- and post-test methodology, the YOS is
designed to track outcomes in the following areas: scholastic competency,
educational expectations, self-assessment, social acceptance, parental trust and
attitudes toward high risk behavior.
Academic Data
Collection of School Records: In order to measure key educational outcomes, pilot
agencies will partner with schools in order to secure school records. The focus in this
grant will be to collect and report on grade level advancement.
Example – extracted from “Establishing YOS Full Report to NLC”
SECTION I--OUTCOMES
The Littles of BBBS of _______ are making positive improvements to change their
perspectives and change their lives. Past independent studies of BBBS by Public/Private
Ventures have shown nationwide positive outcomes. BBBS of ______ can confirm its
own positive results as described below. These data do not describe all our positive
results. The surveys we used asked questions about a limited number of outcomes so we
could get an indication of our results on an on-going basis.
Some of our positive outcomes are in the following areas:
 Parental Trust--parental relationships have been associated with improved
grades and attendance
 Sense of Social Acceptance by Peers--improvements in peer relations have been
linked to a range of positive outcomes, including school achievement, improved
conduct and a lower likelihood of dropping out.
 Confidence in School Abilities--increases in this scale often precede performance
improvement. Higher levels of scholastic competence are associated with better
mental health and higher grades.
 Educational Expectations (how sure the child is that he/she will reach different
levels of schooling)--children and adolescents’ academic expectations and plans
have shown longitudinal associations with school performance and standardized
test scores.
 Grades—improvements in grades in elementary and middle school are
associated with increased likelihood to graduate from high school.
SECTION II-- PERCENT LITTLES WITH POSITIVE OUTCOMES:
o 62% of our SB Littles and 78.1% of our CB Littles have positive outcomes in
multiple areas (list the percent, for example, with two or more outcomes)
SECTION III-- GRADES & ATTENDANCE:
o X% of our Littles showed improvement in grades.
o X% of our Littles showed improvement in attendance.
SECTION IV--SoR:
6
YOUTH OUTCOMES SURVEY (YOS)
Examples
April 2011
o
o
o
X% of our matches felt they had a strong or very strong relationship (composite
score of over 4 at 12 mos.).
X% of Bigs agreed or strongly agreed that
o Their Littles had made improvements since they started meeting.
o They are well matched.
o They feel close to their Littles.
X% of Littles feel most of the time or always:
o Their Bigs have lots of good ideas about how to solve a problem.
o Their Big helps them take their mind off things by doing something with
them.
o They are safe when they are with their Littles.
o Their relationship with their Big is very important to them.
o They feel close to their Bigs.
SECTION V: OVERALL SCORES
o Our agency did better that the national average on overall Outcomes or SoR
scores. And/or: our agency improved by X% in our overall Outcomes or SoR
scores compared to last year.
Agency Example – Accountabilities & Goals
Excerpt: Evidence of Program Effectiveness/Program Evaluation
1. In addition to using the required DFPS instruments such as the PFS pre- and post-tests,
client satisfaction surveys, etc., what other measurement tools will be collected during
the contract period to demonstrate on-going program effectiveness and client
satisfaction?
Big Brothers Big Sisters of _______ will administer a relationship quality survey and
an outcome measure tool (Youth Survey) during the course of a match to
demonstrate program effectiveness. In 2006 BBBS of America contracted with Jean
Rhodes (University of Massachusetts at Boston), Jean Grossman (Princeton
University and Public/Private Ventures), and Carla Herrera (Public/Private
Ventures) to create a new measure for us utilizing existing outcome scales. The
final version of the measure was completed in April 2007, and the Relationship
Quality survey was piloted with a selected group of agencies that year. The new
Relationship Quality survey was officially launched at the beginning of 2008. The
new outcome measure went through a phase 1 pilot in the fall of 2007, and this
year has gone through a second phase of piloting with a selected group of BBBS
agencies—ending in November 2008. The official launch of the new Outcome
measure was in the fall 2009 with post results anticipated beginning fall 2010.
The Relationship Quality survey is administered to youth and mentors in
Community Based programs at 3 months, 12 months and each year thereafter
during their relationship with their mentor. In School Based programs the survey is
administered at 3 months, the end of the school year and each year thereafter. The
7
YOUTH OUTCOMES SURVEY (YOS)
Examples
April 2011
youth survey includes ten questions that examine three dimensions of relationship
quality: “helped to cope” (“My Big has lots of good ideas about how to solve a
problem.”), “not unhappy” (“When I’m with my Big, I feel disappointed.”), and
closeness (“How close do you feel about your Big?”). The mentor survey includes
17 questions on three dimensions of relationship quality: relationship quality (“I
think my Little and I are well-matched.”), closeness (“How close do you feel to your
Little?”), and decision making (who makes the decisions about activities—the Big,
the Little, the agency manager, or someone else like a teacher or parent).
The new BBBS Outcomes Measure will be administered to the youth within a
month before they are matched and then at the end of the school year for School
Based matches (SBM), and at the one year anniversary for Community Based
matches (CBM). It will then be administered on a yearly basis for the length of the
match relationship. The survey has 32 questions that examine six dimensions of the
mentoring relationship impact on the development of social competence:
scholastic efficacy (“I feel that I am just as smart as other kids my age.”); social
acceptance (“I find it hard to make friends.”); educational expectations; grades;
misconduct and skipping; and parental trust (“My parents/guardians respect my
feelings.”).
These measures have been found to be reliable and valid and used in previous
youth development research. In addition several of the measures have been linked
with longer term outcomes. For example, increases in scholastic efficacy, is
associated with improved mental health and higher grades in both children and
adolescents, and increases in social acceptance predicts improvements in peer
relations, increases school achievement, improved school conduct, and decreases
the chance of dropping out. Furthermore, educational expectations have been
linked to increases in school performance and increased standardized test scores.
Lastly, parental trust has been linked with improved grades and attendance, as well
as reduced substance use. All of these are key indicators of improvements in social
competence.
Agency Example – Juvenile Justice
Excerpt: Big Brothers Big Sisters of ______________ seeks to prevent negative behaviors
among at-risk youth that lead to contact with the juvenile justice system. This goal will be
achieved by accomplishing the following objectives after one year of mentoring:
The following example was edited for lesson learned: In other words, include maintain or
increase to accommodate a funder’s need for a higher percentage goal






60% of children demonstrate maintained or increased parental trust
60% of children demonstrate maintained or increased in sense of social acceptance
70% of children demonstrate maintained or increased confidence in school abilities
50% of children demonstrate maintained or increased educational expectations
60% of children demonstrate maintained or improved grades
70% of children demonstrate maintained or improved school attendance
8
YOUTH OUTCOMES SURVEY (YOS)
Examples
April 2011
 60% of children demonstrate maintained or increased disapproval of unhealthy behaviors
The outcomes are based on well-established protective factors that can increase a youth’s
chance of staying in school and becoming a productive citizen.[1] These protective factors
include:
1. Parental Trust. Parental relationships have been associated with improved grades
and attendance;
2. Sense of Social Acceptance by Peers. Improvements in peer relations have been
linked to a range of positive outcomes, including school achievement, improved
conduct and a lower likelihood of dropping out of school;
3. Confidence in School Abilities. Increases in confidence often precede performance
improvement. Higher levels of scholastic competence are associated with better
mental health and higher grades;
4. Educational Expectations. Children and adolescents’ academic expectations and
plans have shown longitudinal associations with school performance and
standardized test scores;
5. Grades. Improvements in grades in elementary and middle school are associated
with increased likelihood to graduate from high school;
6. School attendance. Improved school attendance reduces risks associated with
truancy, including juvenile delinquency and violent crime; and
7. Disapproval of Unhealthy Behaviors. Changing attitudes towards breaking school
rules, violence, and underage tobacco, alcohol, and illegal drug use can impact
behavior as a child matures.
9
YOUTH OUTCOMES SURVEY (YOS)
Examples
April 2011
Logic Models
Agency Examples – for United Way
In developing logic models, agencies will need to consider which outcome measure(s) best
represent the goals of their communities. In this case, the logical model (Example 1) was
developed to meet the United Way’s focus on educational success.
Example 2 merely represents how a logic model might be altered to accommodate—in this
case—the United Way’s requirement for long-term impact as part of their measure of
success.
10
YOUTH OUTCOMES SURVEY (YOS)
Examples
April 2011
MENTORING PROGRAM LOGIC MODEL
INPUTS
ACTIVITIES
OUTPUTS
In order to accomplish
our results we will need
the following resources
and tasks (completed or
ongoing):
In order to achieve our results we will
conduct the following service activity:
We expect that once
completed or underway this
activity will produce the
following outputs:
Youth in need of
mentoring
relationships
Recruit volunteers in the
community
Community adults
seeking opportunities
to be mentors
Meet 2-4 times per month for at
least one hour for a year at school
or community based locations.
Volunteer mentors are matched
with youth mentees, ages6-17
Activities will be youth-oriented:
Mentors and mentees
are matched in a one-toone mentoring 12 month
relationship.
Indicators:
# of volunteer
applications received
# of interviews
conducted
How/When Measured:
 training for mentors
Conduct Agency-sponsored
match activities to encourage
safe, healthy relationship
development.
 ongoing supervision
and support with
mentor and mentee
Provide ongoing support contact
with mentor, mentee, and
caregiver to support the match.
Program provides:
 volunteer mentor
background checks
 interview of
potential mentor
and mentees
 matching of
mentors and
mentees
We expect that if completed
or ongoing this activity will
lead to the following short
term results:
A trusting relationship
forms between mentors
and mentees.
Indicators:
 School based is on-site and
may include sharing lunch,
reading books, or playing on
the playground.
 Community-based activities
may include: recreational
outings to the movies or a park;
eating meals together at
restaurants, sporting events, or
other locations that mentor
and mentee select together.
Qualified staff trained
to administer the
program
SHORT TERM OUTCOMES
#of youth off waiting list
# of matches made
# match meetings
% monthly match
support contact
# attend agencysponsored activities
% of mentors/mentees
with positive responses
(Mentors 3.85 or higher;
Mentees 4.65 or higher)
about their relationship
(e.g. good
communication)
# matched at least 6
months
LONG TERM OUTCOMES
We expect that if completed or ongoing
this activity will lead to the following
long term results:
IMPACT
If outcomes are
successful we
expect this
community change:
Mentees will show improvements
in one or more of the strategic
outcome areas:



Attitudes/Competencies
Educational Success
Risk Avoidance
Indicators:
% of mentees with increased
positive responses-higher scores on
a post-survey-in one of eight
categories (social acceptance,
scholastic competency, educational
expectations, grades, risk attitudes,
parental trust, truancy, special
adult)
Increase of youth
positively
engaged in
society
Increase in high
school graduation
rates
Reduction of
juvenile crime
# matched at least 12 months
How/When Measured:
 AIM report of metrics
How/When Measured:
 AIM report of metrics
 AIM report of metrics
 Mentor Strength of
Relationship Survey (3
months)
 Match Support Notes
(monthly year 1;
quarterly thereafter)
Mentee Strength of
Relationship Survey (3
months)
 Mentor Strength of
Relationship Survey (annually)
 Youth Outcome Survey (prematch, annually)
 Mentee Strength of
Relationship Survey (annually)
11
YOUTH OUTCOMES SURVEY (YOS)
Examples
April 2011
MENTORING PROGRAM LOGIC MODEL
INPUTS
ACTIVITIES
OUTPUTS
In order to accomplish
our results we will need
the following resources
and tasks (completed or
ongoing):
In order to achieve our results we will
conduct the following service activity:
We expect that once
completed or underway this
activity will produce the
following outputs:
Youth in need of
mentoring
relationships
Recruit volunteers in the
community
Community adults
seeking opportunities
to be mentors
Meet 2-4 times per month for at
least one hour for a year at school
or community based locations.
Volunteer mentors are matched
with youth mentees, ages6-17
Activities will be youth-oriented:
Qualified staff trained
to administer the
program
Program provides:
 volunteer mentor
background checks
 interview of
potential mentor
and mentees
 matching of mentors
and mentees
 training for mentors
 ongoing supervision
and support with
mentor and mentee
 School based is on-site and
may include sharing lunch,
reading books, or playing on
the playground.
 Community-based activities
may include: recreational
outings to the movies or a
park; eating meals together at
restaurants, sporting events, or
other locations that mentor
and mentee select together.
Conduct Agency-sponsored match
activities to encourage safe,
healthy relationship development.
Provide ongoing support contact
with mentor, mentee, and
caregiver to support the match.
Mentors and mentees
are matched in a one-toone mentoring
relationship.
SHORT TERM OUTCOMES
INTERMEDIATE OUTCOMES
We expect that if completed
or ongoing this activity will
lead to the following short
term results:
We expect that if completed or
ongoing this activity will lead to
the following intermediate results:
Mentors and mentees
meet regularly for 12
months.
A trusting relationship forms
between mentors and
mentees.
END OUTCOMES
We expect that if
accomplished this activity
will lead to the following
end results:
Mentees will show
improvements in one
or more of the strategic
outcome areas.
Indicator:
Indicator:
Indicator:
# of volunteer
applications received
# of matches made
# of interviews
conducted
# of youth off waiting list
# match meetings
% monthly match support
contact
# attend agencysponsored activities
Percent of mentors/mentees
with positive responses
(Mentors 3.85 or higher;
Mentees 4.65 or higher)
about their relationship (e.g.
good communication)
# matched more than 6
months
How/When Measured:
Indicator: Percent of
mentees with increased
positive responseshigher scores on a postsurvey-in one of eight
categories (social
acceptance, scholastic
competency,
educational
expectations, grades,
risk attitudes, parental
trust, truancy, special
adult)
How/When Measured:
How/When Measured:
 AIM report of metrics
 AIM report of metrics
 AIM report of metrics
 Mentor Strength of
Relationship Survey (3
months, annually)
# matched more than
12 months
 Mentee Strength of
Relationship Survey (3
months, annually)
 AIM report of metrics
 Match Support Notes
(monthly year 1;
quarterly thereafter)
How/When Measured:
 Youth Outcome
Survey (pre-match,
annually)
12
YOUTH OUTCOMES SURVEY (YOS)
Examples
April 2011
Submissions to this Document
Thank you to the agencies who submitted examples for inclusion in this document. It will continue to be a work
in process! If your agency would like to be considered for inclusion in this document, or if you have questions
regarding its content, please contact Sandra LaFleur at [email protected] and cc Veronica Rosa at
[email protected] .
13