!
(&'(
"$%
$%
#
!
"
!
"!
"
'%&&
• Public/Private Ventures
2000 Market Street
Suite 550
Philadelphia, PA 19103
Phone: 215-557-4400
Fax: 215-557-4469
www.ppv.org
January 2012
In 2006, BBBSA partnered with Public/Private Ventures (P/PV) and Dr. Jean Rhodes to create a survey instrument to assess the program’s effects on participating youth. The new Youth Outcomes Survey (YOS) would
replace the Program Outcomes Evaluation (POE) measure that BBBS agencies had been using for almost a decade. The shift to using the YOS would embody larger organizational efforts to outline which targeted outcomes
they believed could and should be achieved through BBBS mentoring and to rigorously assess whether they were
yielding those outcomes.
This was a significant—and brave—step forward. The YOS would enable BBBS agencies to assess their outcomes using well-validated scales, allowing BBBSA to compare their effects to those yielded in other national
evaluations. The survey would also be administered twice instead of once, significantly increasing the rigor of
their evaluation efforts and the conclusiveness of their findings.
However, replacing the POE would be challenging on several levels. For example, the more conservative YOS
would measure change rather than gauge respondents’ beliefs about how youth had benefited, yielding effects
that, on the surface, might not appear to be as clear or as strong. In other words, the organization raised the bar
for its own agencies’ performance—and did so voluntarily. While they recognized that implementing the YOS
would not be a simple task, BBBSA believed it was a critical step toward accurately assessing their outcomes in a
way that would foster both program improvement and growth.
BBBSA has approached the survey instrument’s implementation in over 200 agencies nationwide with rigor
and thoughtfulness. They piloted the YOS with several agencies to ensure that it was user friendly and met their
needs; refined it based on agency feedback; provided agencies with tools and technical assistance to support its
use; and continue to refine it based on early results. This report presents the culmination of that work over the
past five years and the promising effects that have been measured using the YOS. Importantly, BBBSA is also
using these data to understand which of its agencies, programs and matches need more support. BBBSA’s efforts
to integrate a comparison group into this work is another major step toward ensuring that they are getting a
clear and accurate picture of how youth are benefiting from their programs and what can be done to strengthen
these benefits.
Over the past several years, the evaluation field has been moving steadily toward more rigorous methodology. P/
PV supports BBBSA’s response to this trend; we are confident that their decision to move to this stronger assessment tool will provide individual agencies and BBBSA leaders with data that can make a more convincing case
to funders, policymakers, and community partners about the program’s benefits. We also believe it can be used
to further the organization’s continued efforts to improve. We applaud BBBSA’s long-standing commitment to
quality and their thoughtful use of data and evaluation to understand and improve their programs. We believe
the YOS will support their already strong leadership of top-notch mentoring programs for our nation’s youth.
Sincerely,
Carla Herrera
Senior Research Fellow
Public/Private Ventures
P/PV is an equal opportunity employer.
!
(&'(
"$%
$%
#
!
"
!
"!
"
'%&&
• • • #($
! ! ! !#!!# !# &"!
& "! !"##!$! '! '"!&" '
!! )!"!& * # !!
!#"! &"! !!!!
! !! & $!!"!! ! "!
"!"#&-
.)!!"
"!#"!& !'!
, !
"#&&"!!! !-4. " (
)-!$.
• !
! '!
• !" !
'!!
• !!&
! - *!
• "!
!.
• "!%!! • !!!" $ o o o " # &# o o !!
o " o ! • "&
• !!&/
• -#!.
"! " • "&/
• "!%!! /
/-3. " $" ! !"!( !!&'
"&'"!%!! ) !" !!2010*2011
&! ! + * $! !$"" !2005!"&2010!"!&* )"!
"! "#&
!!&"!-5.!!'!!!!
# & !-"&"!&* ! &
* .)
#&!!"#'##!&#!"#!&""##"#("#
!%#!"#&-$,!%!("#!!###!#!"
"#!"#$"#"#!#"#%$#"##!&#!"$#"%"#
!"!%$#"#$"$!!!,
$% #%
! $##$!#!!!#"",$#!""!#&$#
!#!#"$!#"#(#"&#($#$""!!(#
!#"#"""!(#$"#!-"#!#""&#!%$#!",
$#$#!$"#""#"*#!#"#%$#"#$)","!#"##
!%"-##"/!"!#"*!#!*##!!
!#!#""0*$##(!%"!!*!!$#!#"
!"#!"(%!",
!#!""#!"$#(-"!!+
• #
• !#!$"#
• $#
!#!""#!"-"!!/-"-(!#"0+
• #
• $#
##!%#"($#."!!#"#(#!!"#!"
"$##!%"*&"$"#!%#!##!$"#!
"#!/0#!#""##!!#"##&(!","*!*
($##&-(!#"'!!!%#"#($#-(!#",
" "!$#"!#"#!($##%*$## $#
!($#%!"#($!#,&###!%###!(
"(!"##"!$#!#,"!"#!#"##!%#
!"%!"!!"#!!$#!#",&"*"#%
#"&""!$!!!""'$!!!"%"
$"!"! !( !(#""' #"*(" " !!" !
" !!" """'"!"$'"#"&""!
$!!!""' "'#"%! $*
" !!" !#"'+! )
• !""'
• #"&""!
" !!" !+! ,+!+' "!-)
• !""'
• !
• #"&""!
!#!"#" ! " "!!""
"" "!""%' !*!("!% " '#""%' "!
"'#"' "!*
Research outside of the mentoring field suggests that decreasing youth’s engagement in risky behaviors
benefits society, both financially and culturally, by keeping youth out of the juvenile court system and
creating safer communities. Our YOS scale measures attitudes toward risk rather than actual risk
behaviors, and we continue to work on strengthening these scales.
" !!" !#"'+! )
• """#!"% !
o o #!
o o o ""
o #!
o " "'!""!"'!" ! '#"# !+! !*
%"#"'"#!"! "" !!% "!" #"!
"! !*
! %+)** !$ " $ ! &
.,'+2
.+',2
/+'.2
.-'.2
$
-1'.2
-1'-2
1,',2
1*'12
"
/0'+2
/)')2
#
/*'/2
/-',2
$
-.'+2
-*',2
-)',2
-.'.2
(
!
$
!
!
! #" ! ("
&*' ! 83.5%
73.4%
! #" ! ! !
!
! !
*
+
*
-
.
/
0
1
22$/)3
21$*
.)$,
1)$+
/*$1
,1$2
*/$/
-
22$-)3
20$*
-.$+
0.$0
..$-
,,$.
*-$,
+$,
!
" %# " "'"(& "!$%""
" "$" $
! !%
""# $ "#"
+)**$