Increasing the Clinical and Programmatic
Utility of BCFPI and CAFAS
Prepared by:
Sandra Cunning, Ph.D.
Director of Research, The George Hull Centre for Children and Families
On behalf of:
Jewish Family and Child Service
Funded by:
The Provincial Centre of Excellence for Child and Youth Mental Health at CHEO
TABLE OF CONTENTS
Area
Overview
Goals & Outcomes
Training
BCFPI & CAFAS Process Outcomes
Examination of BCFPI & CAFAS Data
Summary and Recommendations
Appendices
•
Appendix A: Logic Model
•
Appendix B: Work Plan
•
Appendix C: CAFAS Overview
•
Appendix D: BCFPI Overview
•
Appendix E: BCFPI Average Scores
•
Appendix F: BCFPI Problem Prevalence
•
Appendix G: BCFPI Income by Family Structure
•
Appendix H: CAFAS Pre-post Test Scores for All and Completed Cases
Page
3
3
4
4
5
6
7
7
9
11
16
22
24
26
28
2
Overview
Jewish Family and Child Service’s (JFCS) Jerome D. Diamond Adolescent Centre (JDD) is a
children’s mental health centre day treatment program. JDD serves approximately 34 youth
between the ages of 12 to 17 in four classrooms staffed by 4 special education teachers from the
Toronto District School Board. The program also is staffed by a program manager, two social
workers, four child and youth workers, and one support staff.
The JDD collects data for the Brief Child and Family Phone Interview (BCFPI) and the Child and
Adolescent Functional Assessment Scale (CAFAS). JDD currently collects BCFPI and CAFAS data
that is submitted to the Ministry of Children and Youth Services. However, due to limited
resources, the Centre has been unable to fully explore the data and apply the findings to develop an
overview of the clients accessing the centre or use the data for quality assurance and program
planning purposes. The main purpose of the project was to increase the Centre’s capacity to
understand and utilize BFCPI and CAFAS data.
Goals & Outcomes
In the initial proposal, JFCS identified 3 key goals related to BCFPI and CAFAS data collected for
use in the JDD Centre. These goals were aimed at:
•
Developing an overview of the clinical and program needs of the youth served by the Centre
•
Assisting staff in developing an increased awareness of the youth they serve
•
Building infrastructure capacity to store and analyze BCFPI and CAFAS data for the purpose of
program evaluation, quality assurance, clinical planning, and program development
Specific outcomes of the project were to:
•
Systematically store, integrate, and analyze BCFPI and CAFAS data
•
Train JDD staff in the analysis of BCFPI and CAFAS data
•
Develop templates for summarizing client data to inform clinical and program development and
to measure the effectiveness of clinical interventions
•
Disseminate findings to engage staff, clients, and families in the interpretation and review of
clinical progress
Although all key outcomes of the project are attainable, some are more distal (e.g., disseminate
findings to engage staff and families in the interpretation and review of clinical progress) and beyond
the scope of the current project. However, more immediate outcomes, as seen below, will position
JDD in achieving these longer-term goals.
A meeting with the Director of Family and Community Services to review the proposal and
staff/centre needs facilitated the development of a brief logic model (Appendix A) that articulated 3
key areas of focus for the current project. For each area, specific activities, deliverables, and
outcomes were identified. The main areas of focus for the current project included increased
understanding, integration, and reporting of BCFPI and CAFAS data.
A work plan to achieve the project goals was developed and presented to the BCFPI/CAFAS
Working Group and is included in Appendix B.
3
Staff Training
As outlined in the logic model (Appendix A), a key outcome of the current project the increased
ability of staff to interpret BCFPI and CAFAS data/reports. Given this objective, training sessions
on BCFPI and CAFAS are scheduled for JDD program staff for May 18th. Training will include an
overview of the purpose, scope, and scores for each measure. Training also will include
interpretation of BCFPI and CAFAS reports from select clients at the JDD Centre. Training
presentations for CAFAS and BCFPI are included in Appendix C and D, respectively.
Note: Due to the limited timeframe of the funding/reporting period, school schedules (e.g., March
break), and holidays (e.g., Passover and Easter), staff training has been scheduled for May. A brief,
follow-up report will be submitted following staff training.
BCFPI and CAFAS Process Outcomes
Although specific training pertaining to BCFPI and CAFAS is scheduled for JDD staff, other staff
members also are involved in the collection, storage, export, and usage of BCFPI and CAFAS data.
Therefore, IT, clinical, and CQI staff also were trained with respect to “data” aspects of the
measures.
In terms of the BCPFI, staff were shown how to:
Export form data, including setting parameters such as referral dates
•
Create “canned” reports including: MH x Months Waiting, MH Details, MH Problem
Prevalence, MH Average Scores, and Family Structure x Income
•
View and manipulate graphs for customized reports (e.g., using pivot tables to breakdown
outcomes by age group)
•
View and manipulate Excel spreadsheets to create lists for various tasks including cleaning data
(e.g., checking for waitlist cases that have opened to service or closed cases that remain open)
and waitlists sorted by problem severity and months waiting
•
Check and clean BCFPI data
•
Add ID’s to sync BCFPI and CAFAS cases
•
IT staff also were able to create a Lotus Notes spreadsheet that captures key demographic (e.g.,
gender, d.o.b.) and Centre information (e.g., ID codes) to facilitate the data cleaning process.
JFCS has been provided with an Excel spreadsheet (hard copy and electronic) that highlights data
cleaning and verification points. Reminders around completing BCFPI include:
•
Adding ID codes to match CAFAS ID codes (not including dashes)
•
Checking key data points (e.g., d.o.b.)
•
Opening or closing older cases as necessary before running MH x Months Waiting reports
•
Verifying filters before exporting data
•
Verifying use of correct database when running reports
•
Checking BCFPI Website for updates and info on best practice (http://www.bcfpi.org)
4
In terms of CAFAS, staff were shown how to:
•
Create a new CAFAS “junk” site to remove problem CAFAS forms (e.g., duplicates, errors)
•
Move CAFAS cases between sites
•
Create an internal standard export including client and clinician names for data cleaning
•
View and manipulate Excel spreadsheets to clean data and export data to SPSS
•
Run some preliminary analysis of CAFAS data in SPSS (e.g., explore, frequencies, t-tests)
JFCS has been provided with an Excel spreadsheet (hard copy and electronic) that highlights data
cleaning and verification points. Reminders around completing CAFAS include:
•
Checking ID codes and removing all dashes (this may pose a problem with SPSS)
•
Ensuring all clients have a birthdate
•
Verifying status of cases that have been open for more than 2 years
•
Rating the most severe behaviour in the past 1 month period as outlined in CAFAS in Ontario
•
Identifying type of rater based on client (e.g., Manager vs. Treating Therapist)
•
Reviewing coding guidelines for CAFAS common data set (hard copy provided)
•
Checking CAFAS in Ontario Website for updates (http://www.cafasinontario.ca)
Examination of BCFPI & CAFAS Data
Overall both BCFPI and CAFAS data for JDD was relatively clean. In terms of BCFPI, cases will
need to be updated before accurate waitlists can be generated. However, the Centre can run various
analyses to explore Average BCFPI scores (Appendix E), Problem Prevalence (Appendix F) and
Income by Family Structure (Appendix G) reports.
Results of all BCFPI’s conducted by the Centre over the last 4 years (N = 184) reveals that clients
presenting to the Centre have, on average, Total Externalizing and Internalizing scores that are in
the Clinical range when compared to the general population. Overall Child and Family Functioning
are also in the Clinical range with Global Family functioning being the highest score. In specific
subscales, BCFPI results suggest that, youth being referred to the Centre are experiencing particular
difficulty in Managing Mood, Self-harm*, Social Participation and School Performance.
In terms of Problem Prevalence, 50% or more of the referrals to the Centre experience problems in
the areas of Regulation of Attention and Impulsivity, Managing Mood and Self-harm*, Social
Participation, Quality of Relationships, School Performance, Family Activities and Family Comfort.
*Note: Calculated based on number of referrals completing these items
For CAFAS data, 4 cases were identified as duplicates and removed from the analyses resulting in a
total of 45 cases with pre-post test data. Initial analyses revealed that 64% of these cases showed
some level of improved functioning between beginning and end of service. The average amount of
improvement between total pre-test (M = 104.00, SD = 34.34) and post-test (M = 78.44, SD =
45.48) was significant [t (1, 44) = 3.91, p < .01] (See Appendix H). However, closer examination of
the data revealed that, of the 16 clients who did not show some level of improvement, 12 clients had
an interruption in their treatment. Removal of clients who demonstrated disruption in treatment
(e.g., quit prematurely) resulted in a sample of 27 clients with complete pre-post test scores. The
average level of improvement between pre-test (M = 99.26, SD = 34.96) and post-test (M = 57.04,
SD = 37.91) was 42.22 points [t (1, 26) = 5.43, p < .01] (See Appendix H). Removal of the clients
5
who did not complete service resulted in 78% of clients showing some level of improved
functioning and less variability in outcome scores. In terms of specific subscales, non-parametric
analyses (Wilcoxon Signed Ranks Test) revealed significant improvements between pre-test and
post-test scores for School Performance, Home Role, Community, Behaviour Towards Others,
Mood, Self-harm and Thinking (p < .05).
Summary & Recommedations
Overall, there is increased capacity for the staff to understand BCFPI and CAFAS data, export and
analyze the data, and integrate these measures into their clinical work and program planning. Results
from the data also suggest that program is receiving referrals from a high needs population and,
although numbers are small at this point, effectively serving those clients, particularly when
treatment is not disrupted. Next steps for JFCS are to:
•
Finish cleaning BCFPI and CAFAS data
•
Explore BCFPI data by running various reports (e.g., referrals vs. active cases vs. nonmaterialized cases, yearly trends in referrals, age x gender profiles)
•
Explore CAFAS data
•
Match BCFPI and CAFAS cases
•
Install BCFPI (and possibly) CAFAS on CQI staff computer
•
Develop customized reports for CQI and client purposes
•
Increase interaction between staff doing different “components” of the process
•
Invest in staff training in Excel to help staff utilize information from CAFAS and BFCPI more
effectively
Beyond the planned outcomes, unintended outcomes from the project also have occurred. For the
first time in recent history JFCS and The George Hull Centre for Children and Families have
collaborated in an evaluation/research process. This has led to an increased understanding between
the Centres and has paved the way for future collaboration in this area. Another outcome is the
ability to use BCFPI data to provide accurate information pertaining to staff workload. For example,
in the 2005-06 fiscal year, the individual responsible for BCFPI entry completed 63 referrals. A final
outcome that will hopefully result from this project is the addition of centre ID’s to BCFPI
“canned” Excel reports to facilitate data cleaning and matching.
6
Appendix A
Logic Model
Jewish Family and Child Service BCFPI/CAFAS Project Logic Model
Objectives
Activities
Deliverables
Outcomes
Increased Understanding of
BCFPI/CAFAS
Staff Training on Measures
•
Overview BCFPI/CAFAS
•
Review of scoring/interpretation
Training/Info Package
•
BCFPI
•
CAFAS
Staff
•
Increased ability to
interpret BCFPI/CAFAS
reports
Increased Integration of
BCFPI/CAFAS
Data Cleaning
•
Cleaning BCFPI/CAFAS data
•
Matching BCFPI/CAFAS cases
Data
•
Improved data quality
•
Data cleaning and
extraction procedures
Integration
•
Increased ability to
generate and analyze
BCFPI /CAFAS data
•
Increased ability to match
BCFPI/CAFAS data
Increased Reporting of
BCFPI/CAFAS
Training/Support
•
Creating plug-in reports and
custom report
Reports
•
Standard reports
•
Template for custom reports
Usage
•
Increased ability to generate
reports for key stakeholders
Appendix B
Work Plan
Staff Training
JEWISH FAMILY AND CHILD SERVICE
BCFPI/CAFAS STUDY WORK PLAN
Description
Date
Developing work plan and training
April 4, 2006
materials
April 15, 2006
Meetings with working group to
March 10, 2006
establish logic model, review work plan March 17, 2006
and materials and presentation of final April 5, 2006
report
April 24, 2006
Additional
Extraction of BCFPI and CAFAS data April 10, 2006
to check integrity. Necessary cleaning
April 11, 2006
of data. Procedures for cleaning and
April 17, 2006
creating. Methods for matching cases.
April 18, 2006
Preliminary reports. Meeting with key
May 5, 2006
staff involved with each measure
Overview of BCFPI & CAFAS
May 18, 2006
Report
Final report of study
Item
Prep
Meetings
Data cleaning and
extraction
April 22 & 23, 2006
Appendix C
CAFAS Overview
The Child and Adolscent Functional
Assessment Scale (CAFAS):
An Overview
Jewish Family and Child Service
May 18, 2006
Adapted from CAFAS Manual and CAFAS in Ontario materials
What is CAFAS?
„
Clinician-rated measure
„
10 - 15 minutes
„
„
„
Measures degree of functional impairment of children and youth
(6 -17 years) with emotional, behavioural, psychological, and
substance use problems
Impairment - problems that interfere with child/youth’s
functioning in various life roles
CAFAS is rated (at minimum) at two different times:
„ Pre-treatment
„ Post-treatment: end of service OR at the end of 1 year.
12
What is CAFAS? (cont.)
„
Contains behavioural descriptors that cover 8 domains of
functioning:
„ School
„ Home
„ Community
„ Behaviour Towards Others
„ Moods/Emotions
„ Self-Harmful Behaviour
„ Substance Use
„ Thinking
What is CAFAS? (cont.)
„
Two additional scales that apply to the caregiver (Primary
Family, Non-custodial Caregiver, Surrogate Caregiver)
„
Material Needs
„
Family Social Support
13
What is CAFAS? (cont.)
For each scale, behaviours are grouped into 4 levels of
impairment:
„ Severe (30): marked impact, severe disruption to
functioning or incapacitation
„
„
„
„
Moderate (20): frequent difficulties with negative impact,
major or persistent disruption
Mild(10): significant difficulties or impacts, problems or
distress but no major dysfunction
Minimal/No impairment (0): no disruption of functioning
CAFAS Uses
„
Assessing level of client functioning
„
Planning treatment
„
Monitoring change over time
„
Monitoring program effectiveness
14
CAFAS Reports
15
Appendix D
BCFPI Overview
16
The Brief Child and Family Phone
Interview (BCFPI):
An Overview
Jewish Family and Child Service
May 18, 2006
Adapted from BCFPI materials
What is BCFPI?
„
Brief Child and Family Phone Interview for children from 6 to 18
„
“Structured” interview completed with parents or adolescents
„
Completed as part of the intake process
„
Takes approx. 30 min.
17
What is BCFPI?
„
„
„
„
„
„
„
„
„
Narrative of Basic Concerns
Mental Health
Child Functioning
Family Functioning
Abuse
Barriers to Service Utilization
Basic Demographic information
Risk & Protective Factors
Readiness for Service
18
MH: Internalizing Subscales
„
Separating from Parents (SP)
„
Managing Anxiety (MA)
„
Managing Mood (MM)
„
Self-Harm
„
Internalizing Composite
Child Functional Impact Subscales
„
Child’s Social Participation
„
Quality of the Child’s Social Relationships
„
School Participation and Achievement
„
Impact on Child Functioning Composite
19
Family Functional Impact Subscales
„
Family Activities
„
Family Comfort
„
Global Family Situation
BCFPI Scores
„
T-Scores
„
Standardized score (e.g., IQ or SAT scores)
„
How a particular individual stands relative to others who are the
same age and gender
„
Mean of 50 and a Standard Deviation of 10
„
T-score of 70 higher than 98% of the population - Clinical Range
„
T-score of 65 higher than 93% of the population - Borderline Range
20
BCFPI Uses
„
Screen/overview of presenting problems
„
Clinically relevant information on type and severity of problems
„
Organizational view of referrals
„
Assists in decision-making
„
CAUTION: not a diagnostic tool/with limitations
21
Appendix E
BCFPI Average Scores
22
Age Group (All) Gender (All) Informant Type (All) Stage (All) Form (All) Status (All) City (All) Postal/Zip Code (All) User Geo Code 1 (All)
Average BCFPI Scores (N = 184)
85
80
75
70
Agency
65
60
55
50
J.D.D. - Jerome D. Diamond Centre
RAIAp
Avg Scr
68.47
COp Avg CDp Avg
Scr
Scr
68.88
66.82
EXp Avg
Scr
71.62
SPp Avg MAp Avg MMp Avg SHp Avg
Scr
Scr
Scr
Scr
61.61
65.15
71.15
72.82
INp Avg
Scr
TMHP
Avg Scr
SocPartP
Avg Scr
QRelP
Avg Scr
69.90
73.52
78.58
69.86
Data
SchoolP ChFp Avg FActP
Avg Scr
Scr
Avg Scr
77.35
80.58
79.78
FcFp Avg GFsP Avg
Scr
Scr
78.22
83.11
Appendix F
BCFPI Problem Prevalence
Age Group (All) Gender (All) Informant Type (All) Stage (All) Form (All) Status (All) City (All) Postal/Zip Code (All) User Geo Code 1 (All)
% Problem Prevalence (N = 184)
100%
90%
80%
70%
60%
Agency
50%
40%
30%
20%
10%
0%
J.D.D. - Jerome D. Diamond Centre
RAIAp %
Prob
COp %
Prob
CDp %
Prob
EXp %
Prob
SPp %
Prob
MAp %
Prob
MMp %
Prob
SHp %
Prob
INp %
Prob
50.0%
49.5%
35.3%
58.7%
34.2%
37.0%
56.5%
57.1%
47.8%
Data
TMHP % SocPartP QRelP %
Prob
% Prob
Prob
63.0%
71.6%
55.2%
SchoolP
% Prob
ChFp %
Prob
FActP %
Prob
FcFp %
Prob
GFsP %
Prob
78.7%
84.2%
53.8%
74.3%
72.0%
Appendix G
BCFPI Income by Family Structure
Drop Page Fields Here
JDD BCFPI Income X Family Structure (N = 182)
% of Total
45%
40%
40%
35%
30%
Family Structure
25%
Partner or spouse
20%
Single parent
15%
10%
9%
8%
5%
2%
7%
2%
1%
7%
3%
7%
4% 3%
3%
3%
1% 1%
1%
1%
0%
1) $0-$9,999 2) $10,000$14,999
3) $15,000$19,999
4) $20,000$29,999
5) $30,000$39,999
Income
6) $40,000$49,999
7) $50,000$59,999
8) Greater
than $60K
(blank)
Appendix H
CAFAS Pre-post Test Scores for All and Completed Cases
JDD Average Pre-Post Test CAFAS Scores for
All (N = 45) and Cases Completing (N = 27) Cases
110.0
104.0
99.4
100.0
90.0
78.4
Total Scores
80.0
70.0
57.0
60.0
Pre Test
Post Test
50.0
40.0
30.0
20.0
10.0
0.0
All
Completed
Case
29