Author Affiliations
Jason R Williams
Casey Family Programs
Lisa Merkel-Holguin, Heather Allan
The Kempe Center for the Prevention and Treatment of Child Abuse and Neglect, University of
Colorado, School of Medicine, Department of Pediatrics
Erin J. Maher
Casey Family Programs
John Fluke, Dana Hollinshead
The Kempe Center for the Prevention and Treatment of Child Abuse and Neglect, University of
Colorado, School of Medicine, Department of Pediatrics
Keyword Suggestions (to be checked against MeSH)
implementation
family preservation
family support
child protective services
Bayesian Model Averaging
Family group conferencing
See specific formatting instructions. Documents will use APA, as well.
http://www.press.uchicago.edu/journals/jsswr/instruct.html?journal=jsswr

Abstracts should be written in present tense
Factors Associated with Staff Perceptions of the
Effectiveness of Family Group Conferences
Abstract
Family Group Conferencing (FGC) challenges the entrenched practice of child welfare agency
service providers making decisions for children and families. Instead, FGC position families as
capable planners and decision-makers given proper support and resources. Implementation
research in general highlights the importance of agency climate, worker characteristics, and
attitudes in predicting buy-in and ultimately the success of evidence-based programming. Prior
research has shown social work staff differ in their opinions of FGC, but little research has
connected these attitudes to staff and agency characteristics and, ultimately, to referrals and
outcomes. In the absence of literature describing agency and staff characteristics associated with
support of FGC in practice, we explore factors related to staff perceptions of effectiveness of
FGC using Bayesian Model Averaging with responses from N = 252 staff surveys. The results
imply that worker attitudes about the effectiveness of FGC are themselves a product of attitudes
towards families, type of work responsibility, and the perception of resources or services in the
external environment. Among those who carry a caseload, worker perceptions of FGC
effectiveness depend on perceptions of the availability and competence of local services to help
families in need. Among those who do not carry a caseload, endorsement of families’ abilities to
make plans to address their issues predicts endorsement of the effectiveness of FGC.
Introduction
Successful implementation of a program or intervention within an organization hinges in
large part on issues of ownership, efficacy, and discretion. Those charged with ground-level
implementation are more likely to do so with fidelity if they are personally invested in practices
that they believe will be effective. A program injected from outside will be treated like an
immune system treats a foreign body. Where there is room for interpretation and authority to
choose from among several possible options, these ground-level implementers become
essentially makers of policy, the results of which may be different than what was intended by
leadership (Lipsky 1980; Križ & Skivenes, 2014). In this paper, we address child welfare agency
staff attitudes and perceptions of family group conferences (FGCs) and the implications for
implementation. Specifically, we examine what predicts worker perceptions that FGCs are an
effective decision-making structure.
Family Group Conferencing in Child Welfare
Family group conferencing (FGC) is firmly rooted in New Zealand’s Children, Young
Persons, and their Families Act of 1989, which required participatory decision making between
child welfare systems and the family groups of children who are abused and/or neglected. This
reform acknowledged the marginalization and exclusion of family groups when their children
come to the attention of the state, honored indigenous and traditional ways of resolving concerns,
and recognized institutional racism within the public systems charged with the protection of
children and support of families. It emerged from a rights-based framework that all children are
entitled to having their family systems come together on their behalf to make decisions about
their well-being. The legislation deconstructed the prevailing, entrenched practice of child
welfare agency service providers making decisions for children and families. Instead, FGC
positions family groups to lead child welfare decision making, with the support of service
providers. It challenged the predominant construct of professionals as experts and required
systems and their professionals to view families more holistically and as solution builders and
leaders able to address challenges that confront their family systems (Ministerial Advisory
Committee, 1988).
FGC is among the most recognized models that fit under the family group decision
making (FGDM) framework. In 2008, five core components of FGDM were defined by experts,
with a sixth component related to follow-up added in 2013 (Kempe Center for the Prevention
and Treatment of Child Abuse and Neglect, 2013, p. 2). These components are:
1. An independent coordinator is responsible for convening the family group meeting with
agency personnel.
2. Child protection agency personnel recognize the family group as the key decision-making
partners and offer time and resources to convene the family group.
3. After initial presentations, family groups meet on their own, without statutory authorities
(e.g., CPS workers), to work through information they have been given and formulate
responses and plans.
4. When agency concerns are adequately addressed, preference is given to the family
group’s plan over any other possible plan.
5. Follow-up processes after the FGDM meeting occur until the intended outcomes are
achieved, to ensure that the plan continues to be relevant, current, and achievable. FGDM
is not a one-time event but an ongoing, active process.
6. Referring agencies support family groups by providing the necessary services and
resources to implement the plan.
Given that FGC is enshrined in legislation and policy in New Zealand, the influence of
service providers’ perceptions about FGC have not been a primary line of inquiry in that country.
That is not the case in other countries, like the United States, where FGC may be considered a
relatively new practice, and its implementation is often at the discretion of frontline social
workers to refer families to participate. The organizational bureaucracy, the procedural nature of
child protection work, and the values individual systems place on these types of decision-making
forums may further hamper or aid successful implementation (Brown, 2003).
Independent of the extent to which policy supports or directs the occurrence of family
group conferences, understanding the attitudes of the child welfare workforce (including
administrators, supervisors, and case workers) is a critical component to evaluating their use and,
ultimately, effectiveness in child welfare. In their influential review of implementation research,
Fixsen and colleagues (Fixsen, Naoom, Blase, Friedman, & Wallace, 2005) pointed out that
practitioners’ skills, attitudes, training, and coaching are keys to successful implementation of
evidence-based practices, though they noted that workforce issues have been understudied.
McCrae and colleagues (McCrae, Scannapieco, Leake, Potter, & Menefee, 2014) found agency
size, stress levels, worker tenure, and job position to be related to buy-in for implementation of a
child welfare practice model. There remains a dearth of literature examining child welfare
professionals’ attitudes and perspectives about FGC and other family meeting models. We
briefly review these studies here.
Some studies have suggested that the child welfare workforce possesses a generally
positive attitude about FGCs, particularly the underlying empowerment and strengths-based
philosophies of the model (Huntsman, 2006; Crow and Marsh, 1997; LeCroy and Milligan
2003). Even so, child welfare staff members have raised a number of concerns about FGCs,
including: skepticism of plans formulated at FGCs compared to other child welfare meetings
(Trotter, 1999), meeting length (Velen and Devine 2005), and excess family empowerment
(Huntsman, 2006). These perspectives may reflect ambiguity around the workforce’s attitudes
toward the core purpose of FGC, which is for the child welfare agency to facilitate and allow for
a process for family groups to make decisions. None of these studies connected these perceptions
to worker characteristics or to outputs or outcomes.
One study is particularly germane to this line of inquiry. In Sweden and the UK, Sundell,
Vinnerljung, and Ryburn (2001) studied the relationship between social worker attitudes toward
FGCs and referrals to the decision making process. In their survey of social work professionals
with limited experience with FGCs, a majority endorsed the need for private family time, the
capacity of extended families to make decisions, and the usefulness of FGC as a mechanism to
resolve child maltreatment concerns. Despite this, the rate of referrals to FGCs was generally low
in both countries. They also found that rates of referrals were higher when social workers: 1)
participated in developing FGC as a service in Sweden but not in the UK; and 2) had a more
positive attitude about FGCs. The age and previous work experience of the social worker did not
impact referral rates to FGCs.
As part of a broader examination of FGC implementation, this study focused on
understanding child welfare professionals’ perspectives about the effectiveness of FGC. These
perceptions are important to study since professionals’ buy-in impacts referrals (Sundell et al.,
2001) and other aspects of implementation. Further, quality implementation with a high degree
of family participation is an important ingredient in FGC practice and one which is hypothesized
to result in the achievement of desired outcomes. While this broader understanding of the role of
workforce attributes on FGC practice and outcomes is a longer-term goal, our purpose with this
exploratory analysis was to understand whether attributes and attitudes measured at the worker
level influence perceptions about the efficacy of FGC. We view this as a necessary precursor for
understanding the broader impact on implementation and, ultimately, outcomes.
Model Development Under Uncertainty
While child welfare staff members’ perceptions of the effectiveness of FGC are likely to
affect fidelity and success of implementation, few factors or combinations of factors have been
identified in the literature as associated with such perceptions. Thus, data collection and analysis
for this project began with few a priori hypotheses about individual factors and no hypothesis
about the best combination of factors to examine. This uncertainty about model composition or
functional form is common in research examining new practices and has implications for the
selection of an analytic approach.
Traditional inferential statistics is predicated on testing a single model against a null
model. In practice, however, even when authors present a single model, it is likely that multiple
models were actually considered. Several combinations of control variables may have been
considered through some formal or informal screening process (Raftery, 1995; Tobias & Li,
2004). Similarly, alternate ways of measuring constructs or capturing, for example, nonlinear
effects may be considered. The more such uncertainty enters into model building, the greater the
number of potential models to evaluate. Existing screening and selection methods that rely on pvalues or R2 or other goodness of fit statistics face numerous problems. These procedures do not
inherently account for the multiple comparisons necessary to generate the final model, and the
resulting “p-values based on a model selected from among a large set of possibilities no longer
have the same interpretation as they did when only two models were ever considered” (Raftery,
1995, p. 112). Standard errors or confidence intervals resulting from these procedures are
downwardly biased because they do not account for the uncertainty in model selection itself.
Given a relatively undeveloped literature from which to develop hypotheses on this topic
and the faults associated with traditional testing of multiple models, we employed Bayesian
Model Averaging (BMA) as our tool to weigh the evidence for each predictor and potential
model. Bayesian methods focus on which model fits the data best, and express all uncertainty—
including about the unknown parameters of interest—in terms of probability. We employed the
BMA methodology of Raftery and colleagues (Raftery, 1995; Hoeting, Madigan, Raftery, &
Volinksy, 1999) as implemented in the package BMA (Raftery, Hoeting, Volinsky, Painter, &
Yeung, 2013) for the statistical program R (R Core Team, 2014). This procedure allows for
competition even among highly correlated variables and facilitates the examination of different
operational definitions of the same construct.
BMA produces a model posterior probability, which is the likelihood that the candidate
model is the “correct” model that produced the data. It also produces inclusion probabilities that
capture the likelihood that each predictor is in the correct model—essentially, a measure of
predictor strength or the evidence supporting the inclusion or exclusion of that predictor
(Hoeting et al., 1999). The “Averaging” part of the name refers to the production of an expected
value of the coefficient weighted by the model posterior probability across all models
considered. A probability weighted standard deviation is calculated similarly, and is analogous to
the usual coefficient standard error except that it captures all uncertainty about the coefficient.
While users can report and act upon the expected values for all considered predictors, BMA is
often used to select a subset of predictors because the inclusion probabilities for some variables
will be quite low, providing moderate or strong evidence against there being an effect. Many
analysts select either the modal model (variables with inclusion probabilities above 0.50) or the
best model (highest posterior probability), which are often one and the same.
In this paper, we utilized BMA within an exploratory model building and testing effort,
combining the tool with more traditional inferential approaches as recommended by Gelman
(2011). We applied this exploratory approach to an on-going child welfare evaluation project,
identifying available variables potentially related to our outcome of interest—staff perceptions of
the effectiveness of FGCs. The data source and potential predictors considered are described
below.
Methods
Data Source
Data for these analyses were collected from child welfare agency staff in three
jurisdictions west of the Mississippi participating in a three-year evaluation of the use of family
meetings in child welfare (referred to as States 1, 2, and 3). These sites were chosen due to their
established family meeting practice; the three agencies had been implementing FGCs and other
family meetings for a minimum of six years and a maximum of 14 years. The survey was
administered as part of an evaluation training that informed staff of the evaluation goals, design,
and their roles in it, specific to their job. The on-line survey contained questions about staff
member demographic characteristics, position, and employment experience and scales pertaining
to case skills, family meeting knowledge and attitudes, organizational culture and climate,
service availability, and child safety vs. family preservation proclivity. In addition, as new staff
connected to the project joined the agency, they were trained on the evaluation and completed
the electronic survey at that time. Completing the survey took approximately 15 minutes.
Approval for this study was received from a western state’s Institutional Review Board.
Sample
The staff survey sampling frame consisted of all staff members who may have a role in
the family meeting evaluation or who may supervise direct service staff. Any staff member who
may have referred families to the study or participated in a family meeting was considered to
have a role in the evaluation. The bulk of survey data was collected in the Fall of 2012 during the
first wave of evaluation trainings at each site, with the remaining data collected via subsequent
webinar trainings for new staff through the Spring of 2014. Staff surveyed included caseworkers,
FGDM facilitators/coordinators, supervisors of both, and some upper level managers and
administrators. While there were 301 survey respondents overall, our analytical sample for the
BMA below was smaller (N = 252) due to missing responses across all variables considered.
Measures
Dependent Variable
The dependent variable was a composite of 10 items ranking FGC effectiveness for
addressing different family problems. The 10 family issues were drug abuse, alcohol abuse,
domestic/intimate partner violence, extreme poverty, extreme child behavior problems, mental
illness, developmental disability, extremely poor parenting skills, educational neglect, and
parent-child conflict. Respondents could select answers from a five-point scale that ranged from
not at all effective to completely effective. A not applicable option was also available to account
for staff who might not have much direct familiarity with FGC. A review of the distribution of
responses across all items revealed that most respondents (73 to 79%) endorsed moderately
effective or very effective. Therefore, we created a dichotomized composite dependent variable to
reflect ratings of at least very effective (i.e. rated a 4 or 5) on all items rated (i.e. excluding items
for which the respondent selected not applicable).
Predictors Examined for Inclusion
Table 1 describes the variables from the survey we tested in the modeling. For each item
or scale, we explored variation in response and, in some cases, covariation with the outcome in
making decisions about operationally defining the variable or creating composites to economize
on degrees of freedom. (Descriptive statistics appear in Table 2 below).
Table 1: Potential predictors tested.
Construct
Variables Composing Construct
Primary job
Position
Current work area FGDM
Caseload carried
Years in position
Years in child welfare
Experience with family meetings
Experience
Minority
Ethnicity
High job satisfaction
Job satisfaction
Family preservation proclivity
Worker attitude
toward families
Perceived changes
in workload due to
family meetings
Family ability to plan
Workload change due to family
meetings
Services composite
Local services for
families
Can find services
Confidence in services for needs
Organizational
climate and
culture
Supervisor competence
Definition
Three-category variable indicating
whether a worker is a supervisor,
program director or trainer; a
coordinator or facilitator; or a case
worker (reference category)
Dichotomous variable indicating
whether a staff person currently
works in a position with FGDM
responsibilities
Dichotomous variable indicating
whether the respondent reported a
caseload > 0
Years in current position
Years of experience in child welfare
Three-category variable
differentiating those who report a
lot, some, or little to no experience
with family meetings (reference
category, combining a little and
none)
Dichotomous variable indicating selfreport of being any ethnicity besides
white, non-Hispanic
Dichotomous variable capturing
those reporting they are very or
completely satisfied with their job
Dalgliesh (2010) scale produces a
single continuous variable reporting
preference for family preservation
vs. child safety
Dichotomous variable indicating that
the respondent agreed or strongly
agreed that families know how to
construct thorough plans for
resolving their issues
Three-category variable indicating a
reported decrease or increase in
workload or no change (reference
category)
Single scale assessing positive
feelings about services that
combines all service ratings
Dichotomous variable indicating the
respondent agreed or strongly
agreed
Mean confidence rating across 25
areas of need
Mean of supervisor skill ratings
(Butler Institute for Families, 2009;
New Mexico Children, Youth, &
Families Department [NMCYFD],
Leadership
Vision, professionalism, and
commitment
State child welfare
agency
State
2011; New York State Social Work
Education Consortium, 2005)
Mean shared/effective leadership
rating (Butler Institute for Families,
2009; NMCYFD, 2011)
Mean perception of coworkers’
shared vision, professionalism, and
commitment (Ellett, 2009)
Three-category indicator of working
in State 1 (reference category), State
2, or State 3
The staff survey included several questions assessing respondent perceptions of the local
service mixture, including ease of finding appropriate services, ease of working with service
providers, and cultural responsiveness of providers. With no clear way to combine what were
highly overlapping items, we tested and compared three different ways of assessing the quality
of the local service mixture. One combined the items capturing confidence in 25 specific local
services to meet individual family needs (e.g., early childhood services, legal services) into a
single scale by meaning the five-point scale item responses that ranged from not at all confident
to completely confident. The second measure was a single item capturing agreement that “I can
usually find services in my community that can help keep children safe in their home,”
dichotomized into agree or strongly agree versus not. The services composite variable combined
these two formulations plus the ratings of ease of working with service providers in the
community and the responsiveness of local services in meeting needs of culturally diverse
groups into a single scale (scored as a mean proportion among items completed) with scores
closer to 1 indicating more enthusiastic endorsement of local services.
Results
Descriptive Statistics
Descriptive statistics for the outcome variable and candidate predictors are presented in
Table 2. Missing data were not a substantial problem for this study. For individual items the
amount of missing data ranged from 1% to 4% of the total sample. Given the low rates of
missingness, data were presumed to be missing at random (MAR). Combining BMA and
multiple imputation is beyond the scope of this paper.
Table 2: Descriptive statistics for outcome and potential predictors
FGC at least very effective in all areas rated (0 – no, 1 –
yes)
Primary job:
Case worker (0 – no, 1 – yes)
Supervisor/ Program Director/trainer (0 – no, 1 - yes)
Coordinator/ facilitator (0 – no, 1 – yes)
Current work area FGDM (0 – no, 1 – yes)
Caseload carried (0 – no, 1 – yes)
Years in position
Years in child welfare
Experience with family meetings:
Little to none (0 – no, 1 – yes)
Some (0 – no, 1 – yes)
A lot (0 – no, 1 – yes)
Minority (0 – no, 1 – yes)
High job satisfaction (0 – no, 1 – yes)
Family preservation proclivity
Family ability to plan (0 – disagree, 1 – agree)
Workload change due to family meetings:
No change (0 – no, 1 – yes)
Decrease (0 – no, 1 – yes)
Increase (0 – no, 1 – yes)
Services composite
Can find services (0 – disagree, 1 – agree)
Confidence in services for needs
Supervisor competence
Leadership
Vision, professionalism, and commitment
Mean
0.313
SD
0.465
Min
0
Max
1
Missing
0
0.698
0.168
0.134
0.164
0.683
2.832
7.103
0.460
0.375
0.341
0.371
0.466
3.329
7.320
0
0
0
0
0
0
0
1
1
1
1
1
23
31
0
0
0
0
0
0
0
0.202
0.267
0.531
0.532
0.511
0.413
0.828
0.402
0.443
0.500
0.500
0.501
0.241
0.378
0
0
0
0
0
0
0
1
1
1
1
1
1
1
0
0
0
10
0
5
0
0.508
0.164
0.328
0.705
0.612
3.326
5.055
4.668
4.913
0.501
0.371
0.470
0.111
0.488
0.718
0.888
0.901
0.701
0
0
0
0.283
0
1.440
1.750
1.467
2.167
1
1
1
1
1
5
6
6
6
0
0
0
4
4
4
2
2
2
Mean
SD
Min
Max
Missing
0.460
0.272
0.487
0
0
0
1
1
1
0
0
0
State:
State 1 0.302
State 2 0.080
State 3 0.618
Note: SD is standard deviation, Min is minimum, and Max is maximum. N = 252.
Modeling Results
We used the BMA package function bic.glm (Raftery et al., 2013), which applies
BMA to generalized linear models, specifying a logistic regression model. The resulting
coefficient estimates reflect the estimated effect on the log odds of agreeing that FGC is very or
completely effective in all areas assessed. The four best models from the initial stage of our
analysis are presented in Table 3. The results reflect much uncertainty as to the correct model.
Raftery (1995) cites posterior odds of 20:1 as reflecting strong evidence, implying an inclusion
probability above 95% (with rounding) provides strong evidence of an effect, while a probability
below 5% implies strong evidence against an effect. While many of our variable inclusion
probabilities strongly rule out potential effects, a number are between 5% and 95%. The highest
inclusion probability—p(β≠0)—is for carrying a caseload, with a 61.7% chance of being in the
“correct” model, which represents positive but not strong evidence (Hoeting et al., 1999).
Table 3: BMA results, with 4 best models
p(β≠0)
EV
SD
Supervisor/PD/trainer
0.003
0.053
Coordinator/facilitator
Current work area FGDM
Caseload carried
Years in position
Years in child welfare
Experience with family
0.005
0.270
-0.516
0.000
0.000
0.079
0.470
0.474
0.006
0.003
Primary job:
model 1
model 2
model 3
model 4
-0.892
-0.840
-0.810
-0.915
0.5
29.8
61.7
1.2
1.1
0.0
p(β≠0)
EV
SD
39.3
1.0
1.7
1.0
0.000
0.000
0.263
0.000
0.009
0.001
0
0
0.378
0.033
0.107
0.041
42.2
17.8
0.224
-0.035
1.860
0.126
0.478
0.176
2.401
0.312
52.0
0.328
0.356
2.9
1.0
0.006
0.000
0.045
0.018
1.7
0.003
0.036
0.000
0.000
0
0
model 1
model 2
model 3
model 4
meetings:
Some
A lot
Minority
High job satisfaction
Family preservation proclivity
Family ability to plan
Workload change due to
family meetings:
Decrease
Increase
Services composite
Can find services
Confidence in services for
needs
Supervisor competence
Leadership
Vision, professionalism, and
commitment
State:
State 2
State 3
BIC
Posterior probability
0.733
0.606
20.5
4.646
0.692
4.533
0.649
0.0
-1121.10 -1121.07 -1120.32 -1119.78
8.5
8.4
5.8
4.4
The inclusion probabilities for the service ratings need to be interpreted in the context of
the high interrelationships among the variables. Examining the pattern of inclusion across all 72
averaged models, a measure of services was absent in only two highly unlikely models. While
not entirely mutually exclusive (they appear together in three highly unlikely models), the
inclusion probabilities for the services composite and the confidence in services across family
needs together indicated relatively strong evidence that one of them should be in the model. In
short, staff considerations of services appeared to affect perceptions that FGC is at least very
effective in addressing family needs.
The weak evidence for the inclusion of the other predictors tested plus the uncertainty
around which measure of services is best resulted in little to distinguish a best model. The
nominal best model had a posterior probability of only 8.5%. Furthermore, the difference in
Bayesian Information Criterion (BIC) values between the best and fourth best models was less
than 2, implying weak evidence in favor of the former (Raftery, 1995) and against the inclusion
of minority status (the only difference between the models).
While primary job was not related to ratings of FGC effectiveness, there was some
evidence that the similar variable of whether or not a worker carries a caseload was a likely
predictor. Carrying a caseload or not implies two very different groups of workers, and some of
the weakness in evidence for predictors might be due to different correlates of effectiveness
ratings for those who regularly work with children and families versus those who do not. To
explore whether the pattern of overall results might be different depending on whether workers
carried a caseload and whether this improved modeling results, we split the sample on the
caseload carrying dichotomy and ran separate BMA analyses on each. The results, not shown,
suggested that carrying a caseload may, in fact, be an important moderating variable in
predicting ratings of effectiveness.
Thus, we created four interaction terms to capture differential effects of four variables
that had very different inclusion probabilities for those with and without a caseload. Specifically,
Caseload×Supervisor and Caseload×Services captured the additional effect of higher supervisor
competence and service competence and availability ratings, respectively, among those carrying
a caseload. Caseload×Minority captured a higher or lower level of agreement that FGCs are
effective among the 35.2% of the sample who were both a member of a minority ethnic group
and carried a caseload. Caseload×Family captured a different intercept for the 55.7% who carried
a caseload and agreed that families are capable of making thorough plans for addressing their
issues. We added these terms to the prior potential predictors and submitted them to BMA.
Again, multiple overlapping potential predictors created much uncertainty in model
selection. As seen in Table 4, no predictor had an inclusion probability above 50%. The best
model had a posterior probability twice that of the second and third best models, but accounted
for only 7.9% of the total likelihood among the 80 averaged models, and the BIC difference
indicated weak preference over any of models 2 through 6 (not shown).
Table 4: BMA results with caseload interactions, with 4 best models
p(β≠0) EV
SD
model 1
Primary job
0.0
Current work area FGDM
13.2
0.113
Caseload carried
24.9
-0.316 0.844
Years in position
0.0
0.000
0
Years in child welfare
0.0
0.000
0
Experience with family meetings
0.0
Minority
39.3
0.326
0.482
High job satisfaction
0.0
0.000
0
Family preservation proclivity
0.0
0.000
0
Family ability to plan
17.1
0.574
1.503 4.546
Workload change due to family
meetings:
20.0
0.218
0.473
Decrease
Increase
model 2
model 3
-0.892
-0.840
model 4
0.326
0.733
0.721
4.646
4.992
-0.033 0.176
Services composite
42.4
1.993
2.520
Can find services
9.0
0.064
0.227
Confidence in services for needs
40.4
0.267
0.354
Supervisor competence
2.4
0.009
0.073
Leadership
0.0
0.000
0
Vision, professionalism, and
commitment
State:
0.0
0.000
0
0.8
0.692
p(β≠0) EV
SD
State 2
0.008
State 3
-0.003 0.056
model 1
model 2
model 3
model 4
0.103
Caseload×Supervisor
10.5
-0.019 0.073
Caseload×Minority
7.8
-0.075 0.301
Caseload×Services
25.8
0.686
Caseload×Family
35.2
-0.786 1.639 -5.013
2.078 5.428
-0.799
BIC
-1122.51 -1121.10 -1121.07 -1120.80
Posterior probability
7.9
3.9
3.9
3.4
The best model included an effect for agreeing that families know how to make thorough
plans to resolve with their issues, the interaction of this variable with caseload carrying status,
and an effect for service ratings only among those carrying a caseload. Put another way, we had
one effect for those who do not carry a caseload, and two apparent effects for those who do.
Together, these three predictors created a model that correctly classified 69.4% of the sample as
agreeing or disagreeing with the effectiveness of FGC. The resulting logistic regression model
(conducted on all cases with complete data for the four variables involved) had a likelihood ratio
test statistic of G2 = 30.674, p < 0.001.
To aid in interpretation of these results, we present the results of this final model
graphically. We simulated the predicted probabilities and confidence intervals accounting for
uncertainty in the model parameters using the R package simcf, utilizing 10,000 draws from
the multivariate normal space defined by the variance-covariance matrix of the resulting
parameters. We conducted this simulation separately from the BMA analysis. The resulting 95%
confidence intervals in Figure 1 reflect uncertainty in the parameters within the chosen model,
but not the uncertainty in choosing this model over others, and thus are biased downwards
(liberally). Figure 1 presents the predicted probability of endorsing effectiveness of FGC
(vertical axis) as a function of the variables in the best model selected above, with the horizontal
axis capturing responses on the services composite variable (better feelings about local services
being closer to 1).
Figure 1: Predicted probabilities of agreeing that FGC is effective in all areas
In summary, results of our model results suggested that among those who did not carry a
caseload, agreeing that families are able to make plans made a large difference in the probability
that the respondent would endorse FGC as effective in all areas. The relatively few respondents
who did not carry a caseload and were more pessimistic about families’ abilities had near 0
predicted probability of endorsing FGC as effective, while their colleagues who were positive
about family ability to construct plans had a 50% probability of endorsing FGC as effective,
compared to a base rate of 31%. In fact, the only group predicted to have a higher probability of
agreeing that FGC is effective—but not significantly so, as indicated by the confidence
intervals—was those who carried a caseload, disagreed about family ability to plan, and had the
most positive regard for local services to aid those families. However, for those with a caseload,
the confidence intervals indicated that agreeing about family ability to construct thorough plans
made little to no difference in the likelihood of agreeing that FGC is effective, and thus may not
be a factor for the case-carrying group.
Judging FGCs to be effective is, of course, related to multiple factors, some of which are
difficult to capture in a brief survey. Nonetheless, three general factors were the strongest
correlates of whether the staff members in this analysis agreed that FGC is effective: Their
reported beliefs about local service competence and availability, their reported beliefs about
families’ capability of constructing plans to resolve child abuse and neglect concerns, and
whether they carried a caseload. The effect of the latter appeared to operate through service
ratings: Those who worked directly with families held FGC in high regard if they held local
services in high regard. Among those who did not work with families every day, feelings about
FGC appeared to depend upon their beliefs about family ability to make plans. Lowest regard for
FGC appeared to be among those who did not carry a caseload and were pessimistic about the
ability of families to plan, and among those who carried a caseload but were pessimistic about
local services to help families.
Discussion
Social workers often serve as the ground-level gatekeepers to services, despite any
policies or mandates on the books. For this reason, gaining a better understanding about child
welfare professionals’ perspectives about FGC effectiveness may help child welfare agency
leaders (administrators, managers, and supervisors) establish protocols and policies that more
fairly disperse the service, coach their workforce on the purpose of the FGC, and remove barriers
to the effective implementation of FGCs.
From a rights-based perspective, the undergirding philosophy of FGCs is that all families
are entitled to the opportunity to be decision makers when children in their family group are
vulnerable. As such, agency staff’s perceptions of the family’s decision-making abilities and
capacities should not be a consideration when making a referral. In addition, given that plans that
emerge from FGCs tend to blend a combination of community, family, formal, and informal
resources (Marsh and Walsh, 1997; Burford and Pennell, 1998; Morris, 2007), the perception of
service competence and availability as an important predictor of FGC effectiveness may reflect
traditional thinking about the efficacy of child welfare interventions. Staff perceptions of formal
services as ineffective or unavailable, while possibly overlooking and undervaluing other family
and community resources that may be leveraged, present an opportunity for those guiding
implementation. Given these findings, it may be helpful to highlight the notion that caseworkers
are unlikely, at least initially, to fully understand the informal resources a family group can
access or how the family group will align a range of resources to meet needs during the FGC
process.
However, the rights-based framework is not the framework employed by the majority of
jurisdictions in the United States, nor elsewhere in the world, despite the growing value placed
on family engagement across child- and family-serving agencies. At this stage, family meeting
models, even those with empirical support, remain a new and sometimes foreign approach,
subject to the discretion of those charged with implementation, particularly those who make
referrals. As a result, a greater understanding of staff attitudes around FGC and how attitudes
might impact referrals and quality implementation is essential for understanding how FGC
affects outcomes for the children, youth, and families receiving them. Implementation research
clearly points to addressing staff characteristics and attitudes as key to successful implementation
of evidence-based practice (e.g., Fixsen et al., 2005; McCrae et al., 2014).
Additional research will examine whether and how staff attitudes influence referrals,
fidelity to the model, and outcomes. This analysis is a first step in understanding what does and
does not influence worker perceptions of the effectiveness of FGCs. Still, if FGCs truly are a
beneficial service and the goal is to maximize their effectiveness by increasing capacity to offer
them to the maximum number of families, then removing barriers to referral should be a goal.
Understanding what worker characteristics facilitate or impede FGC referrals, as understood
through staff perceptions of the practice, is significant to this goal. If judgments of effectiveness
do indeed indicate whether those with a caseload would refer families to FGCs, this research
indicates that those least likely to refer are those with the lowest opinion of the availability and
ability of local services to help those families. Whether these low opinions of services arise from
unavailable services, providers who have proven difficult to work with, or a mix of local
providers with poor results, addressing the service offerings available may be one ingredient to
increasing referrals and improving effectiveness. In addition, child welfare agencies
implementing FGC may benefit from re-tooling all staff on the core philosophies undergirding
FGC as they challenge the entrenched decision making structures.
This paper also demonstrates the use of BMA in an area of research that is limited in its
ability to test hypotheses based on theory or prior findings. BMA is emerging as a statistical
approach in many fields, with limited application in child welfare research to date. It provides a
different lens for analyzing relationships and the resulting models illuminate some relationships
that we may not have uncovered otherwise. The significance of carrying a caseload in
perceptions of effectiveness and the finding that assessments of service quality and availability
appear to affect perceptions of effectiveness provide important touchstones for both
implementers and researchers investigating other aspects of worker attitudes, referrals, fidelity,
or actual child welfare outcomes.
Limitations
While these results shed light on correlates of attitudes towards family group
conferencing, several factors may limit the generalizability of these results. As with other
surveys of perceptions, social desirability and other subjectivity issues may have created
measurement error, attenuating correlations. Lack of variability from ceiling effects and possible
social desirability led us to make certain choices regarding dichotomization of results. These
choices were part of an initial culling of degrees of freedom, for which other analysts might have
made different decisions. Similarly, the choice to dichotomize the outcome was informed, in
part, by a recognition that ordinal regressions require larger samples than binary regressions.
Having more than 252 surveys might have allowed testing an ordered outcome and resulted in
finding a different set of covariates. Item nonresponse reduced the sample size somewhat, and
we have operated here under an assumption of missing at random. One factor that does appear to
be related to item missingness is simple fatigue: The FGC ratings are less than halfway into a
survey that is 8 ½ pages (if printed), and all items with more than one missing value were asked
later in the survey. Finally, the three child welfare jurisdictions represented in our sample may
not be representative of other child welfare agencies, though we did not find that site had an
influence on perceptions of effectiveness in our models.
Despite these limitations, the study examines an understudied but important area of
implementation in jurisdictions that have been operating FGCs for some time. The results
illustrate that worker attitudes about FGCs and their effectiveness are themselves a product of
attitudes towards families, type of work responsibility, and the perception of resources or
services in the external environment. Future investigations that seek to understand the perceived
impact of practices from the worker perspective should consider including these concepts.
Worker characteristics and attitudes, while understudied, likely play a significant role in
determining what services families receive and the quality of those services, and we expect, their
outcomes.
References
Adolph, C. (2013). simcf: Counterfactuals and confidence intervals for estimated models. R
package version 0.2.10. http://faculty.washington.edu/cadolph/index.php?page=60
Burford, G., & Pennell, J. (1998). Family group decision making: After the conference —
progress in resolving violence and promoting well-being: Outcome report (Vol I & II). St.
John's, NF: Memorial University of Newfoundland, School of Social Work. Accessed 28
March 2014 from http://faculty.chass.ncsu.edu/pennell/fgdm/OutcomeReport/index.html
Brown, L. (2007). The Adoption and Implementation of a Service Innovation in a Social Work
Setting – a Case Study of Family Group Conferencing in the UK. Social Policy and Society,
6(3), 321-332.
Brown, L. (2003). Mainstream or margin? The current use of family group conferences in child
welfare practice in the UK. Child and Family Social Work, 8, 331-340.
Butler Institute for Families. (2009). The Western Regional Recruitment & Retention Project:
Final report. Denver: Author.
Crow, G. & Marsh, P. (1997). Family group conferences, partnership and child welfare: A
research report on four pilot projects in England and Wales. Sheffield: University of
Sheffield.
Dalgliesh, L. (2010). Balance of work focus in child welfare. Unpublished manuscript.
DePanfilis, D. (2006). Child neglect: A guide for prevention, assessment, and intervention.
Washington DC: US Department of Health and Human Services Office on Child Abuse and
Neglect.
Doolan, M. (2007). Duty calls: The response of law, policy and practice to participation rights in
child welfare systems. Protecting Children, 22(1), 10-18.
Eigen Kracht. (no date). Eigen Kracht Centrale: What we do (web page). Retrieved August 15,
2014 from http://www.eigen-kracht.nl/en/inhoud/what-we-do
Ellett, A. (2009). Intentions to remain employed in child welfare: The role of human caring, selfefficacy beliefs, and professional organizational culture. Children and Youth Services
Review, 31(1), 78-88. [doi: 10.1016/j.childyouth.2008.07.002].
Gelman, A. (2011). Induction and deduction in Bayesian data analysis. Rationality, Markets and
Morals, 2, 67–78.
Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M. & Wallace, F. (2005).
Implementation Research: A Synthesis of the Literature. Tampa, FL: University of South
Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation
Research Network (FMHI Publication #231).
Hoeting, J.A., Madigan, D., Raftery, A.E., & Volinksy, C.T. (1999). Bayesian Model Averaging:
A tutorial. Statistical Science, 14, 382-417.
Huntsman, L. (2006). Family group conferencing in a child welfare context. New South Wales,
Australia: Centre for Parenting & Research.
Kempe Center for the Prevention and Treatment of Child Abuse and Neglect. (2013). Family
group decision making in child welfare: Purpose, values and processes. Version 2. Retrieved
28 March 2014 from
http://www.ucdenver.edu/academics/colleges/medicalschool/departments/pediatrics/subs/can
/FGDM/Documents/FGDM%20Web%20Pages/About%20FGDM/FGDM%20Purpose%20V
alues%20and%20Processes.pdf
Križ, K., & Skivenes, M. (2014). Street-level policy aims of child welfare workers in England,
Norway and the United States: An exploratory study. Children and Youth Services Review,
40, 71-78.
LeCroy and Milligan Associates, Inc. (2003). Family group decision making: Annual evaluation
report. Tucson, AZ: LeCroy & Milligan Associates, Inc.
Lipsky, M. (1980). Street-level bureaucracy: Dilemmas of the individual in public services. New
York: Sage.
Marsh, P., & Walsh, D. (2007, January). Outcomes of family group conferences: More than just
the plan? Retrieved December 9, 2008, from
http://www.kent.gov.uk/NR/rdonlyres/9C8F2763-5B80-4F00-B9A2602DE5AF1DAE/11750/fgcresearchreport.pdf
McCrae, J. S., Scannapieco, M., Leake, R., Potter, C. C., & Menefee, D. (2014). Who's on
board? Child welfare worker reports of buy-in and readiness for organizational change.
Children & Youth Services Review, 37, 28-35. doi:10.1016/j.childyouth.2013.12.001
Ministerial Advisory Committee on a Maori Perspective for the Department of Social Welfare.
(1988). Puao-te-ata-tu (day break). Wellington, New Zealand: Department of Social
Welfare.
Morris, K. (2007). An evaluation of outcomes: Camden family group conference service. LB
Camden.
New Mexico Children, Youth, & Families Department. (2011). New Mexico CYFD – Protective
Services organizational health assessment: Report for protective services staff. Santa Fe:
Author.
New York State Social Work Education Consortium. (2005). Workforce Retention Survey.
Albany: State University of New York, Albany.
R Core Team (2014). R: A language and environment for statistical computing. Vienna, Austria:
R Foundation for Statistical Computing, http://www.R-project.org/.
Raftery, A.E. (1995). Bayesian Model Averaging in social research. Sociological Methodology,
25, 111-163.
Raftery, A., Hoeting, J., Volinsky, C., Painter, I., & Yeung, K.Y. (2013). BMA: Bayesian Model
Averaging. R package version 3.16.2.3. http://CRAN.R-project.org/package=BMA
Sundell, K., Vinnerljung, B., & Ryburn, M. (2001). Social workers’ attitudes towards family
group conferences in Sweden and the UK. Child and Family Social Work, 6, 327-336.
Tobias, J.L., & Li, M. (2004). Returns to schooling and Bayesian Model Averaging: A union of
two literatures. Journal of Economic Surveys, 18, 153-180.
Trotter, C., Sheehan, R., Liddell, M., Strong, D. & Laragy, C. (1999). Evaluation of the
statewide implementation of family group conferencing. Department of Human Services,
Government of Victoria, Melbourne.
Velen, M., & Devine, L. (2005). Use of FGDM with children in care the longest: It’s about time.
Protecting Children, 19 (4), 25-34.