DEVELOPING A PLAN FOR MEASURING OUTCOMES IN MODEL
SYSTEMS OF CARE FOR AMERICAN INDIAN AND ALASKA NATIVE
CHILDREN AND YOUTH
Douglas K. Novins, M.D., Michele King, and Linda Son Stone
Abstract: The Circles of Care initiative emphasized the
importance of developing an outcomes measurement plan
that was consonant with the model system of care as well
as community values and priorities. This analysis suggests
that the Circles of Care grantees achieved this key
programmatic objective, but that a major constraint was the
tendency of funders, including the Substance Abuse and
Mental Health Services Administration (the funder of Circles
of Care), to mandate their own outcomes measurement
plans. Funders are encouraged to balance their needs for
commonality of measures across programs for their own
evaluation purposes with the needs of service providers to
utilize measures that meet their unique programmatic and
community contexts.
Mental health services are expected to demonstrate positive outcomes
for the children, adolescents, families, and communities they serve (Nixon &
Northrup, 1997). Indeed, the importance of demonstrating such positive
outcomes for programs serving American Indian and Alaska Native (AI/AN)
communities was emphasized by many of the parent and community
participants in the Circles of Care (CoC) planning process, who advocated
for services that improved the mental health of their children and adolescents.
However, parents, community members, program staff, and
evaluators raised the following key questions throughout the CoC planning
process: (a) what constitutes a positive outcome for AI/AN children,
adolescents, and their families; and (b) how would these outcomes be
measured? Underlying these questions was the concern that mainstream
approaches to measuring outcomes for mental health services were
inappropriate for programs serving AI/AN communities. The major weakness
88
American Indian and Alaska Native Mental Health Research
Copyright: Centers for American Indian and Alaska Native Health
Colorado School of Public Health/University of Colorado Anschutz Medical Campus (www.ucdenver.edu/caianh)
MEASURING OUTCOMES IN MODEL SYSTEMS
89
of these mainstream approaches was their emphasis on clinical
conceptualizations of mental health, with measurement focusing exclusively
on problems rather than strengths.
Fortunately, these concerns were anticipated in the CoC Guidance
for Applicants (GFA), which identified explicitly the development of a plan for
measuring outcomes as a key goal of the initiative (SAMHSA, 1998a). Indeed,
the introductory section of the GFA stated that “The program is intended to
support tribes and urban Indian organizations in their efforts to develop
service delivery models, which will generate the outcomes selected by
American Indians/Alaska Natives for their own children [emphasis added].”
Other sections of the GFA amplify this intention:
The program is also intended to support the development
of measures and processes that will be useful to tribal and
urban Indian organizations in evaluating their service models
against the outcomes they have selected.
Thus, the underlying message of the GFA was that strategic plans
must include a plan for culturally and programmatically relevant approaches
to measuring outcomes. Such a plan would assure that the model, once
implemented, would be evaluated using the methods and measures consistent
with its design, objectives, and values (SAMHSA, 1998a).
In this paper, we describe the framework, process, and products of
this key component of the CoC evaluation. First, we describe the framework
provided by the Circles of Care Evaluation Technical Assistance Center
(CoCETAC) to the CoC grantees for developing their plan for measuring
outcomes. Next, we describe the process the grantees used for developing
their plans and a series of pragmatic issues that shaped this process. Then,
using the framework provided by CoCETAC, we provide an overview of the
grantees’ plans for measuring outcomes. Finally, we analyze the process
and products of this evaluation component and their implications for
communities, evaluators, and policymakers.
The Circles of Care Framework for Developing a Plan for Measuring
Outcomes
Grantees were presented with an idealized approach to develop a
plan for measuring outcomes that reflected the values, objectives, and
programmatic design of the strategic plan itself. CoCETAC identified the
following five aspects of measurement for the grantees to consider as they
developed their Outcome Measurement Plans: (a) “Domains of Measurement,”
(b) “Levels of Assessment,” (c) “Assessment Approaches,” (d) “Informants,”
and (e) “Timeline.” To ensure that the outcome plans were not simply
reflective of those measures that were most popular or expedient, grantees
were encouraged to consider each of these aspects before choosing the
American Indian and Alaska Native Mental Health Research
Copyright: Centers for American Indian and Alaska Native Health
Colorado School of Public Health/University of Colorado Anschutz Medical Campus (www.ucdenver.edu/caianh)
90
VOLUME 11, NUMBER 2
specific outcome measures they would employ. Each of these aspects of the
Outcomes Measurement Plan is described in more detail below.
Domains of Measurement
First, the grantees identified areas or “domains” that would be
impacted by their planned services. The following potential domains were
identified by grantees and CoCETAC through the Needs Assessment activities
described by Novins, LeMaster, Jumper Thurman, & Plested (2004) in another
paper in this volume: local concepts of health and mental health, symptoms,
indicators of health and dysfunction, resiliency and risk, tribal identities,
spirituality, family profiles, availability of services, barriers to accessing services,
and acceptability of services. Additionally, grantees were encouraged to
develop other domains that were appropriate to their service area needs.
Levels of Assessment and Assessment Approaches
Next, the grantees determined the “levels of assessment” they would
measure. Would they measure individual-based outcomes, family-based
outcomes, and/or community-based outcomes? In addition, grantees were
asked to choose specific approaches to their assessments. Would they
concentrate on measuring outcomes from a problem-based perspective,
typical of the measures used for programs serving non-Indian/American
Indian and Alaska Native communities? Alternatively, would they also measure
outcomes from a strength-based perspective, which was more consistent
with AI/AN concepts of health and balance? Would they use some combination
of these two approaches? The combination of these two aspects of the
framework can be conceptualized as a 2 x 3 matrix as presented in Table 1.
Table 1
A Matrix of Levels of Assessment and Assessment Approaches
Assessment Level
Assessment Approach
Individual
Family
Problem-Based
Strength-Based
American Indian and Alaska Native Mental Health Research
Copyright: Centers for American Indian and Alaska Native Health
Colorado School of Public Health/University of Colorado Anschutz Medical Campus (www.ucdenver.edu/caianh)
Community
MEASURING OUTCOMES IN MODEL SYSTEMS
91
Informants
Next the grantees identified the informants they would utilize in
measuring their outcomes. CoCETAC and the grantees generated a working
list of potential informants. Identified informants were the following: the
child/adolescent themselves, their parents/caregivers, the extended family,
elders, traditional healers, community members, project staff members,
biomedical clinicians, and secondary data (e.g. county mental health, schools
and juvenile probation). Project staff members were included in this list
because of the multiple perspectives they bring, including that of parents,
extended family members, and community members in addition to those of
clinicians and planners.
Timeline
The grantees then produced a project timeline. Grantees were
asked to consider when they would expect their programs to demonstrate a
measurable difference in the domains they had identified and how long they
would expect these impacts to last. For example, a baseline could be
established as ‘entry into the system’ for measuring many aspects of problems
and strengths at an individual or family level. Specific follow-up intervals
could then be specified that matched the grantee expectations for meaningful
changes. However, some outcomes might be difficult to match to an individual
child or adolescent’s entry into the System of Care. For example, information
on school-wide test scores, suspensions, and rates of graduation would only
be available on an annual basis consistent with school district or state
reporting requirements.
Selecting Specific Measures
After specifying the aspects of the Outcomes Measurement Plan
noted above, the final step for developing this plan was to select specific
measures to employ. CoCETAC and the grantees developed a substantial
library of potential measures which was revised several times as new
measures were identified.
The rationale for this approach of identifying the specific aspects of
their plans prior to selecting specific measures was that grantees would be
able to evaluate and choose potential measures based on utility for their
community contexts and specific service delivery models rather than popularity
in non-AI/AN programs and usage in county, state, and federal funding efforts.
American Indian and Alaska Native Mental Health Research
Copyright: Centers for American Indian and Alaska Native Health
Colorado School of Public Health/University of Colorado Anschutz Medical Campus (www.ucdenver.edu/caianh)
92
VOLUME 11, NUMBER 2
Factors Affecting the Development of the Plans
While the approach outlined above was considered ideal, three
important logistical and political issues resulted in a more pragmatic approach
for many of the grantees. First was the issue of time. The CoC evaluation
effort is demanding and time-consuming. By the time the grantees reached
this component of the evaluation (outcomes measurement), which was
usually well into the third year of their grants, they needed to move
expeditiously in order to meet their planning deadlines. Indeed, First Nations
Community HealthSource wrote that the specific challenge met in completing
this component of the evaluation was the “lack of time.”
Second was the issue of potential funding sources for implementing
their plan. Many funders, including CMHS, require specific outcomes
measures as a part of their agreement to fund services. For example, CMHS’
own Comprehensive Community Mental Health Services for Children and
their Families Program, which was identified by many of the grantees as an
important potential source of funding, mandates an extensive Outcomes
Measurement Plan (SAMHSA, 1998b). Many grantees were reluctant to
develop a plan that they would be unable to use under this and other initiatives.
Third, many of the service organizations that would participate in the grantees’
model systems of care had their own Outcomes Measurement Plans that
would have to be incorporated into the grantees’ plans as well. Many of
these measures were mandated by federal, state, and third party funders of
these programs.
Therefore, many of the grantees decided to simultaneously review
these existing measures and slot them into the aspects of their Outcomes
Measurement Plan identified above. This enabled them to identify those
outcomes that would not be measured by mandated instruments, and consider
whether additional measures were needed to ‘cover’ these outcomes. Again,
a pragmatic issue the grantees faced here was participant burden. With
extensive measurement plans already in place or mandated by potential
funders, the grantees had to decide whether additional measures would
create undo burden for participants in their systems of care.
We now review the characteristics of these plans.
Characteristics of the Outcomes Measurement Plans
Domains of Measurement
Table 2 displays the domains of measurement covered by the
grantees’ Outcomes Measurement Plans.1 Indicators of Health and
Dysfunction and Resiliency and Risk were covered by all eight of the grantees
that submitted plans for analysis in this paper. The domain of Symptoms
American Indian and Alaska Native Mental Health Research
Copyright: Centers for American Indian and Alaska Native Health
Colorado School of Public Health/University of Colorado Anschutz Medical Campus (www.ucdenver.edu/caianh)
MEASURING OUTCOMES IN MODEL SYSTEMS
93
was covered by seven grantees. Three grantees added the domain of
Satisfaction with Services to the original list.
Table 2
Domains of Measurement Covered by the Circles of Care Outcomes
Measurement Plans
Domains
Number of Grantees Including this
Domain in their Plan
Indicators of Health and Dysfunction
Resiliency and Risk
Symptoms
Service System Needs (Availability)
Spirituality
Barriers to Accessing Services
Acceptability of Existing Services
Local Concepts of Health and Mental Health
Tribal Identities
Family Profiles
Satisfaction with Services
Acculturation (or Cultural Identity)
8
8
7
6
5
5
5
4
4
4
3
2
Notes: Only those domains identified by two or more grantees are included in this Table.
Levels of Assessment and Assessment Approaches
These two aspects of the grantees plans are summarized in Table
3. Grantees developed plans that were very balanced, both in terms of using
problem- and strength-based approaches to measurement and in terms of
assessing outcomes at individual, family, and community levels. Indeed, the
emphasis on measuring community-level outcomes is unusual for mental
health service systems, but consistent with the broad goals of the grantees’
plans and the CoC initiative as a whole.
Table 3
Levels of Assessment and Assessment Approaches Included in the Circles
of Care Outcomes Measurement Plans
Assessment Level
Assessment Approach
Problem-Based
Strength-Based
Individual
Family
Community
6
6
6
6
5
6
American Indian and Alaska Native Mental Health Research
Copyright: Centers for American Indian and Alaska Native Health
Colorado School of Public Health/University of Colorado Anschutz Medical Campus (www.ucdenver.edu/caianh)
94
VOLUME 11, NUMBER 2
Informants
The list of informants included by the grantees in their plans is
summarized in Table 4. All grantees included parents/caregivers as
informants in their plans; seven included children/adolescents themselves;
five included extended family members and elders. There was considerable
divergence in informants beyond these three, fewer than half the grantees
including informants such as biomedical clinicians, secondary data, and
traditional healers.
Table 4
Key informants Included in the Circles of Care Outcomes Measurement
Plans
Domains
Parent/Caregiver
Child/Adolescent
Project Staff Member
Extended Family
Elder
Biomedical Clinician
Secondary Data
Community Member
Traditional Healer
Stakeholders
Number of Grantees Including this
Domain in their Plan
8
7
6
5
5
3
3
3
2
2
Notes: Only those informants identified by two or more grantees are included in this table.
Timeline
The timelines utilized by the grantees in their plans are summarized
in Figure 1. Baseline, 6-month, and 12-month data collection points were
the most commonly utilized by grantees in their timelines. In terms of number
of intervals utilized in the plan, four of the grantees collected data at four
points in time; one grantee collected data at two and one grantee at eight
points. While most grantees (four) tied their plans to time since entry into
the program, one grantee tied follow-up data collection to leaving the program.
American Indian and Alaska Native Mental Health Research
Copyright: Centers for American Indian and Alaska Native Health
Colorado School of Public Health/University of Colorado Anschutz Medical Campus (www.ucdenver.edu/caianh)
MEASURING OUTCOMES IN MODEL SYSTEMS
95
Figure 1
Timelines for Measuring Outcomes
Baseline: 8 grantees
3 months: 2 grantees
6 months: 6 grantees
9 months: 1 grantee
12 months: 6 grantees
18 months: 2 grantees
24 months: 4 grantees
Upon leaving the program: 3 grantees
6-12 months after completing program: 1 grantee
Annual review of secondary data: 1 grantee
36 and 48 months from baseline: 1 grantee
Specific Measures
The measures most commonly included in these plans are
summarized in Table 5. Six of the eight grantees developed (or intended to
develop) local instruments to use in their plans. These instruments were
largely intended to measure outcomes not normally addressed in commonly
used instruments (e.g., spirituality), tap into secondary data to measure
community impacts (e.g., decreased domestic violence), or to measure
domains from an AI/AN perspective. Only five commonly used individual/
family-based measures were selected by more than one grantee: the Behavior
and Emotional Rating Scale, the Child Behavior Checklist (and related
measures such as the Youth Self-Report), the Child and Adolescent Functional
Assessment Scale, Client Satisfaction Questionnaire, and the Family and
Youth Satisfaction Questionnaire. The Voices of Indian Teens Survey was
the only measure identified by more than one grantee that had been developed
specifically for use with AI’s. Community Readiness (Oetting, JumperThurman, Plested, & Edwards, 2001) was the most common approach to
examining community-level outcomes without relying on secondary data.
American Indian and Alaska Native Mental Health Research
Copyright: Centers for American Indian and Alaska Native Health
Colorado School of Public Health/University of Colorado Anschutz Medical Campus (www.ucdenver.edu/caianh)
96
VOLUME 11, NUMBER 2
Also notable is that of the eight measures chosen by two or more grantees,
four were measures included in the mandatory evaluation plan for the CMHS’
Children’s Mental Health Systems grants (SAMHSA, 1998b).
Table 5
Key Measures Included in the Circles of Care Outcomes Measurement Plans
Domains
Number of Grantees Including
this Domain in their Plan
Locally-developed measures
Child Behavior Checklist and Related Measures
Behavior and Emotional Rating Scale
Child and Adolescent Functional Assessment Scale
Community Readiness Measures
Client Satisfaction Questionnaire - 8
Family and Youth Satisfaction Questionnaires
Voices of Indian Teens Survey
6
5
3
3
3
2
2
2
Conclusions and Implications
The CoC grantees developed outcomes measurement plans that,
while affected by the demands of potential funding mechanisms, also
incorporated instrumentation that would allow them to focus on the domains,
assessment levels, assessment approaches, and timelines most appropriate
to their strategic plans. Indeed, the process and products of the CoC plans
for measuring outcomes have important implications for communities, clinical
programs, evaluators, and policymakers.
For communities and the clinical programs that serve them, the CoC
grantees demonstrated that Outcome Measurement Plans can focus on both
individual and family strengths and problems. Indeed, as a group the CoC
grantees identified a menu of measures that approached outcomes from
these different perspectives. Such a balanced approach to measuring
outcomes should provide a more complete assessment of the progress a
child and family make while they are receiving services and afterwards, and
should serve as a model for other systems of care.
Furthermore, these results underscore the importance community
members place on measuring community level outcomes. Thus, there is an
expectation among community members that mental health services will not
only provide positive outcomes for children, adolescents, and families who
receive these services, but that the impact of services will extend to the
greater community as well. Not only should children and adolescents who
receive mental health services be more likely to graduate from high school,
but effective mental health services should raise the graduation rate for
entire schools. Thus, an effective system of care was viewed by the CoC
communities as a healing process not only for individuals and families, but
for communities as well.
American Indian and Alaska Native Mental Health Research
Copyright: Centers for American Indian and Alaska Native Health
Colorado School of Public Health/University of Colorado Anschutz Medical Campus (www.ucdenver.edu/caianh)
MEASURING OUTCOMES IN MODEL SYSTEMS
97
For evaluators, the Outcome Measurement Plans produced by the
CoC grantees demonstrate the importance of a participatory approach to
developing these plans. Given the constraints already placed on programs
and communities by policymakers and funders, evaluators need to be
particularly attuned to their program and community partners and work to
identify existing measures that meet the community’s visions of positive
outcomes rather than relying on those problem-focused measures that are
typically employed in mental health programs. Given the likely possibility
that no existing measures will be completely consistent with this vision,
evaluators should be prepared to work with their partners to develop
measures and measurement approaches that fill in the gaps that will almost
certainly exist. A full appreciation of the scope, strengths, and weaknesses
of existing measures, the process involved in developing community-specific
measures, and the use of implicit measurement techniques (Brook & Cleary,
1996) for particularly complex areas of assessment (e.g., whether an child,
adolescent, and family are “in balance,” and how they are “out of balance”)
is a particularly important ability for evaluators to possess.
There are two important lessons for policymakers in the Outcomes
Measurement Plans produced by the CoC grantees. First, these plans serve
as both a wonderful example of what communities are capable of when
given the time and necessary fiscal and technical support to develop model
programs. The resultant plans for measuring outcomes are particularly
impressive in their comprehensiveness, the ways they reflect community
values and beliefs, and their responsiveness to clinical perspectives of mental
health service delivery. As such, the process for these developing plans,
embodied in the CoC approach to strategic planning and program evaluation,
are a model for similar efforts in both AI/AN and non-AI/AN communities
alike.
Second, these plans also demonstrate the power the funders hold
in shaping the entire discussion on measuring Outcomes. The more funders
specify the use of specific outcomes measures, the less communities and
clinical programs will pursue innovative approaches to measurement. Thus,
funders must be thoughtful in balancing their need for commonality in outcome
measurement in the programs they support and the need for communities
and service systems to measure the outcomes that reflect the values and
beliefs of the communities they serve.
Douglas K. Novins, M. D.
Director, Circles of Care Evaluation Technical Assistance Center
American Indian and Alaska Native Programs
Nighthorse Campbell Native Health Building
P.O. Box 6508, Mail Stop F800
Aurora, CO 80045-0508
Tele: (303) 724-1414
FAX: (303) 724-1474
Email: [email protected]
American Indian and Alaska Native Mental Health Research
Copyright: Centers for American Indian and Alaska Native Health
Colorado School of Public Health/University of Colorado Anschutz Medical Campus (www.ucdenver.edu/caianh)
98
VOLUME 11, NUMBER 2
References
Brook, R. H., & Cleary, P. D. (1996). Measuring quality of care. New England
Journal of Medicine, 335, 966-970.
Nixon, C. T., & Northrup, D. A. (Eds.). (1997). Evaluating Mental Health Services:
How Do Programs For Children “Work” in the Real World. Thousand Oaks,
CA: Sage.
Novins, D. K., LeMaster, P., Jumper-Thurman, P., & Plested, B. (2004). Describing
community needs – Examples from the Circles of Care initiative. American
Indian and Alaska Native Mental Health Research: The Journal of the National
Center, 11(2), 42-58. Available at: http://www.uchsc.edu/ai/ncaianmhr/
journal/11(2).pdf
Oetting, E. R., Jumper-Thurman, P., Plested, B., & Edwards, R. W. (2001).
Community readiness and health services. Substance Use & Misuse, 36(67), 825-843.
SAMHSA. (1998a). Circles of Care: Planning, designing, and assessing mental
health service system models for Native American Indian and Alaska Native
children and their families. Rockville, MD: Center for Mental Health Services,
Substance Abuse and Mental Health Services Administration.
SAMHSA. (1998b). Comprehensive Community Mental Health Services for Children
and Their Families. Rockville, MD: Center for Mental Health Services, Substance
Abuse and Mental Health Services Administration.
Footnote
1
Eight of the Nine Circles of Care grantees provided Outcomes Measurement Plans for this report.
American Indian and Alaska Native Mental Health Research
Copyright: Centers for American Indian and Alaska Native Health
Colorado School of Public Health/University of Colorado Anschutz Medical Campus (www.ucdenver.edu/caianh)