W
e all agree with the need to be
accountable. Yet, is it easily
integrated into our work?
There are many terms
applied to outcomes - results-based
performance,
benchmarking
and
performance indicators. It is even described
as "measurement mania".
They seem to all point to the fact that
publicly funded programs are asking for
statements on value for their cost. I use the
term measure to refer to the yardstick for
assessment of programs. The focus is on
what yardsticks determine outcomes or the
difference a program has made.
Let's take a serious look at embracing outcome measures in the area of recreation. I
argue it's important to have your work and
integrity reflected in assessment. To ensure
this, you must understand outcome measurements and be prepared to challenge them
both from your understanding of their application and from the perspective of your own
work and its broader contribution to society.
Programs Have Many
Outcomes
Your programs probably produce outcomes at a number of levels. Many programs
concentrate on changes in program participants. To assess the outcome on individuals
use measures of attitudes, values, knowledge,
skills and behaviour.
For some attributes, outcome measures
re difficult to find or develop but you
attempt to measure according to program
goals. You may, for instance, seek opinions of
participants about the program outcomes.
Cycling Path - Toronto
By Kaireen Chaytor, Professional Evaluator of Social,
Health and Education Programs
What do they believe they gained?
Other programs may produce changes
in institutions rather than in people such
as making local bureaucracies more
responsive to residents or increasing access
to facilities. For these programs, indicators
of program outcomes will be measures of
institutional characteristics.
Some programs attempt to change a
whole network of agencies or to change a
community
through
providing
an
integrated service. In cases like these,
outcome measures have to relate not only
to the individuals who live in the
communities and the improvements in
their quality of life, but also to the social
munity and the increase in coordination and
cooperation among them. For these programs, we
attempt to assess community change. Programs
intended to reduce youth crime may be an example.
Measures have to relate to categories of individuals,
agencies, a community or the public.
If a program seeks to alter public values or
attitudes, the appropriate indicator of outcomes is
obviously the public's views. You may want to know
the effects of a community education program
designed to alter public views. The program may be
altering the public's views on topics such as litter
reduction or the preservation of a piece of land for a
park.
which you report really reflect what you
believe are your accomplishments? Some programs, such as stewardship of the environment, do not lend themselves to simple outcome measures. The goal is to create the setting or opportunity for other events to occur.
The Cautions - The Challenges
We accept that we no longer measure
inputs (the resources allocated) but we discuss outputs (the activities or products) or
outcomes (the difference we have made).
Bert Perrin in an article published in The
American Journal of Evaluation (Vol. 19:3,
1999), identifies concerns with assessment
by outcomes. Some concerns include:
Whether your program is intended mainly to have an
effect on participants, agencies, a community or the
broader public, you need to know the social context of
your program.
Know the Context
Whether your program is intended mainly to
have an effect on participants, agencies, a
community or the broader public, you need to know
the social context of your program. What
legislation is directing your effort? What is your
mandate? Does the legislation direct or suggest who
you should reach?
Canadian legislation tends to have a broader
reach, including the hard-to-reach population, than
does American legislation. Evaluation designs based
on American legislation may not be appropriate for
the Canadian context when measuring outcomes.
Measuring the Hard to
Measure
In areas such as recreation, important
outcomes may be changes over the long
term. The shift to measuring outcomes
has given rise to slogans such as "what gets
measured, gets done".
When the impact of a program is difficult
to measure, and measures are not
obtainable in the short run, does it call into
question the value of the program? The
pressure to use measures of outcome may
mean we select meaningless and irrelevant
measures of outcomes. Ask yourself, do
"...We hold many interpretations of the
'same' terms and concepts around outcomes.
...No matter how clearly we believe we
define terms, they will invariably be interpreted in different ways, at different levels,
by different people."
Because there is no consistency in interpreting measurements, they lose much of
their punch when used in making an argument. The need, therefore, is for program
staff to agree upon outcome measures.
There is a danger that programs may use
the outcome measures as an end in themselves. This may be happening with the
results-based legislation in the United States.
People will be encouraged to use those measures which make a program look like it's
doing a good job.
An outcomes-based approach is often
the form of evaluation required by external
funding agencies. They often do not take
into account the reality of program implementation or the environment in which a
program is operating. Outcome measures
may remain constant while the environment, needs and program are changing. This
approach to evaluation results in an emphasis on justifying and defending what was
done and a reluctance to admit that
improvement is needed.
Outcome measures are not necessarily
useful for decision-making and resource
allocation. Outcome measures by themselves
can be useless because they do not answer
the how and why questions, answers necessary for a good understanding of whether
investment in certain programs is worthwhile. It may be that programs fail to meet
We must develop programs that enhance the quality of life for individuals and communities. We are in an outcomes era. The
challenge is to ensure that the outcomes are measured by yardsticks that appropriately reflect your accomplishments.
their output targets because the theory behind the program was bad. However,
you will not find this out from an outcome-evaluation only.
It is important to recognize that outcome measures are most appropriate in
planning and monitoring, not for evaluation. When evaluating, you must be
open to what has in fact occurred.
Secondly, with every evaluation method, including an outcomes-based
approach, there are limitations. These can be overcome by using a combination of
evaluation methods.
And finally, be strategic. Not all programs lend themselves to a measure of
outcomes. There are certain programs where it is impossible to measure outcomes. It is preferable to find alternative ways to report on a program than to
apply inaccurate or irrelevant measures.
We must develop programs that enhance the quality of life for individuals
and communities. We are in an outcomes era. The challenge is to ensure that the
outcomes are measured by yardsticks that appropriately reflect your accomplishments. •
The Author
Kaireen Chaytor has a MA and PhD in Adult Education from Dalhousie
University where she teaches program evaluation in the School of Public
Administration and also teaches at Henson College in the Non-profit Sector
Management Program. Kaireen worked for ten years in continuing professional education and now conducts evaluations of many social, health and education programs
on a contract basis.