Cultivating
Participatory
Evaluation in a
World of
Centralized
Accountability
Jean A. King
University of Minnesota
2
The Land of 10,000 Lakes
J. A. King
3
The Mississippi Headwaters
J. A. King
4
Birthplace
of Judy
Garland. . .
J. A. King
5
. . . and of
Bob Dylan
J. A. King
6
. . . and
Charlie
Brown and
Snoopy
J. A. King
7
Home to Garrison
Keillor’s
Prairie Home
Companion
J. A. King
8
. . . including
Lake
Wobegon,
where. . .
J. A. King
9
. . . the women
are strong,
J. A. King
10
. . . the women
are strong,
the men are
good looking,
J. A. King
11
. . . the women
are strong,
the men are
good looking,
and all the
children are
above average
J. A. King
12
My Background
Junior
high school English teacher
in upstate New York
Teacher educator in New Orleans
Head of a collaborative research
center at University of Minnesota
Professor of evaluation studies
J. A. King
13
“Cultivating Participatory Evaluation (PE)
in a World of Centralized Accountability”
 What
is participatory
evaluation (PE)?
 How does centralized
accountability relate to PE?
 How can one cultivate PE in
such an environment?
 What cause is there for
hope?
J. A. King
14
My PE principles
1.
2.
3.
Involving people effectively
in evaluations is essential.
Participation in evaluations
can and should be a
learning experience.
Building people’s capacity to
think evaluatively matters.
J. A. King
15
What is participatory
evaluation (PE)?
Do our definitions matter in practice?
J. A. King
17
Interactive Participation Quotient
HIGH
Program
leaders,
staff,
community
members
Involvement in
decision making
and implementation
Evaluator
LOW
Evaluatordirected
Collaborative
ZONES
Participantdirected
J. A. King
20
What I have learned about PE
The
farther to the right on the
IPQ, the longer the evaluation
process will take
The benefit of the time spent
comes from people’s learning
about evaluation
J. A. King
21
What are the principles of PE?
 Participants
OWN the evaluation
 The evaluator facilitates; participants plan
and conduct the study
 People learn evaluation logic and skills as
part of the process
 ALL aspects of the evaluation are
understandable and meaningful
 Internal self-accountability is valued
(Adapted from Patton, 2008)
J. A. King
22
What are characteristics of PE?
 Control
of the evaluation process [ranges
from evaluator to practitioners]
 Stakeholder selection for participation
[ranges from primary users to “all
legitimate groups”]
 Depth of participation [ranges from
consultation to deep participation]
(From Cousins & Whitmore, 1998)
J. A. King
23
Cousins & Whitmore’s framework
J. A. King
24
Examples
 Graduate
Review Improvement Process in
my department
 Evaluation of the Special Education
Program in Minnesota’s largest school
district
 Community Listening Project at
Neighborhood House, a social service
agency in West St. Paul, MN
J. A. King
25
One unfortunate confusion
Too many labels
for different types of PE
J. A. King
26
Competing participatory
approaches
1.
2.
3.
4.
5.
6.
7.
Practical participatory evaluation
Transformative participatory evaluation
Collaborative evaluation
Empowerment evaluation
Democratic deliberative evaluation
Inclusive evaluation
Values-driven evaluation
J. A. King
27
Related terms adding to the confusion
Utilization-focused
evaluation
All kinds of action research
Responsive evaluation
Evaluation capacity building
Organizational learning
Others?
J. A. King
28
What’s in a name?
1.
Why does the field have so many
different names for similar
processes?
J. A. King
29
What’s in a name?
1.
2.
Why does the field have so many
different names for similar
processes?
What difference does this make to
the practice of PE?
J. A. King
30
What’s in a name?
1.
2.
3.
Why does the field have so many
different names for similar
processes?
What difference does this make to
the practice of PE?
What are the most important
distinctions?
J. A. King
31
The most important distinctions
Who
is invited to the table?
Who is responsible for the
process(for decision making, for
implementation)?
The extent to which the study is
framed in broader societal
terms
J. A. King
32
How does centralized
accountability relate to PE?
Is it helpful to have accountability demands?
J. A. King
34
On the one hand. . .




Accountability mandates typically come from
outside organizations
They can exert incredible pressure on frontline staff (feelings of helplessness)
They may not reflect the reality of meaningful
requirements for achieving better or different
outcomes
They often rely on a quantitative
epistemology (“Prove that your program
delivers these results”)
J. A. King
35
On the other hand
 Programs
should be held accountable
 Programs should engage in ongoing
evaluation to improve their practice
 PE engages people in collecting data
and reflecting on their practice
 “The best defense is a good offense”- PE
can help programs both understand and
defend their practice
J. A. King
36
Addressing the claim
that PE is “biased”
 What
is bias? Systematic distortion or an
inclination that prevents “unprejudiced
consideration”
 Why can’t PE processes minimize this?
 Solution: documenting potential concerns
and reflecting on them publicly
 Still have the benefit of participation
(possible trade-off)
J. A. King
37
How can one cultivate PE in an
environment of accountability?
Standing one’s ground makes a difference
J. A. King
38
Three possibilities
1.
2.
3.
Shoe string interactive evaluation
practice (IEP)
Evaluation capacity building (ECB)
Strategic instruction of policy
makers
J. A. King
39
Three possibilities
1.
“Shoe string” interactive evaluation
practice (IEP)
J. A. King
Good
Cheap
Quick
Choose two.
J. A. King
An evaluator’s rule of thumb:
40
41
Interactive evaluation practice
“. . . The intentional act of
engaging people in making
decisions, taking action, and
reflecting while conducting
an evaluation study”
J. A. King
1. Cooperative (three-step)
interviews
2. Data dialogues
3. Carousel and concept formation
tasks
J. A. King
Shoe string techniques for engaging
people/collecting data
43
44
Three possibilities
1.
2.
Shoe string interactive evaluation
practice (IEP)
Evaluation capacity building (ECB)
J. A. King
45
What is evaluation capacity
building (ECB)?
Using an evaluation for
(1) its results
AND
(2) the explicit purpose of
building people’s capacity
to evaluate again
J. A. King
46
Evaluation capacity building
“Intentional work to constantly
co-create and co-sustain an
overall process that makes
quality evaluation and its uses
routine in organizations and
systems”
Stockdill, Baizerman, & Compton (2002)
J. A. King
47
For the ECB practitioner, the focus
“. . .[is] on responding to requests
for evaluation services while
simultaneously considering how
today’s work will contribute to
sustaining the unit in the longer
term”
(Compton, Glover-Kudon, Smith, & Avery, 2002, p. 55)
J. A. King
48
Where did ECB come from?
Free range evaluation
J. A. King
49
Goals of ECB
1.
2.
3.
4.
Increase an organization’s capacity to
design, implement, and manage
effective evaluation projects
Access, build, and use evaluative
knowledge and skills
Create support for program evaluation
as a performance improvement strategy
Cultivate a spirit of continuous
organizational learning, improvement,
and accountability
J. A. King
50
Why now?
Rising
accountability
demands
The cost of evaluation
The availability of technology
The value (and fun) of the ECB
process
J. A. King
51
Evaluation capacity-building
(ECB) continuum
Formative /
Summative
evaluation study
Evaluation
Evaluation for
specifically for
organization
building capacity
development
to evaluate
----------------------------------------------------------------------
Use of single
study
process / results
ECB = creating
capacity to
conduct
evaluations
Capacity to
sustain change
J. A. King
52
Three possibilities
1.
2.
3.
Shoe string interactive evaluation
practice (IEP)
Evaluation capacity building (ECB)
Strategic instruction of policy
makers
J. A. King
53
The PE “curriculum”
 Proof
of causation in the social sciences is
a measurement challenge
 PE is a valid and viable form of
collaborative inquiry
 Research documents that involvement
leads to increased use
 PE leads to added benefits as
organizations build their capacity to
evaluate over time
J. A. King
54
Three possibilities
1.
2.
3.
Shoe string interactive evaluation
practice (IEP)
Evaluation capacity building (ECB)
Strategic instruction of policy
makers
J. A. King
55
What cause is
there for hope?
While there’s life, there’s hope. . .
J. A. King
56
To
travel
hopefully
Thank you!
is a better thing
than to arrive.
-Robert Louis Stevenson
J. A. King
Thank you!
Jean A. King
([email protected])