Using Theories and Frameworks of
Evaluation Practice to Inform and
Improve our Work
Christina (Tina) Christie
UCLA
Danish Evaluation Society
2014 Meeting
1
Evaluation Defined
• Program evaluation is the use of social research methods to
systematically investigate the effectiveness of social intervention
programs. (Rossi, Lipsey, Freeman, 2004, p. 28)
• Evaluation refers to the process of determining the merit, worth, or
value of something, or the product of that process. (Scriven, 1991,
p. 139)
• The evaluation of educational and social programs aspires to be an
institution for democratizing public decisions by making programs
and policies more open to public scrutiny and deliberation. (House,
1993, p. 1)
• Program evaluation is the systematic collection of information
about the activities, characteristics, and outcomes of programs to
make judgments about the program, improve program
effectiveness, and/or inform decisions about future programming.
(Patton, 1997, p. 23)
2
Question
• Which definition best reflect the kinds of
evaluations you do?
3
Theories in Evaluation
• Social Science Theory
• Program Theory
• Evaluation Theory
4
Evaluation Theory is Who We Are
(Shadish, 1997, AEA Presidential Address)
A diverse set of theoretical writings held
together by the common glue of having
evaluation practice as their target
5
Evaluation Theory is Who We Are
• A common language
• Identification of
important issues (e.g.,
evaluation use)
• Defines themes
• An identity defining
evaluation from other
professions
• A face for presentation
to the outside world
• A unique knowledge
base
6
Two Types of Theory
Scriven (1998)
• External theories of
evaluation
– Theories about the
multiple dimensions of
evaluation practice
– Encompass most of our
evaluation theory
literature
• Internal evaluation
theories
– Concerned with
evaluation as an
intellectual discipline
– Two levels more abstract
than evaluation practice
7
External Theories
• There are many ways (roadmaps) to approach an
evaluation of a program or policy
• Roadmaps are also called “theories”
• There are many evaluation theories and there is no one
“right” theory or way of doing an evaluation
8
What are Evaluation Theories/Models?
Prescriptions for Practice
• Qualitative models, points-of-view, persuasions,
and approaches to the process of evaluation
• Principles, rationales, and organization for the
procedural choices made by evaluators
• Orient practitioners to the issues and problems
with which they must deal
9
Why Use an Evaluation Theory/Model?
10
Why Use an Evaluation Theory/Model?
• Provides a conceptual framework
• Helps organize procedures
• Allows for selective application
11
Why Know More About Evaluation
Theories/Models?
• Determine the strengths and shortcomings
comparatively
• Develop a critical view of alternatives so to
apply optimal framework
• Identify personal values related to the
evaluation enterprise
12
What Distinguishes Models?
• What one believes to be the purpose of
evaluation
• The techniques and strategies used to
carry out the purpose
13
Some Questions Asked to Help
Distinguish Theories/Models
• The role of the
evaluator in:
– Valuing data
– Making causal claims
(rather than descriptive)
– Accommodating the
political setting
– Providing information for
decision-making
• Who is the primary
audience?
– What role do they play?
• What is the role of
those affected or
interested in the
evaluation?
– In what ways do they
participate?
14
Scoring for Paired Items
• Give yourself 1-point for each of the
following statements that you checked:
1A, 2A, 3B, 4B, 5A, 6A, 7A, 8B, 9A,
10A, 11B, 12B, 13A, 14B, 15A
• Score range: 0 to 15
– 0 to 5 = someone who values a more qualitative/naturalistic
worldview
– 11 to 15 = someone who values a more
quantitative/positivist worldview
15
“Evaluation Theory Tree”
(2012)
 Theorists
- identified with a particular evaluation point of
view or approach
- NOT individuals who are primarily
methodologists
- NOT “evaluation issue analysts” who write
about a variety of topics
- NOT those who are primarily practitioners of, or
writers about evaluation
16
“Evaluation Theory Tree”
• A theory about theories (or theorists)
• Elements common to all theories
- Methods
- Valuing
- Use
• The “Roots” Theory Tree
- placed on the tree to reflect relative emphasis
- sense of historical sequence (or influence)
- Acknowledgement: North American centric- Solely North American theorists in the 2012
version
17
18
19
Tree “Roots”
• social accountability- an important motivation for
evaluation, a way to improve programs and society
• systematic social inquiry—methodical and justifiable
set of procedures for determining accountability
• epistemology- nature and validity (or limitations) of
knowledge- the legitimacy of value claims, the nature
of universal claims, and the view that truth (or fact)
is what we make it to be
20
21
Methods
• more accurate to describe these
approaches as emphasizing research
methodology
• evaluation is primarily centered on
research methodology- “knowledge
construction”
• models are mostly derivations of the
randomized control trail, and are intended
to offer results that are generalizable
22
23
Valuing
• placing value on the subject of the evaluation, the
evaluand, is essential to the process
• initially driven by the work of Michael Scriven (1967)
and Robert Stake (1967, 1975), this work firmly
establishes the vital role of valuing in evaluation
• split in two—objectivist and subjectivist—which
distinguishes the two fundamental perspectives
informing the valuing process
24
25
Use
• the pioneering work of Daniel Stufflebeam
(initially with Egon Guba) and Joseph Wholey,
originally focused on an orientation toward
evaluation and decision making
• an explicit concern for the ways in which
evaluation information will be used, and
focuses specifically on those who will use the
information.
26
27
Thank you for your time and attention!
30