Evaluation Research: Its Current and Potential Place in
Policy Analysis in Brasil and the United States
Abstract
Evaluation research – or program evaluation – plays an important role in public policy and
public administration practice, education, and research in the United States. However, the place
of this form of applied research is problematic in several important ways. Evaluation research is
inconsistently applied and often not completely applied in the design, implementation, and
evaluation of public initiatives. Evaluation research is also essentially treated as set of practices
apart from policy analysis. In Brasil evaluation research appears not to be widely practiced or
taught and, as in the case of the U.S., is not actively linked with policy analysis research,
practice, and education. In this presentation it is argued that evaluation research should play an
important role in Brasilian pubic policy and administration. It is also argued that policy analysis
in Brasil and the United States can be enhanced through active links to the practices and products
of evaluation research.
Introduction
I recently heard an interview on pubic radio in the U.S. with the man who, until recently, was the
most powerful central banker in the world – Ben Bernanke. Professor Bernanke was promoting
his new memoire. As you might expect, the conversation with him largely focused on his role in
dealing with the Great Recession. Something that struck me in Professor Bernanke’s comments
was the disconnect - before, during, and after the Great Recession – between policy and “ground
level” outcomes in the daily lives of those impacted by public policy. Professor Bernanke works
– as a professor, former policy maker, and member of the Brookings Institute – in a world of
economic modeling and macro-aggregated data. Data aggregated to the point that it is
meaningless in terms of the daily lives of those impacted by public policy.
Unlike the policy analysis perspective in Professor Bernanke’s world, evaluation research asks
questions and includes methods that can generate evidence that can make policy development,
implementation, and analysis more realistic – and better connected to those who are impacted by
public policy. It does this by asking questions and providing methods that address:
2

the need for public action;

identify evidence-based theory that can be used to construct public policy and
programming;

design processes that link necessary public resources that can generated needed
public goods and services;

evaluate the effectiveness of programs and policies in dealing with public needs;
and,

assess the cost consequences associated with public policy and programmatic
action.
Today I will discuss what I see as a need to connect evaluation research with policy analysis. I
will this by first reviewing the characteristics of evaluation research as I practice and teach it. I
will then consider the imperfect application of evaluation research in the United States. Finally, I
will offer my speculation regarding a new model of policy analysis that takes into account what
evaluation research has to offer.
Characteristics of Evaluation Research
Overview of evaluation research
Evaluation research should be an important component of public policy and administration
education programs. However, this is generally not the case. To understand this, we need to
address the following questions:

What is the value of evaluation research to public policy and administration?

How does evaluation research fit into the public policy process?

What are the characteristics of evaluation research?

How does evaluation research fit into the broader world of social science research?

What is the value of evaluation research to evidence-based practice in public sector
research, education, and practice?
Evaluation research is social science directed at collecting, analyzing, interpreting and
communicating information about the design, workings and effectiveness of social programs. It
can be seen as a form of appraisal, using valid and reliable research methods, to examine the
3
processes or outcomes of policies or programs that exist to fulfill societal needs. As such,
evaluation research is an effort to apply the principles of scientific inquiry to public policy and
program decision-making. It identifies the characteristics of effective public goods and services
production and delivery so that policies and programs can be well designed.
Like other forms of research, evaluation research is driven by the asking and answering of useful
questions. Where evaluation research differs from other forms of social science research lies in
the sources of its questions and how it goes about answering these questions.
Rather than questions that arise in the minds of academics seeking to advance their careers, good
evaluation research questions are driven by real-life and real-time questions that are related to
making society work more effectively, and improving the lives of people in need.
Evaluation researchers must collaborate with skate holders associated with the problem to be
addressed by public policies and programs. They do this to first identify the questions that must
be answered to address public needs. Then they must find ways in the toolbox of social science
methods to answer these questions. Evaluation researchers cannot be limited to one or a few
methods of research. They must be flexible in finding methods that fit the evaluation challenge.
They do this, even if it means that they must change methods in the course of an evaluation
project.
Thorough evaluation research projects are not limited to one, two, or three research questions as
we typically see in other forms of social science research. They usually address many complex
and interrelated questions.
The questions asked are practical questions related to:

identifying the need for public policy or programmatic intervention

evidence-based approaches to address these needs

assessing the operations of these policies and programs

measurement of their effectiveness

and their cost consequences
I will briefly discuss the characteristics of the five forms of evaluation research associated with
each of these sets of questions.
4
As I describe each of the elements of evaluation research, remember that evaluators should be
involved in two ways: as collaborators in policy and program design, and in the assessment of
policy and program performance.

As I will discuss later in my presentation, in the United States trained and experienced
evaluators are not frequently included in policy and program design.
Needs assessment
The first element of evaluation research that we will consider – needs assessment – should be
viewed as essential to the design of a public policy or program.
Needs assessments are intended to answer a variety of interrelated questions:

What are the characteristics of a societal problem?

What is the extent of the problem?

Does the problem require the creation of a new policy or program?

What are priorities among the needs for services related to the problem?

If new services are needed, how will they be funded?

Do other policies and programs exist that are intended to address the problem?

If existing policies and programs exist, why are they not successful in addressing the
subject problem?

To what extent should existing interventions be augmented or replaced?
To answer these questions, evaluation researchers utilize all of the methods of social science,
including secondary analysis of existing data and existing research, and the various forms of
survey research.
In conducting a public policy or program needs assessment, the evaluation researcher, in
collaboration with a variety of stakeholders, will begin to apply the principles of evidence-based
practice to identify policy and programmatic interventions that might be appropriate to address
problems under consideration.
5
As I will further discuss, unfortunately needs assessment are often not performed, are
incompletely performed, and/or not performed with the assistance of skilled evaluation
researchers.
Policy or Program theory
Public policy and programmatic interventions are introduced based upon sets of ideas regarding
what can be done by public organizations to create changes that address societal problems. These
ideas are represented in the policy or program theory that will guide the design of a public
intervention.
The policy or program theory describes what an intervention will do for those that it is intended
to benefit.
The policy or program theory includes the assumptions about the changes that are expected to
result for beneficiaries of the public intervention from their exposure to the policy or program.
It also describes how the policy or program will connect the target population to the benefits that
it is intended to provide.

It should detail how the target population will be reached.

It should include a complete description of the processes that will be required to
accomplish the intended intervention relationship with the target population.
The policy or program theory should include a thorough organizational plan describes the
program resources that are required for the intervention to provide the intended services.

These include service personnel, facilities, and administrative and support personnel.

The plan should describe how these resources will be organized to accomplish the
intended activities to create a service delivery system.

Since “new” policies and programs typically involve collaboration of existing
organizations and the “re-engineering” of existing organizational resources, the
organizational plan of the policy or program theory is of particular importance.
The policy or program theory will describe the impacts and outcomes that the intervention is
intended to accomplish.
6
The theory of an intervention is about cause and effect relationships.

It says that if certain activities take place, beneficial effects will result.

It will describe which intervention components will have which intended proximal and
distal effects.

The theory of the intervention will emphasize the reliance of distal outcomes on the
successful realization of proximal outcomes.
Process analysis
In the United States one of the roles for which we see evaluation researchers recruited to perform
involves evaluation of policy and program processes.

This is the case, although evaluators typically are not involved in designing these
processes.
Process evaluation is about determining whether the policy or program is operating as intended
by its planners.
Process evaluations seek to make the following determinations

Is the policy or program reaching the target population?

Is the intervention providing the intended services to the target population?

Does the intervention provide the intended quantities of services to the intended
beneficiaries?

What is the quality of public goods and services provided through the policy or program?
Process evaluation measures a policy or program’s operation in terms of service utilization and
organization.

In assessing service utilization the evaluator determines the extent the target population
receives the intended benefits of the intervention.

In assessing policy or program organization the evaluator compares the plan for the
intervention’s operation with what is actually happening, particularly in terms of service
provision.
7
For process analysis to be most effective, public policies and programs should be designed to
collect data that support continuous monitoring of intervention processes.

This requires the development of management information systems designed to support
the requirements of the policy or program theory.

Unfortunately, my experience shows that this rarely happens in the U.S.
Outcome analysis
Analysis of policy and programs impacts or outcomes is the most common reason for evaluation
researchers to be hired by policy makers or funders.
Regardless of how well an intervention is organized, how well its processes function, and how
pleased beneficiaries are with the services that it provides, if it does not realize the impacts for
which it was intended and designed, the intervention will not be considered successful.
In light of the importance of outcome evaluation, evaluators seek to apply methods that are as
rigorous as possible.

To identify meaningful and measurable outcomes.

To collect and analyze outcome data that will allow them to make meaningful
assessments regarding the impact of policies and programs.
Identifying and applying methods in outcome analysis is a challenge.

It is difficult to apply methods that are considered to be rigorous enough to satisfy policy
or program stakeholders, while also meeting the demands of evidence-based practice.

Those interested in evidence-based practice – particularly academic researchers – demand
experimental research designs with high levels of power.
o Frequently random assignment is unethical and low power levels resulting from small
sample sizes is typical.

Since management information systems intended to support policies and programs are
not designed and implemented, evaluation researchers often must use secondary sources
of data that may not be adequate to meet the requirements of outcome analysis.
8
Since needs assessments, program theories, and carefully designed and documented processes
are not in place, evaluators are often forced to measure impacts and outcomes that are not the
best indicators of the performance of policy and program interventions.
Cost analysis
Ultimately, policy makers and funders of policies and program demand information about the
cost consequences of public policy and program interventions.
Cost analysis takes three basic forms:

Cost effectiveness analysis considers the cost to provide an intervention.

Cost efficiency analysis considers comparisons of the cost effectiveness of alternative
intervention approaches.

Cost benefit-analysis compares the value gained through intervention as compared to its
costs.
On the surface it appears that these should be fairly straightforward matters of definition and
analysis.
However, cost analysis is full of controversial issues. These include:

What theory of cost analysis should be utilized?

What costs should be considered?

What are acceptable sources of cost data?
For me the most important controversy involves which theory or cost analysis approach should
guide the analysis.
Economists embracing neo-classical economic theory have dominated this conversation in the
past.

As a result, an analytic approach essentially focused on marginal cost analysis has
dominated cost analysis in evaluation research.
I argue that economist tend not to understand the organizational reality of public policies and
programs.
9

They are products of complex mixtures of organizational resources among multiple preexisting organizations.

They are subject to the budgetary forces that public administrators better understand.
As a result, I further argue that a cost analysis approach should be utilized that recognizes the
organizational and public administration reality of public policy and programmatic interventions.
I have represented these considerations in a cost analysis approach that I have utilized in my
evaluation work.

It is called the Transactional and Institutional Cost Analysis (TICA) approach.

This model has been utilized in dozens of evaluations in the United States.
Imperfect Use of Evaluation Research in the United States
I think that my discussion thus far represents a set of practices that are reasonable,
straightforward and offer obvious value to public policy and administration.
However, my experience indicates that the principles and practices of evaluation research rarely
are fully and completely applied in the United States.
Policy makers and funders in the United States are typically only interested in outcome and cost
analyses – and as soon as possible after the implementation of a policy or program.
As a result, inadequate attention is directed to developing and evaluating needs assessments,
intervention theories, and process assessments.
The push to have policies and programs evaluated shortly after their implementation may not
allow for distal impacts of interventions to be realized.
Inadequate attention to needs assessment and intervention theory development may result in the
introduction of policy and programs that are not good fits for the problem of concern to
stakeholders.
Inadequate attention to intervention theory development and process design often results in no
management information created to collect intervention-specific data and support process
monitoring.
10

This makes it more difficult for the evaluation researcher to collect data, make
measurements, and perform analyses demanded by policy makers and funders as part of
outcome and cost analyses.
A central problem is that evaluation researchers are not involved in program and policy design –
in the performance of the needs assessment, intervention theory development, and process design
work that I argue is needed to design policies and programs that have a chance of effectively
addressing the societal problems of interest to us.
Why is this the case in the United States?
It is the result of pressures with which you are very familiar:

Too little funding.

Too little time.

Inadequate numbers of trained and experienced evaluation researchers.

Lack of understanding of and/or respect for evaluation research.

Dominance of reliance on a dominant ex post economics-dominated view of policy
analysis.
Perceived Need for Evaluation Research in Brasil
Well, this is my understanding of evaluation research as it is practiced in the United States.
But, why I am speaking to you today here in the capital of the Federative Republic of Brasil?
My students and colleagues at the University of Brasilia, the Federal University of Goiás, ENAP,
and the recent annual EnANPAD congresso in Belo Horizonte have told me of their perception
of a need to expand the use of the evaluation research approach that I have described to Brasil.
This perception of a need for the assistance of evaluation research is reinforced by the facts that
Brasil has less than 30 years experience under its new constitution, and has seen dramatic policy
changes, particularly in social policy, over the past 15 years.
This perception is further reinforced by work that my colleague at UFG, Vicente Ferreira, and I
recently performed for our to-be-published article that compares policy and program analysis in
the U.S. and Brasil.
11

While we found that there has been substantial published work in this area in Brasil, there
is little evidence that the principles and practices of evaluation research that I have
described to you have been utilized in Brasil.
Thus, over the past two years I have working to bring my experience in evaluation research to
Brasil.

Last year and this year I have worked with my colleagues at UFG to establish a public
policy and administration center that, in part, will bring evaluation research to public
administration, particularly to municipal administration.

This year I taught a short course in evaluation research at UnB to doctoral and masters
students.

I have been invited by the João Pinheiro Institute and Getúlio Vargas Foundation to
discuss evaluation research.
A New Model of Policy Analysis “Grounded” in Evaluation Research?
As I presented Vicente and my article in Belo Horizonte a few weeks ago, and discussed its
implications with colleagues from around Brasil, our conversations returned to a central topic:

My assessment that policy analysis as it has been practiced in the United States and
Brasil needs to be examined with an eye toward to bringing to it the principles and
practices of evaluation research.
As I briefly noted in my opening comments concerning the recent Ben Bernanke interview, there
is a disconnect in the United States between policy development and analysis wherein the
impacts on the lives of Americans tends to get lost in the conversation.
I think that this situation is largely the result of the dominance of economics and economists, and
the public choice-oriented political scientists that they have influenced.

Thus, we see a reliance on make-believe macro level modeling most prominently seen in
the use of econometrics.
What I discussed with my colleagues at Belo Horizonte, and will continue to discuss at João
Pinheiro and Vargas, is the need for a new model of policy analysis in the U.S. and Brasil that
incorporates “grounding” from the principles and practices of evaluation research.
12
I argue that connecting evaluation research with policy analysis will make policy analysis much
more realistic.
I argue that a “grounded” model of policy analysis will more effectively connect it with the real
problems and experiences of Americans and Brasilians.
Yes, we have seen a growth in the use of meta-analysis in the search for evidence-based policies
and practices.

However, in that meta-analyses focus on research with particular methodological rigor,
they largely overlook the evidence that is produced in evaluation research.

So, I see meta-analysis as a weak substitute for the grounded model of policy analysis for
which I am arguing.
I hope that the time that I am sharing with you today will help support a continuing conversation
concerning the perceived need for evaluation research in Brasil and consideration of a new
grounded model of policy analysis.
Examples from David Crumpton’s Work
Since 1999, as I have worked at the Institute for Governmental Service and Research at the
University of Maryland, the Hatfield School of Government at Portland State University, and the
State of Maryland, I have participated in more that 75 evaluation research projects. This has
included work in the U.S. states of Oregon, California, Washington, Nevada, Minnesota,
Michigan, Indiana, Maryland, and New York. I have preformed work for state governments in
these states and for the U.S. federal government. I have also performed work for local
governments.
My hosts today have asked me to briefly discuss projects from my experience. They have asked
me to consider methodological issues and societal problems that evaluation research has studied.
I will consider two projects. One involved mental health problems and the other considered a
basic educational issue.
The United States faces a societal problem that Brasil and all other modern societies face – how
to address the needs of people who experience mental health problems. In the U.S. we have had
to face a particularly unfortunate aspect of this problem – the fact that many people with mental
13
health problems are found in our criminal justice system. One policy response to this problem is
a public program intervention that we refer to as the mental health court.
Mental health courts are intended to identify people in the criminal justice system whose
criminal activities are related to their mental health problems. Under the supervision of a judge,
these courts programs require that these people participate in mental health services. My
associates at the Institute for Governmental Services and Research were funded by the State of
Maryland and the U.S. Federal Government to evaluate the effectiveness of this intervention
intended to deal with this major social problem.
Our experience in evaluating mental health courts offers examples of several of the most difficult
methodological problems involved in evaluation research. The first and perhaps most important
problem that we faced was identifying reasonable outcomes to associate with participation in this
intervention. Mental health courts cannot cure their participants of their mental health problems.
Their ability to limit program participant contact with the criminal justice system is also
problematic. So, what can we expect mental health courts to accomplish? What can we measure
to show that mental health courts are effective?
Another challenge that we as evaluators faced involved access to data, and use of this data.
Mental health records in the U.S. are confidential. So, data that could support analysis of
important metrics associated with program outcomes were not available to the evaluators. This
situation was further complicated by the fact that multiple systems were involved in data
collection associated with program participants. Public and private mental health service
providers, police departments, corrections departments, and the courts – all of these systems had
different sets of data on the program participants. And none of this data was collected in ways
intended to support evaluation of mental health courts.
So, the evaluation team was not able to provide the type of information that could satisfy the
interest of policy makers concerning the effectiveness of mental health courts.
The second area of evaluation work that I will discuss with you involves another major problem
that the U.S. and Brasil face – school attendance. This problem in the U.S. is particularly intense
among poor and Black Americans. Again, the State of Maryland and the U.S. Federal
Government funded my team at the Institute for Governmental Services and Research to
compare three different approaches to reducing school absences among students in Maryland.
14
The three approaches that we studied represented different points on a continuum of intensity
and compulsion – from a voluntary, mediation-based approach, to one that included legal
consequences for students and their parents.
The most difficult problem that we faced in this project again involved data. We found that local
school authorities and individual schools were highly inconsistent in the quality of data that they
collected. We also found problems with basic data definitions across jurisdictions.
In the school attendance project we also found another problem frequently faced by evaluators –
lack of cooperation from important stakeholders in the interventions studied. In this case we
found that promoters of the three approaches that we studied wanted us to bias our assessment of
their programs to make them appear to be the most attractive among the options that were
considered. This made every step in the evaluation process difficult – particularly the preparation
of the final report.
These two cases highlight some basic problems in evaluation research. In neither of these
examples could the evaluators consider the application of the “gold standard” of social science
research – use of an experimental research design. Both of these cases exemplify a reason that
evaluation research tends not to receive much respect from academic researchers – evaluation
research is not neat and orderly. Evaluation researchers must adjust their methods to fit their
contextual challenges. Evaluation researchers must deal with the political and social system
realities of the contexts within which their research happens. This definitely does not fit the
orderly research model to which academic social science researchers aspire.
However, these two cases represent some of the many reasons that I like evaluation research.
Evaluation research embraces the disorderly social world in which we live and work as public
administrators. I like this!