Background Information
Operations
Research
Definition, Purpose and Applicability
by Lena Siegel
World Vision Institute
Am Houiller Platz 4
61381 Friedrichsdorf
Tel.: (06172) 763–176
Fax.: (06172) 763–270
[email protected]
Contents
Preface ............................................................................................................................................................................3
Abstract ..........................................................................................................................................................................4
1.
Why is it relevant to address Operations Research? .................................................................................5
2.
Scientific Background of OR.............................................................................................................................6
3.
2.1.
Purpose and Definition of OR ................................................................................................................6
2.2.
History of OR .............................................................................................................................................6
2.3.
Characteristics of OR ...............................................................................................................................7
Other Types of Research ............................................................................................................................... 12
3.1.
3.1.1.
Exploratory Research ................................................................................................................... 13
3.1.2.
Descriptive Research .................................................................................................................... 13
3.1.3.
Explanatory Research ................................................................................................................... 14
3.2.
4.
Applied Research .................................................................................................................................... 14
3.2.1.
Evaluation Research ...................................................................................................................... 14
3.2.2.
Action Research ............................................................................................................................. 15
3.3.
Other aspects of research .................................................................................................................... 16
3.4.
Differences between Evaluation Research and OR ........................................................................ 16
OR in the Field of Social and Health-related Research .......................................................................... 17
4.1.
Characteristics ......................................................................................................................................... 18
4.2.
The OR Process ...................................................................................................................................... 19
4.2.1.
Problem Identification and Design of Experiment ................................................................. 19
4.2.2.
Solution/Strategy Selection .......................................................................................................... 20
4.2.3.
Solution/Strategy Testing and Evaluation ................................................................................. 21
4.2.4.
Information Dissemination and Program Adaptation ........................................................... 22
4.3.
5.
Basic Research ......................................................................................................................................... 13
An Example of Social OR ...................................................................................................................... 23
Final Remarks .................................................................................................................................................... 29
Appendix 1.................................................................................................................................................................. 31
Operations Research Checklist ........................................................................................................................ 31
Appendix II.................................................................................................................................................................. 33
Annotated Bibliography ........................................................................................................................................... 37
2
Preface
At a time when the rationale for international development assistance has been seriously questioned
not only by people in donor countries but also by an increasing number of representatives of
recipient countries1, it seems increasingly vital to assess the impact of development programs. One of
the reasons for questioning development aid is the economic stagnation and social predicament in
which many of the poorest countries, notably in Africa, still find themselves despite decades of
concerted efforts to reduce poverty and injustice in the developing world.
The necessity of impact assessment has always been a crucial and yet ambiguous issue. On
the one hand, the development business has been subjected to a multitude of evaluation appraisals to
satisfy donor agencies trying to ascertain impact evidence. On the other hand, development
programs rarely have an inbuilt mechanism for ongoing program adaptation for the purpose of
ameliorating, let alone optimizing, their impact. Furthermore, many observers have satisfied
themselves with the finding that there was a perceptible impact at all, regardless of the costs,
according to the dictum: As long as there is a significant positive impact, we don’t question the costeffectiveness ratio. Since the development business is not subject to profit expectations by donors or
shareholders, the issue of cost-effectiveness has often been sidelined or even completely ignored.
Private non-governmental organizations such as World Vision have only fairly recently
recognized the importance of demonstrating not only the effectiveness of their programs but also
their cost-effectiveness, i.e. the efficiency of their approaches. This has led to the recognition that
evidence-based research must be drawn upon to ascertain certain development objectives: (1) to
improve or even optimize the effectiveness and efficiency of the programs; (2) to recognize that
there may be more than just one solution to a particular problem and to test these alternative
solutions; and (3) most importantly, to make ongoing adaption and amelioration an integral part and
parcel component of the programs right from the start. Few research methods will meet these
criteria. Operations Research (OR) claims to do this. Unfortunately, there is as yet a widespread lack
of understanding of what OR is and what criteria it must fulfill to be OR.
Encouraged by World Vision’s international Health, HIV/AIDS and WASH CoP Team, the
World Vision Institute has set out to define and explicate OR characteristics and to show its
applicability in the social and health field. I thank Lena Siegel for undertaking this important task. In
my view, she herewith submits an excellent paper of great practical relevance and applicability.
Kurt Bangert
Director Research
World Vision Institute for Research and Innovation
1
See, for instance: Dambisa Moyo: Why Aid is Not Working and How There is a Better Way for Africa, 2009.
3
Abstract
The goal of this paper on Operations Research (OR) is to provide a basis for further
research within World Vision’s programs. It should serve as a guideline for OR that is conducted to
improve the effectiveness and efficiency of the programs. Therefore, it is important to understand
the concept of OR in all its details.
The paper starts with a short introduction describing why it is important to address
Operations Research.
The second part covers the scientific background of OR including the purpose, the history
and the characteristics of OR. To apply this approach to social and health-related research it is
crucial to understand from which scientific field OR derives and which steps are essential while
conducting an OR study.
The next part demonstrates other types of research. These types either fall in the category
of Basic Research or Applied Research. This chapter should contribute to a better distinction between
different types of research and to highlight the difference to OR.
The fourth part concentrates on OR conducted in social and health-related research and
displays the specific characteristics as well as the process of OR adapted to this field. This part
concludes with an example of an OR process in the field of primary health care.
The paper ends with final remarks about which minimum standards studies must meet to be
classified as OR.
4
1. Why is it relevant to address Operations Research?
Looking at the research conducted to measure the performance of programs that are
implemented in the development field, we more and more see that it is classified as “OR”. But what
is behind this concept, which comes from the field of mathematics, informatics and economics? What
are the steps to be taken in order to conduct this kind of research? And how can this approach be
adapted to the social and health-related field? What is the purpose of an OR study and how does this
concept differ from other types of research?
When conducting an OR study, we must answer each of these questions. With this paper I
hope to contribute to a deeper understanding of the components of OR and to its adaptation to the
social and health-related field.
5
2. Scientific Background of OR
In this part of the paper, I will describe the scientific background and the characteristics of
Operations/Operational Research. I will use the term “Operations Research”. The term
“Operational Research“ is primarily used in the UK.
2.1. Purpose and Definition of OR
As the name implies the basic definition of OR is “research on operations”. OR is applied to
complex problems about how to conduct and coordinate the operations and activities within an
organization. Organizations may seek a very wide range of operational problems in terms of
efficiency, customer service, quality or cost.
The objective of OR is to find practical solutions to operational or strategic problems to
enhance the effectiveness of the whole organization. Thus, OR addresses the practical management
and needs of the organization and therefore must provide understandable information on how to
overcome the problem for the decision makers.
Furthermore, rather than just improving the status quo, OR aims at finding the best solution
amongst a variety of possible solutions, and that best solution is called the optimal solution. The goal is
to find a best possible course of action, which means the “search of optimality”. Usually, there will be
multiple solutions that need to be tested in order to find out the optimum.
OR is being used in diverse areas such as manufacturing, transportation, construction,
telecommunications, financial planning, healthcare, the military and public services. Thus, there exists
a wide field of application for OR.
The Association of European Operational Research Societies (EURO) states that OR can be
described as a scientific approach to the solution of problems in the management of complex systems
(cf. EURO 2011). The German Society of OR (GOR) has a more general definition as it defines OR
as “the development and the application of quantitative models and methods to support decision
making.”(GOR 2011).
2.2. History of OR
The roots of OR can be traced back many decades. Since the industrial revolution,
organizations have grown in complexity and size. The division of labor and the segmentation of
management responsibilities started. This development created new problems. The different areas of
an organization grew into relatively autonomous areas with its own goals and values. Thus, it may
happen that different areas work for their own goals and loose the overall objectives of the whole
organization so that different areas end up working at cross-purposes.
6
“A related problem is that as the complexity and specialization in an organization
increase, it becomes more and more difficult to allocate the available resources to the
various activities in a way that is most effective for the organization as a whole.”
(Hillier & Lieberman 2010, p.1.)
From these kinds of problems and the need to find a better way to solve them OR emerged.
Generally, World War II is referred to as the beginning of OR because there was a need to
allocate scarce resources to the various military operations and to the activities within each
operation in an effective way. Therefore, the British and U.S. military management called upon
scientists to apply a scientific approach to dealing with this and other strategic and tactical problems.
They were doing research on (military) operations.
At the end of war, the success of OR resulted in using OR outside the military as well. The
problems caused by the increasing complexity and specialization in organizations were again arising. It
became clear that these problems were the same that had been faced by the military but in a
different context. In the early 1950s, people who were engaged in OR during the war had introduced
the use of OR to a variety of organizations in business, industry and government. The rapid spread of
OR soon followed.
Nevertheless there are also other important factors that played a key role in the rapid
growth of OR. First of all, a remarkable progress in improving the techniques of OR had been made.
Many of the standard tools such as linear programming, dynamic programming, queuing theor and
inventory theory were developed during that time. Second, the computer revolution played a key
role in the development of OR because a large amount of computation is usually required to deal
with the complex problems typically considered by OR.
In the 1980s OR got a further boost because of the development of more powerful personal
computers and good software packages for doing OR. The progress further accelerated in the 1990s
and into the 21st century. Today, millions of people have access to OR software.
2.3. Characteristics of OR
OR has some specific characteristics that are important for planning and conducting it.
Quantitative mathematical techniques are the main methods to conduct OR. But that does not mean
that a practical OR study is primarily a mathematical exercise. The mathematical analysis forms only a
small part of the total OR study. Therefore, a group of individuals with diverse backgrounds and skills
is needed to execute OR. No single individual can be an expert on all the many aspects of OR work
and problems. The OR team typically needs to include individuals who are highly trained in the fields
of mathematics, statistics, probability theory, economics, business administration, computer science,
7
engineering, physical sciences, behavioral sciences and the special techniques of OR. However, which
experts will actually be included certainly depends on the nature of the problem and the resources
available to solve it. OR uses an approach that resembles the way research is carried out in
established scientific fields. This approach is often referred to as the “scientific method”.
A typical OR study can be divided into six major phases that may overlap each other
(cf. Hillier & Lieberman 2010, p. 8):
1. Defining the problem
of interest and gathering
relevant data
2. Formulating a
mathematical model to
represent the problem
6. Implementation
3. Developing a computerbased procedure for
deriving solutions to the
problem from the model
5. Preparation for the
ongoing application of
the model as prescribed
by management
4. Testing the model
and refining it as
needed
Although the six steps may not all together be applicable to the social field, in what follows here we
are nevertheless giving a brief description of what typically constitutes OR:
Phase 1
The first phase in conducting an OR study consists of the careful observation and formulation
of the problem including gathering all relevant data. In this first step, it is crucial to develop a welldefined statement of the problem. The statement must include relevant considerations of the
problem such as appropriate objectives, limitations on what can be done, the context of the problem
and possible alternative courses of action.
Another important part of the first phase is the involvement of the decision makers. An OR
study has normally the goal to advise the management. The OR team first performs a detailed
8
technical analysis of the problem and then presents recommendations to the management. The
management will make the final decisions. Therefore, it is important for the OR team to cooperate
with the management and to identify the right problem from management’s point of view. The OR
team needs the support of the management for the implementation of the study.
Additionally, OR is concerned with the welfare of the whole organization. But often
problems only concern one component of the organization. In this case, it is the challenge for the OR
team to formulate the objectives as specific as they can be while still incorporating the main goals of
the decision makers and the overall objectives of the organization.
Gathering all the relevant data that is necessary to conduct the study is very time-consuming.
The research team needs much data to gain an accurate understanding for the development of a
mathematical model in the next phase.
In many cases, it will be necessary to create a Design of Experiment. Equipped with such a
Design, researcher will test certain assumptions. For more information on Design of Experiment, pls.
go to p. 20.
Phase 2
In the second phase, the OR team has to develop a model that represents the existing
problem for further analysis. It is hypothesized that the model represents the real problem for all
practical purposes so that the conclusions and solutions can be adapted to the real problem and are
valid for it. Typically this model has to be a mathematical model2 and requires the formulation of a
mathematical function (Figure 1). The function consists of equations and mathematical expressions.
Therein, the decisions to be made are represented as variables (x1, x2, …, xn). The values of the
variables have to be identified. The goal might be to choose the values of the variables in a way to
maximize the whole mathematical function (e.g. maximize the profit) or to minimize the function (e.g.
minimize the number of rejections). Assigned appropriate values to the variables of the model is a
challenging process. It is important to really understand how the solution will change if the values for
the variables are changed because often the values are only estimated from the data.
In this context it is important to distinguish between variables and parameters both of which
may figure in the model. Variables are concepts whose numerical value is under the control of
management. Parameters are concepts whose numerical values are given by the environment or
outside forces. The value of parameters can be measured but not changed. The value of variables can
be set.
„ [Such] models are invaluable for abstracting the essence of the subject of inquiry, showing interrelationships, and
facilitating analysis. Mathematical models are [also] idealized representations but they are expressed in terms of
mathematical symbols and expressions.” (Hillier & Lieberman 2010, p.11)
2
9
Figure 1: This figure shows a mathematical model with three variables (retrieved from Stat-Ease).
Phase 3
In the third phase of the OR process the goal is to develop a procedure for deriving
solutions to the problem from the model. That is usually a computer-based procedure that works
with algorithms. OR has the goal to search for an optimal solution but in practice the researchers
have to keep in mind also the cost-benefit ratio. At some time, the decision makers want to see
results from the study and do not want the OR to be too time-consuming and cost-intensive.
Time and work load may be a constraining factor – not for the OR-modeling itself, but for
data acquisition needed to provide key information to run the model. So in practice and for various
reasons, it is sometimes not possible to find the optimal solution. While it is still the goal for OR to
find the optimal solution, in light of existing constraints, however, one will sometimes have to settle
with a good suboptimal solution.
After finding a solution, it is important to do further analyses (postoptimality analysis). In that
kind of analysis the question is what would happen to the solution if different assumptions are made
about future conditions. It is a process of experimenting with changes in the model by testing
different values of the variables.
Phase 4
The fourth phase consists of testing the whole model. In order to avoid mistakes,
researchers sometimes vary the values of parameters and/or the decision variables in order to check
whether the output from the model behaves in a plausible manner. The researchers try to find and
correct as many errors as possible. This process is also called validation or model validation because it
is the testing and improving of the model to increase its validity. It is important to document the
process for future use of the model.
10
It is worth noting that none of the tested options may turn out to be the optimal solution.
Instead, OR may find the optimal solution without actually having tested it. (For instance, if one has
tested for a variable equal to 1, 3 and 5, OR may determine the optimal solution for that variable to
be 2.4.)
Phase 5
After testing and improving the model, the next step is to install a system for applying the
model. This system usually includes the model, solution procedures and operating procedures. This is
usually a computer-based system that consists of many different computer programs (e.g. databases
that provide up-to-date data for the model or programs that automatically interpret the output of
the model). The system for application can be called upon even when personnel may change. Once
the system is well in place, variables can easily be adjusted whenever parameters may have changed.
This way, an optimal solution (optimal efficiency/effectiveness) can be achieved even under altered
environmental circumstances (i.e. changed parameters).
Phase 6
In the final phase of the OR process the implementation of the system is carried out. It is
important that the OR team participates in this phase to make sure that the whole system works.
This phase depends greatly on the support of the management. Therefore, it is crucial to involve the
management in the whole process of OR. The system may be used for years. Thus, it is crucial to
continue to obtain feedback on how well the system is working. If necessary, modifications in the
system can be made. In the whole OR process it is very important for the research team to
document each step clearly for any future work. Concluding the description of the different phases of
an OR study, it is important to state that it is impossible to write down any standard procedure that
should always be followed by OR teams because OR requires innovation.
Finally, I will conclude with my own definition of OR trying to incorporate all the aspects of
OR that I mentioned above and trying to define OR in a short but precise way. It is an attempt to
make it concrete and useable for the practical work:
OR is the process of identifying and analyzing a problem within the operations
(activities) of an organization or a program to be set up or already in place, and
of finding an optimal solution for this problem by testing alternative ways of
action (models) which will serve as a basis for decision making regarding
successive modifications of key variables to achieve the overall goal of enhancing
or even optimizing the efficiency and effectiveness of the program or
organization.
11
3. Other Types of Research
It is not easy to make distinctions between different forms of research and research
methods. The literature shows that there are many different terms used for the same types of
research and that there does not really exist a consistent terminology. Another problem in the
distinction of different kinds of research derives from the mix-up of research designs and research
methods. This causes the sometimes incorrect use of these terms and complicates a distinction. In
the first part of this section, I will try to differentiate between the different types of research
regarding their purpose. But we have to keep in mind that any research project can have more than
one purpose.
First we can make a distinction between Basic Research and Applied Research. Basic
Research is designed to add to our fundamental understanding and knowledge of the social world
regardless of practical or immediate implications, whereas Applied Research is intended to be useful
in the immediate future and to suggest action, or to increase effectiveness in one area. Applied
research is designed to provide information that is immediately useful to those participating in
institutions or programs. This kind of research can be done for organizations and communities or
with them. So also OR can be classified as Applied Research.
Emily Stier Adler and Roger Clark stated that “a great deal of research can be seen as falling
somewhere between the poles of basic and applied research” (Stier Adler & Clark 2011, p. 382).
That statement shows the difficulty of distinguishing different forms of research from each other.
Nevertheless, I assigned the following types of research to one of these forms regarding their
purpose and intention.
Applied
Research
Basic
Research
Descriptive
Research
Explanatory
Research
Exploratory
Research
Operations
Research
Action
Research
Evaluation
Research
12
3.1. Basic Research
3.1.1. Exploratory Research
Exploratory Research can be described as research on a relatively unstudied topic or in a
new area. This kind of research uses an inductive approach. The researcher starts with observations
about the subject of study and tries to develop tentative generalizations about it. Therefore, data is
collected from only a few cases through observations or in-depth interviews. That qualitative data is
analyzed and interpreted. During the research process, the researcher also tries to spot themes that
emerged from the data for further research.
Example: Managing motherhood from prison.
Definition:
Exploratory Research is research mostly inductive on an unstudied topic or in a new area
through gathering and analyzing qualitative data that is gained from a few cases, with the
overall goal to become familiar with this area, to develop some general ideas about it, to
generate theoretical perspectives and statements on the topic and to spot themes for
further research.
3.1.2. Descriptive Research
Descriptive Research intends to describe groups, activities, situations or events with a
focus on structure, attitudes or behavior. Typically, the researcher already has a general
understanding of the topic before he/she starts to collect the data. The overall goal of this kind of
research is to gain an accurate and precise picture of the topic under study. It describes how things
are or how things have changed over time. Typically, the data is collected through surveys or content
analysis and involves a large number of cases, which requires the use of quantitative data analysis.
Example: Census.
Definition:
Descriptive Research is designed to describe a topic with a focus on structures, attitudes
and behavior, about which the researcher typically already knows something before the
data collection, which is done by gathering and analyzing data about a large number of
cases using quantitative data analysis with the overall goal to gain an accurate and
precise picture of the topic.
13
3.1.3. Explanatory Research
Explanatory Research is conducted to explain why things are the way they are, and
therefore looks for causes and reasons. This kind of research uses a deductive approach by moving
from statements that are more general to less general. Thus, in explanatory research, pre-existing
theories are often used to develop first ideas of a possible explanation for a question and then
further data is collected to verify or modify the explanation. This kind of research typically requires a
large number of cases so the data is collected using quantitative methods like surveys, and is then
further analyzed.
Example: Why are some people more likely than others to do work for pay after they retire
from their regular lifetime work?
Definition:
Explanatory research is intended to explain variations in subjects by identifying causes
and reasons for these variations using a deductive approach which implies consulting preexisting theories to identify the variables that influence the variations and to develop
preliminary explanations which are then further proved by collecting data from a
relatively large number of cases and using quantitative data analysis to finalize the
explanations.
3.2. Applied Research
3.2.1. Evaluation Research
Evaluation Research is a form of applied research because it is conducted to provide organizations
with information that can be used in the immediate future. Evaluation research has the purpose of
assessing the impacts of programs, policies and legal changes. Often it focuses on whether a program
or policy has reached the intended outcomes. The results and implications of an evaluation research
have to be communicated to the clients or decision-makers for further planning and continuance of
the program.
Within evaluation research, we can further find different types of evaluation research
regarding their specific purpose and focus: Formative and Summative Evaluation, Outcome and Process
Evaluation, Needs Assessment and Cost-Benefit Evaluation. Formative evaluation focuses on the design or
early implementation stages of a program or policy. Outcome evaluation, which is also referred to as
summative evaluation or impact evaluation, is designed to assess the effects or impacts of a program
or policy and to determine whether the intended goals are accomplished. Needs assessment can be
described as an analysis that determines the needs of a geographic area or a specific population.
Cost-benefit evaluation is research that compares a program’s cost to its benefits.
14
Definition:
Evaluation Research is a complex form of research that can have in itself many different
purposes but its overall goal is to assess the impacts of programs, policies or legal changes
and to determine whether they have achieved its intentions with the practical focus to
communicate the results and their implications to clients and decision-makers for the
future development of the program or policy.
Formative
Evaluation
Process
Evaluation
Outcome
Evaluation
Evaluation
Research
CostBenefit
Evaluation
Needs
Assessment
3.2.2. Action Research
Action Research (synonyms: participatory research or participatory action research) has the focus
to influence or change some aspect of the subject of research. Improvement as well as involvement is
an essential part of action research. Central to this kind of research is the collaboration between
researchers and those who are the focus of the research (e.g. community) and their participation in
the process. The people in the organization or community under study participate actively with the
professional researcher. They are involved in each step of the research process from the initial
design to the final presentation of the results and the discussion of action implications. The action
that comes from the research is intended to help the community or organization rather than the
research community. Action Research has a social purpose component namely the empowerment of
low status people in the organization or community to make decisions and take actions. Those who
are normally the subject under study are actively engaged in the research process. The focus of
action research is often an evaluation, so there are links between these two types of research.
Therefore, some particular variants of evaluation have the same goals as action research (e.g.
participatory evaluation, empowerment evaluation).
15
Definition:
Action Research is intended to promote a change in an aspect of the subject of research and
is characterized by the participation of the people who are directly affected by the research
in the whole research process to empower them in decision-making and action taking.
3.3. Other aspects of research
Based on which kind of purpose the research will cover, the researcher will decide which
research design and research methods are appropriate. In short, research designs can range from
non-experimental to true-experimental and can be cross-sectional, longitudinal or case study designs.
Regarding the methods of research, one can in general differentiate between qualitative and
quantitative methods. Quantitative research methods are typically methods that involve large
numbers of cases such as surveys, questionnaires, standardized tests and attitude scales that need to
be analyzed using statistical methods. In qualitative research, the researcher is more interested to
really understand individual cases instead of questioning high numbers of people. Methods that are
typically used in qualitative research involve individual or group interviews, focus group discussions
and observations. Nevertheless, it is possible and reasonable to combine quantitative and qualitative
research methods.
3.4. Differences between Evaluation Research and OR
In considering each definition of the different types of research, I think it is easier to see the
difference between OR and the first three types of research (exploratory, descriptive, explanatory
research). Because evaluation research is often used in social and health-related fields I want to
highlight this distinction again. Evaluation aims to measure the impact of a program or strategy.
Generally, there is one solution/strategy implemented to overcome a problem and to reach a desired
outcome. With an evaluation one wants to assess whether that solution/strategy was successful for
achieving the desired outcome or not. In OR however, several solutions/strategies are tested and
modified to find the optimal solution to overcome the problem. This optimal solution is then
implemented to enhance the effectiveness and efficiency of the program. Another important
distinction between Evaluation Research and OR is that typically an evaluation does not entail a
control or comparison group, whereas OR ideally makes a comparison between a target group and a
control group. In Evaluation Research a change of the program may or may not be one of the
consequences of the research; in OR, however, the change or adaption of the program is part and
parcel of the research. Evaluation Research aims to take a retrospective look at a program to assess
its impact3; whereas OR is characterized by a prospective orientation and focus on action-oriented
decision making and management (International Healthcare Consultants 2011).
3
However, when an evaluation contains both a prospective baseline survey and a retrospective follow-up
survey, then such evaluation may, at least in part, be a prospective exercise.
16
4. OR in the Field of Social and Health-related Research
In the field of social and health-related research, OR becomes more and more important.
Programs and projects that are implemented to improve the health and nutrition situation, the
educational situation and the social situation need to be effective and efficient because NGOs and
other social services providers need to maintain and strengthen their legitimacy to operate. These
programs always face complex tasks. Therefore, detailed planning at all levels, close coordination of
program implementation efforts, careful training and supervision of personnel, and continuous
evaluation of program development and impact is needed. OR is a way to support and inform these
essential planning, coordinating, training, and evaluation functions.
The Global Fund to Fight AIDS, Tuberculosis and Malaria, the Special Program for Research
and Training in Tropical Diseases (TDR) and an inter-agency technical working group developed a
Framework for Operations and Implementation Research in Health and Disease Control Programs.
The framework was finalized within a meeting in April 2008 attended by over fifty participants
representing the Global Fund, TDR, the World Health Organization, the Joint United Nations
Program on HIV/AIDS (UNAIDS), United States Agency for International Development (USAID), the
World Bank, field-based programs, policy-makers and research communities from all over the world
(cf. Global Fund 2008, p.3).
The participants in this meeting agreed on a rather broad definition of OR that is used in this
framework:
“Any research producing practically-usable knowledge (evidence, findings, information, etc)
which can improve program implementation (e.g., effectiveness, efficiency, quality, access, scaleup, sustainability) regardless of the type of research (design, methodology, approach) falls within
the boundaries of operations research.” (Global Fund 2008, p.12)
The scope of this paper is to describe the characteristics of OR, its process and especially to
distinguish it from other forms of research. Therefore, this definition is a bit problematic because it is
a more universal definition that may apply to all forms of applied research as described in the
previous chapter and which makes a distinction difficult. For this reason, I think other definitions of
OR (e.g. from WHO or Fisher et al.) are more useable in the practical work. Also with my own
definition I tried to develop a more specific definition:
OR is the process of identifying and analyzing a problem within the operations
(activities) of an organization or a program to be set up or already in place, and
of finding an optimal solution for this problem by testing alternative ways of
action (models) which will serve as a basis for decision making regarding
successive modifications of key variables to achieve the overall goal of enhancing
17
or even optimizing the efficiency and effectiveness of the program or
organization.
In this part of the paper I will try to describe the characteristics and methods of OR in the social and
health-related field. From my point of view, OR in these fields needs to be adjusted but still has to
meet minimum standards to be classified as OR.
4.1. Characteristics
The World Health Organization defines OR as “the use of systematic research techniques
for program decision-making to achieve a specific outcome. It provides policy-makers and managers
with evidence that they can use to improve program operations (WHO 2003, retrieved from Global
Fund 2008, p. 49).”
OR in this field focuses on specific problems within the operations or activities of the
programs that are controlled by managers and administrators. Operations or activities in this field
could be for training of personnel, logistics, public information or counseling.. Through OR, program
problems can be identified and solutions can be developed through comparing different approaches
against another to increase the efficiency, effectiveness, quality of services delivered by providers and
the availability, accessibility, and acceptability of services desired by users (cf. Fisher & Foreit 2002,
p.2). The practical objective of OR is to provide managers, administrators and policymakers with the
information they need to apply the results of the OR to the programs. It helps them to design and
implement new programs and to review, redirect and restructure programs that have already been in
place (cf. Population Council 2002, retrieved from Global Fund 2008, p. 49). The goal should be to
identify as early as possible any shortcomings of a program that may affect the expected outcomes
(The International Food Policy Research Institute 2005, retrieved from Global Fund 2008, p.49).
OR utilizes systematic data collection to accumulate evidence for supporting decision-making.
But there is no single set of methods or designs that are unique for OR. Especially in the social and
health-related field, the applied methods range from the qualitative to the quantitative, from the nonexperimental to the true experimental. It is not the application of a particular set of methods or the
preference of one design over another that distinguishes OR from other forms of research; to a
greater extent, it is the focus or objective of the research that distinguishes OR from other forms of
research, namely to obtain a better understanding of the operations of a program so that needed
improvements can be made to strengthen the program and maximize its effectiveness (cf. The
International Food Policy Research Institute 2005, Global Fund 2008, p. 49). An OR study is only
successful if the results are used to make program decisions.
To conduct an OR study, collaboration between managers and researchers is necessary
during the whole research process to identify the research problem, to develop the study design, to
18
implement the study and to analyze and interpret the results. Furthermore, an interdisciplinary
research team is required that ideally should include professionals from diverse fields such as
mathematics, statistics, economics, sociology, anthropology, management, information technology,
health sciences and social sciences.
4.2. The OR Process
In the following I shall describe the process of OR in health and development fields as follows:4
1. Problem Identification and Design of
Experiment
2. Solution/Strategy Selection
3. Solution/Strategy Testing and Evaluation
4. Information Dissemination and Program
Adaptation
4.2.1. Problem Identification and Design of Experiment
The most important task for starting the OR process is to identify and clearly formulate the
problem. That step is important because it leads to the statement of research objectives, to the
hypotheses, to a definition of key variables and to a methodology for measuring the variables. It is
crucial to analyze the problem in detail to define its scope, magnitude, seriousness, characteristics
and probable causes. Therefore, the researchers first need to gather all information and data about
the problem, which is already available. It can be useful to review relevant literature, examine current
statistics, seek educated opinions from persons concerned about the problem and obtain probable
reasons for the problem from theory. A good diagnosis of the problem will help to establish its
parameters and variables. In this first step of the OR process, it is important to involve program
managers but also other people such as village chiefs, teachers and health personnel. This ensures
that the researchers really understand the problem as a whole. Another benefit of early involvement
4
Fisher & Foreit (2002) distinguish the last phase in information dissemination and information utilization.
19
of all key stakeholders in the OR process is that it increases their interest later in reviewing and using
the results.
Once the problem has been identified, the researchers and decision makers need to
determine the desired objectives, which should be achieved through implementing one of the
alternative solutions. Only after accomplishing the objectives one can search for various possible
alternative solutions.
Following the definition of the problem and the determination of the desired objectives, it is
useful, if not mandatory, to perform a Design of Experiment. In fact, determining the desired objectives
may also be considered part of such a design. A Design of Experiment is needed for testing
hypothetical assumptions. With their experiments, researchers will try to establish a causal
relationship between variables. A well-designed experiment will be robust under questioning and will
yield good statistical data. A good Design of Experiment is needed because experiments can be costly
and may not be repeated just because the experiment did not yield good statistical data or because a
control group was not really useful for comparison. Building a design means carefully choosing a small
number of experiments that are to be performed under controlled conditions. A Design of
Experiment will usually involve three interrelated steps

Define the objective of the experiment

Describe the desired outcome variables (response variables)

Define the variables to be controlled during the experiment (design variables)
The experiments designed in such a way may quite often have to be carried out first before a
mathematical model is developed. But to actually do the right experiments and the right amount of
experiments, a Design of Experiment is advised to be done at the beginning.
4.2.2. Solution/Strategy Selection
Developing a Model
In the first step of this phase a model of the problem needs to be constructed that
represents the problem and the expected outcome. The model should describe the relationships
between different variables that are considered relevant to the problem. It is expressed in terms of
objectives and hypotheses about the problem. If it is possible, the model should be a mathematical
model consisting of equations and mathematical operations. That can make it easier to test different
solutions to the problem. It is important to define the key variables of the model because that is a
prerequisite for the measurement of the variables and gives meaning to them. It is also necessary
because otherwise the variables might be interpreted in different ways. Additionally, it is necessary to
clearly define the terms of the hypothesis (e.g. meaning of terms like more than, significant,
increased) that describe the relationships between the variables. Therefore, a clear terminology is
crucial.
20
Alternative Solutions
Another step in this phase is the development of solutions to overcome the problem. It is
crucial to first identify the problem and develop a model of it to select the solutions on this basis. It
can be useful to review existing theories or strategies that other people have already used to solve a
similar problem. It can be also useful to consult with people who are affected by the problem
because they often have good ideas for strategies to solve it. At this point, it is also important to
look for strategies that are possible and manageable for the organization to implement in terms of
facilities, training, personnel, time, finance, logistics and management, and that can be sustained over
time. The strategy needs also be acceptable for the key stakeholders and the community, and needs
to be in line with the organization’s goals and objectives.
When an approach is identified that may useful to overcome the problem and to reach the
desired goals, the research team has to think about alternative solutions, which are in line with this
overall approach. So the alternative solutions differ from each other in terms of changes in the
variables such as the number of visits to pregnant women by Community Health Workers or the
interval of training for Community Health Workers or both.
4.2.3. Solution/Strategy Testing and Evaluation
In this phase of the OR study the alternative solutions need to be tested to find the optimal
solution to overcome the problem and achieve the desired outcome. Before testing the solutions the
researchers have to think about how to measure the outcome of each alternative solution to later
make a decision which solution is the optimal one.
If the problem can be represented by a mathematical model, mathematical methods could be
used to find the optimal solution as described in chapter 2 to measure the outcome. However when
OR is conducted in the social and health-related field, it might not always be possible to develop a
mathematical model to test alternative solutions.
Therefore, it may be necessary to immediately apply the selected alternative solutions and
test them directly in the field. In this case, it might be more difficult to measure the outcome such as
the effectiveness of a primary health care program. The researchers have to determine criteria that
show the effectiveness of the program. These criteria can be changes in attitude, behavior and
knowledge. After defining criteria, measures need to be identified to determine the effectiveness. For
example, community health workers are appointed to pass on knowledge about HIV/AIDS
transmission to pregnant women. That can be measured through a questionnaire or an interview
during which questions regarding HIV/AIDS transmission are posed. The number of right answers
will in this case demonstrate the knowledge acquired (8 or more = High knowledge; 4-7 = Medium
knowledge; 1-3 = Low knowledge; No correct answers = No knowledge).
In the case that there are so many alternative solutions that it is not possible to test them all
directly in a field situation the researchers may need to go another way. They can estimate the
21
outcomes of each alternative using previous experience from the implementation of similar
strategies. On the basis of this knowledge they can rank the alternatives related to the desired
outcome(s). After this ranking, they can conduct a field test for example of the three best
alternatives to test them in real situations.
At the end of the testing, the alternative solutions need to be compared to identify the most
optimal solution in terms of achieving the desired outcome(s). However, one cannot expect that one
of the tested solutions will turn out to be the optimal solution. Rather, it is the model which, given
the results of the tests, will calculate the optimal solution.
The last step in this phase should be the sensitivity analysis to assess the reliability of the
findings. As the researchers analyzed the problem and developed alternative solutions to overcome
it, they made assumptions about uncertain quantities. For example they have to estimate certain
costs (e.g. costs for personnel or items) and outcomes (e.g. births, deaths or dropouts). Before
making recommendations to the decision makers, they should test whether variations in those
assumptions would change the solution they would recommend to the decision makers. If this
analysis indicates that some variables are sensitive to changes in the assumptions, the researchers
should discuss that topic with the decision makers in the next phase.
During the third phase, it is also important to consider ethical issues. This is especially true
when it is not possible to test alternative solutions through mathematical methods. In that case, it is
necessary to test the solutions in the field so that the most effective in terms of overcoming the
problem and reaching the objective can be identified. We have to keep in mind that when testing
different strategies, some target groups may not benefit from optimal services. On the other hand,
only through testing is it possible to identify the most effective strategy, which can then be
implemented on a grander scale.
4.2.4. Information Dissemination and Program Adaptation
Because OR is primarily conducted to inform program or policy decision makers about the
changes needed to optimize impact, it is crucial to present the findings to them. It can even be very
helpful to have frequent small meetings with key managers and stakeholders throughout the research
process to keep them informed. The researchers also have to think about an appropriate way to
present the findings to the decision makers.
Finally, once the tests have been carried out and the model has calculated the best solution
under the design of the experiments, the decision makers will need to take the necessary steps to
implement that solution. The solution to be implemented may not be the best solution under any
circumstances but only the best solution given certain constraints. Such a constraint could be a
budgetary constraint which would set upper limits to certain experiments and, hence, the solution to
be found. But these constraints should be considered before the experiments are designed. Once the
22
experiments have been carried out, a number of variables have been tested, and an optimal solution
has been calculated, decision makers should implement that solution.
It may be noted that the process by which the optimal solution is put into operation may be
considered change management; in exact parlance, it is no longer part of OR per se; however, since
OR is to be carried out not for academic reasons but for the purpose of actually achieving an
optimization of impact, that process certainly must be a necessary consequence of OR and has
therefore been considered part of OR.
4.3. An Example of Social OR
An example of social OR is the “Kenya Adolescent Reproductive Health Project”5 that was
conducted from 1999 until 2002 by the “Frontiers in Reproductive Helath Population Council” in
collaboration with the “Program for Appropriate Technology in Health” (PATH) and the
Government of Kenya, with funding support from USAID. This project is a multi-sectoral approach
to providing reproductive health information and services to young people in Western Kenya.
Background
Informing adolescents about appropriate and acceptable behaviors, and ways to protect
themselves against unwanted and unprotected sex has proved problematic in Kenya. Education
programs for in- and out-of-school adolescents are lacking, and there is controversy around
providing services to sexually active adolescents and a pervasive concern that sexuality education and
contraceptive services may lead to promiscuity.
In response to this situation the Population Council and the three Kenyan ministries planned and
designed this OR project.
1. Problem Identification
The researchers started with a diagnostic phase to understand adolescents’ reproductive
health issues. Therefore, they first reviewed existing statistics that showed that many Kenyan youth
experience reproductive health problems. After that, the researchers tried to understand
adolescents’ reproductive health issues in the socio-cultural context of the two districts where the
study was to be conducted through:



Securing clearance/approval from community gatekeepers
Securing gatekeepers’ support and commitment
Creating an inventory of existing health facilities and providers, schools, adult, and youth
networks, and agencies providing health and youth services in the intervention sites
5
A Multi-Sectoral Approach to Providing Reproductive Health Information and Services to Young People in
Western Kenya: Kenya Adolescent Reproductive Health Project.
23




Conducting interviews and discussions with community target groups, including religious
leaders, teachers, health providers, parents, and community leaders to assess their
perceptions of and attitude towards adolescents reproductive health issues and to get their
advice on what should be emphasized in the intervention
Conducting interviews with service providers, especially private service providers, to assess
their ability and willingness to participate in the study and any constraints that may hinder
their successful participation.
Interviewing youth leaders, teachers and health providers to identify information, education
and counseling material needs and reproductive health training needs for various categories
of health and educational service providers.
Assessing existing adolescents’ (10-19 years) utilization of reproductive health services
through a review of clinical and community-based providers’ service data in both control and
intervention sites.
Summing it up, the results from the diagnostic phase were that many Kenyan youth face reproductive
health problems such as:

unwanted pregnancy

unsafe abortion

pregnancy-related school drop out

high age-specific fertility rates

sexually-transmitted diseases, including HIV/AIDS
Additionally, the researchers identified a number of factors that caused these problems:

early onset of sexual activity

low use of contraceptives and STD/HIV/AIDS preventive measures

lack of correct and reliable sources of reproductive health information which leads to poor
reproductive health knowledge

poor access to convenient, confidential and low cost reproductive health services

lack of effective prevention programs for youth in and out-of-school

the wider environment such as religious and community leaders is not supportive of youth
acquiring reproductive health knowledge and services.
2. Solution/Strategy selection
Resulting from the problem identification and description of causes the researchers defined
the overall objective of the OR project as to determine the feasibility, effectiveness and cost of a
systematic intervention to foster a supportive environment that will address reproductive health
problems of both in-school and out-of-school adolescents (10-19 years) by making existing services
more accessible to adolescents and youth, providing them with reproductive health information and
skills. The initiators of this study wanted to obtain sufficient information to guide the future policies
of the Ministries regarding provision of youth reproductive health education and services.
24
Resulting from the reproductive health problems Kenyan youth face and their causes, the
researchers identified three specific goals that needed to be addressed by possible interventions:

Improve knowledge about reproductive health and encourage a responsible and healthy
attitude towards sexuality among adolescents

Delay the onset of sexual activity among younger adolescents

Decrease risky behaviors among sexually active adolescents
To reach these goals a supportive environment at all levels was identified as being crucial. The
researchers hypothesized that there are three fields that are important in which to carry out the
interventions: the environment, the health services and the school. In detail, the hypotheses are as
follows:
Environment
a) Providing adolescents with reproductive health knowledge and skills at the community will
increase reproductive health knowledge, preventive behaviors and health seeking behavior
among both married and unmarried male and female youth (10-19 years).
Health Services
a) Training health care providers on provision of youth friendly services will improve
providers’ attitudes towards provision of reproductive health services to sexually active
adolescents and the quality of reproductive health services they provide to the youth.
b) Strengthening the ability of health care providers to provide youth friendly reproductive
health services will increase married and unmarried male and female adolescents (10-19
years) utilization of reproductive health services.
School
Providing adolescents with reproductive health knowledge and skills at school will increase
reproductive health knowledge, preventive behaviors and health seeking behavior among
both male and female youth (10-19 years).
The researchers expected that advocacy in support of adolescent access to reproductive health
information, counseling and services through sensitization workshops and public meetings targeting
opinion leaders and parents will increase community support for provision of adolescents’
reproductive health services in the intervention sites. Therefore, the following interventions
(solutions) to the problem were designed with regard to all of the three fields:
Environment:



Sensitization, advocacy and community mobilization targeting gatekeepers and opinion
leaders
On-going community campaigns on reproductive health issues
Training of peer educators and adult helpers
25
Health Services:


Training health providers
Improving the youth-friendliness of facilities
School:




Training teachers, supervisors/inspectors, parents and students
Forming reproductive health clubs/ integrating reproductive health into existing clubs
Sensitizing parents and teachers
Establishing links between schools and health facilities
3. Solution Testing and Evaluation
The field study was conducted in two districts in Kenya. The researchers used a quasiexperimental design with two experimental strategies, pre-intervention and post-intervention
measurements to test the effectiveness of the interventions. To control for cultural and other sociodemographic diversity, the experimental and control sites would be expected to be similar with
respect to:




Socio-cultural characteristics
Rural/urban characteristics
Accessibility to health care services
Population density
In each district there were three study sites (A, B, C) to which the community was randomly
assigned. Each of the three sites got different interventions to determine the effectiveness of each
intervention.
Design
Site A
Site B
Site C
(Intervention site)
(Intervention site)
(Control site)
Environment
Intervention
Yes
Yes
No
Youth Friendly Health
Services based
Intervention
Yes
Yes
No
School-based
Intervention
No
Yes
No
To measure the changes in the key variables and thus the effectiveness of each intervention, a baseline and end-line survey was conducted in all three sites of each district. To measure the key
variables indicators were determined:
26
Variables
Measurement Indicators
Feasibility: The ability of community opinion leaders,
parents, school and health care officials to provide
support to the project and participate in its
implementation.
Focus group discussions (FGDs), key informant
interviews and informal discussions/meetings with
community leaders before and after the onset of
the interventions. Reports of outcome of
scheduled sensitization and advocacy workshops/
meetings and periodic monitoring reports.
The cost analysis will determine the incremental
cost of the components of the interventions to the
MOH and MOE and will include startup costs for
training and materials development and recurrent
costs to carry out day-to-day operations.
This will be measured through a comparison of a
pre-intervention and post-intervention survey of
married and unmarried adolescents (age 10-19
years), assessing RH knowledge, attitudes,
behavior and utilization of services in the two
intervention sites and one control site.
This will be measured through in-depth interviews
and FGDs with community leaders, parents, health
providers and youth.
Cost: What it costs to build the capacity and run the
new RH programs in schools, with health care
providers and with community-networks to provide
adolescents RH information and services.
Effectiveness: The effect of the interventions on RH
of adolescents.
Environment: Refers to attitudinal aspects of the
community that either facilitate or hinder the
successful provision of RH services to young people.
This includes attitudes of community leaders, parents,
health providers and youth towards providing RH
information and making RH services to married and
unmarried adolescents aged 10-19 years more
accessible.
Service Utilization: The proportion of sexually
active adolescents who report use of RH services.
Youth-friendliness: Youth friendliness will be defined
as the ability of service providers to treat youth
seeking RH information and services with respect,
ensure privacy and confidentiality of the consultation
and provide the required service in a competent, nonthreatening and non-judgmental manner.
RH knowledge: This includes being able to correctly
describe the fertile period, body changes at puberty,
various STIs and symptoms and signs of STDs and
HIV/AIDS, ways of pregnancy and STD/HIV infection
prevention, correct condom use, sources of various
RH services in the community, and consequences of
early (adolescent) pregnancy and child birth as well as
unsafe abortion. .
RH attitudes: These include opinions about ideal
family size, use of modern contraceptive methods and
condoms among youth and adult population, sex
education for youth, youth engagement in sexual
intercourse, and provision of RH services (maternal
and child health, family planning, post-abortion care,
STD diagnosis and treatment and emergency
contraception) to married and unmarried adolescents.
Service statistics on number of married and
unmarried adolescents provided with RH services
will be used to measure utilization of services.
This information will be gathered before and at the
end of the intervention through a review of
service statistics and interviews with adolescents.
Youth friendliness will be assessed through indepth interviews and FGDs with adolescents, use
of mystery clients to collect data from providers,
and pre-intervention and post-intervention surveys
with unmarried and married adolescents 10-19
years.
Knowledge will be measured through preintervention and post-intervention surveys
focusing on unmarried and married adolescents
aged 10-19 years
These attitudes will be assessed through FGDs and
in-depth interviews with parents, community
leaders, adolescents and health care providers. In
addition, parents and youth’s attitudes will be
assessed through the pre-intervention and postintervention survey of married and unmarried
adolescents aged 10-19 years.
27
RH behavior: The kinds of behavior we will be
focusing on include reported age at initiation of sexual
activity; sexual activity within the past six months;
number of sexual partners; consistent use of condoms
and use of modern contraceptive methods for the
sexually active; obtaining diagnosis and treatment from
recognized health practitioners for those who report
having contacted STDs; accessing services for prenatal
and postnatal care for those who have been/are
pregnant and/or with child; and accessing emergency
contraception and post-abortion care for the girls
requiring such services.
This information will be assessed through the preintervention and post-intervention survey of
married and unmarried adolescents aged 10-19
years.
Gender attitudes: These include opinions about
equal education and work opportunities, family size and
family planning decision-making, physical and sexual
abuse and performance of household tasks, regarding
males and females.
These attitudes will be assessed through FGDs and
interviews with youth as well as through the preintervention and post-intervention survey.
4. Information Dissemination/Stakeholder Involvement
Research findings and lessons learnt were disseminated at various levels, nationally and locally
and at different times during project implementation. Key audiences for district and provincial level
dissemination were identified during the diagnostic phase.
At the national and international level, an update of the project’s achievements, constraints,
lessons learned and research findings were disseminated in project update briefs. These were mailed
periodically to policy-makers in the involved Ministries and to program managers in other
organizations that are implementing youth programs, donor agencies and other interest groups.
Midway into the project, stakeholders and other key audiences were contacted to provide a
progress report and information on any preliminary findings and to get feedback on the research
interventions.
On completion of the research, the final report was published and distributed to both
technical and non-technical audiences. Also a summary was prepared for policy makers in the
Ministries and donor agencies. National dissemination activities included a national seminar, use of
the media, publications in professional journals and presentations to professional and interest groups
such as educators, health providers and youth advocates.
Planning for sustainability and scale up had already begun while the country project was being
designed. The Population Council had begun seeking partners who were able to support scale up
efforts. Scale up was also enhanced by having an advisory committee of influential people and
involved throughout the project. The members of the advisory committee played an advocacy role to
influence government and other youth serving organizations to scale up or replicate the project if it
is successful.
28
5. Final Remarks
OR is the process of identifying and analyzing a problem within the operations
(activities) of a program or an organization, and of finding an optimal solution for
this problem by testing alternative ways of action (models) which will serve as a
basis for decision making regarding successive modifications of key variables to
achieve the overall goal of enhancing or even optimizing the effectiveness and
efficiency of the program or organization.
Starting from this definition, a research study that is classified as OR needs to meet all
conditions that are mentioned in this definition. So first a problem within the operations of a
program or of the whole organization must be identified. Then, the problem needs to be analyzed.
For this purpose, all data relevant to this problem need to be gathered and analyzed to identify the
key parameters and variables influencing the problem. This is necessary to gain a deeper
understanding of the whole problem and to develop solutions for overcoming it. Next, different
alternative solutions are tested in terms of the desired outcome. The results need to be presented to
decision makers who will then decide which solution should be implemented or which modifications
to the program should be made to overcome the problem and to enhance the effectiveness and
efficiency of the program as the overall goal.
Finally, I will mention the main characteristics that distinguish OR from other forms of
research according to the definition of OR from the WHO that is included in the Framework for
Operations and Implementation Research in Health and Disease Control Programs (Global Fund 2008,
p.49):

It addresses specific problems within specific programs, not general health issues.

It addresses those problems that are under control of managers, such as program
systems, training, pricing and provision of information.

It utilizes systematic data collection procedures, both qualitative and quantitative to
accumulate evidence supporting decision-making.

It requires collaboration between managers and researchers in identification of the
research problem, development of the study design, implementation of the study and
analysis and interpretation of results.

It succeeds only if the study results are used to make program decisions; publication
alone is not a valid indicator of successful OR.
29
By restating these criteria at the end I want to highlight the main characteristics of OR again
to make the distinction to other forms of research clear.
If a study does not consist of all the components that are necessary for conducting an OR, it
has to be clearly stated that the study only contains some aspects of an OR study. It should be the
goal within World Vision’s research to conduct OR to identify program problems and to find a
solution to overcome them because that will enhance the effectiveness and efficiency of the
programs.
For the funding component of such OR within World Vision, this means that the OR project
should set aside funds to
(i)
conduct a problem analysis and alternative solutions/develop a model
(ii)
involve researchers, program managers and policy makers in the design of the OR
project
(iii)
support the implementation of the identified solution(s) or model(s)
(iv)
support the implementation of the research component
(v)
support participation of researchers, program managers and policy makers in
interpreting and disseminating the results and planning for utilization of the results.
In the context of Child Well-Being Outcome (CWBO) measurements, Area Development
Programme (ADP) and non-ADP baseline and follow-up surveys, categorized as program evaluation
research, will present evidence on contribution that WV is making in improving CWBOs. The gap is
in identifying which of the varying strategies/ models that can be applied to meet specified CWBOs
are more effective/ cost-effective. OR projects relying on prospective control or comparison group
design offer the best evidence to attribute change to particular intervention models.
Research that does not contribute to improving the impact of World Vision’s programs may
be fine for the sake of research; but the goal of research within World Vision’s programs should be
to provide information for enhancing the effectiveness and efficiency. So ideally OR should be the
method of choice.
30
Appendix 1
Operations Research Checklist
1. Problem Identification and Diagnosis
Yes
No
Is the problem identified, analyzed and clearly
formulated?
Is all relevant information and data gathered?
Are the program managers involved?
Are the objectives determined that should be
achieved through implementing a solution?
2. Solution/Strategy Selection
Has a model that represents the problem been
developed?
Are the key variables of the model defined?
Are multiple solutions developed to overcome
the problem?
Are the solutions manageable for the
organization to implement?
Has a Design of Experiment been drawn up?
31
3. Solution/Strategy Testing and Evaluation
Are criteria defined to measure the outcome
of each alternative?
Are ethical issues regarding the testing
considered?
Are the alternative solutions tested?
Are the alternative solutions compared
regarding the desired outcome(s)?
Is further analysis conducted to assess the
reliability of the findings?
Is the optimal solution identified?
4. Information Dissemination and Program Adaptation
Are the results of the study presented to the
decision makers?
Have the decision makers decided on a
solution/strategy to implement?
Is the solution/strategy implemented or are
adaptations to the program made?
32
Appendix II
Note: The following example was given at the end of the text of an earlier version of this paper; it has been
criticized for not truly representing OR. Hence, the example was put into an appendix where it is followed by
Discussion/Critique at the end.
The Ministry of Health has implemented a primary health care program. The overall goal of
this program is to reduce child mortality and morbidity. The program has stalled because a large
number of the community health workers that are appointed to visit the population have not been
trained even though the Ministry of Health developed a systematic plan of training. Through OR the
most cost-effective way of training should be found out.
First, the researchers analyze the problem in collaboration with the Ministry of Health. They
assess that the system does not work in terms of quality of the training and the number of trained
Community Health Workers. From other experience the Ministry of Health and the researchers find
out that an approach that uses a core staff of trainers and audiovisual materials has proved to be very
successful. Now they want to find the most cost-effective way to adapt this method.
Although they have decided on a general training approach, the actual structure of that
approach has not been determined. Therefore they have to identify the variables of the strategy that
include:
1. Training content
2.
3.
4.
5.
Training methods
Training materials
Trainers
Duration of Training
6. Class size
7. Training location
8. Level of supervision following training
The researchers decide to standardize the variables 1, 2 and 3 and they determine that the
training can be completed in one day (variable 5) based on past experience. For budgetary reasons
they decide that the training will be held in district offices. So three variables are left. Next, the
researchers determine the options for the remaining variables.

Trainers: regional staff or district staff (variable 4)

Class size: 20 or 40 promoters per class (variable 6)

Supervision: monthly or semi-annual supervisory visits (variable 8)
When combining the different variations of these variables there are 16 alternatives. These
alternatives are tested in terms of the defined outcomes namely cost, effectiveness and cost33
effectiveness ratio. The economist on the research team identifies the relevant costs and collects the
data needed to test all alternatives. This can be done through mathematical methods.
But also the effectiveness of the alternatives needs to be measured. Although the ultimate
goal of the program is to reduce child mortality and morbidity the research team determines that it
is impossible to measure any significant change in the short term of the research. Therefore, the
training of the Community Health Workers who provide the needed service to reduce child
mortality and morbidity is selected as effectiveness measure.
The team selects knowledge, skills and behavior of the promoters and the knowledge, skills
and behavior of parents as the components for measuring the effectiveness. They also define these
terms, so for example the knowledge of promoters is determined by their understanding of
symptoms of malaria and their ability to administer tablets and so on. Most of these data will be
collected and analyzed during the field test of the alternative. Before that field test, the research team
will have to rely on two measures to determine the most-effective alternative, which will then be
tested: the estimated percentages of all promoters that will have to be trained and the estimated
percentage of the trainees that will pass a knowledge and skills test at the end of training. Estimates
for these two measures can be obtained from experience with current training programs and from
the recent experience with the new training method among census workers. Thus, the team makes
estimates on the percent of all promoters that will be trained and the estimated percentage of the
trainees that will pass a knowledge and skills test at the end of training. These estimates can be
obtained from experience. No estimates can be made on the effects training will have on promoters’
behavior or on parents.
Immediate output :
Number of promoters expected to complete training
Total number of promoters
Immediate effects:
Number of promoters expected to pass a knowledge skill test
Total numbers of promoters
The results will lead to a ranking of the 16 alternatives. In the next step the researchers will
identify the least costly alternative and the most effective alternative. If this examination would not
lead to the same alternative, the research team will calculate the cost-effectiveness ratios for each
alternative in terms of promoters expected to be trained and promoters expected to pass the test.
So for each alternative there are two calculations. Based on these calculations the research team
recommends the most cost-effective alternative. This alternative may not be the most effective but
including the costs it is the best alternative that considers both costs and effectiveness.
Because the research team made assumptions about costs and effectiveness they will conduct
a sensitivity analysis by varying their assumptions. They are confident about the estimated costs and
do not think that changes in their assumptions would influence the ranking of the alternatives. But
34
they are not sure about their assumptions related to the effectiveness of the alternatives. They
discuss this topic with the minister and decide that a new alternative should be examined. Also for
this alternative they calculate the costs, expected effectiveness and the cost-effectiveness ratio and
present the data to the Minister.
Finally, they decide to test the new alternative and the alternative, which was previously
determined as the most cost-effective alternative. After the field test the research team will calculate
the actual cost, effectiveness and cost-effectiveness and the most cost-effective alternative will be
implemented.
Discussion/Critique:
1. I am not sure where the number 16 came from, as there should only be 8 experiments
from the presentation.
2. The authors select several variables in the beginning. As presented in the example
section, the variables are trainers (x), class size (y) and supervision (z). We have two
measures of result, namely cost (c) and effectiveness (e). We thus desire a mathematical
model (c,e) = f(x,y,z). For experimentation, each variable takes on two different values.
This would result in 8 different experiments as each combination must be tested.
3. However, we may assume that the three variables are independent of each other. Thus
we do not need to engage in all 8 experiments. A design of experiment would have
produced the following experiments:
X = regional, y = 20, z = semi-annual.
X = district, y = 20, z = semi-annual.
X = regional, y = 40, z = semi-annual.
X = regional, y = 20, z = monthly.
4. These 4 experiments are sufficient in the presence of independence. As each dimension
has only two values (both in this design of experiment as well as in the original design),
we can only model the system linearly. Thus the function takes the form (with the a_i as
parameters):
C = a_1 x + a_2 y + a_3 z + a_4
E = a_5 x + a_6 y + a_7 z + a_8
35
5. As we have 4 experiments and two outcome measurements per experiment, we have 8
experimental facts. We also have 8 unknowns in our model and so everything is welldetermined.
6. The example now claims that the experimenters chose the best of their 16 experiments.
This is not generally correct. We have used the experiments to get the parameters of
the model above. Now we wish to compute the optimal point. As the outcome measures
C and E interact and are mostly probably constrained to each other (effectiveness and
cost are related to each other), the optimal solution is a negotiation between these two
dimensions. The constraints must be formulated.
7. One way out of the conflict between two goals is to determine which is more important
and to introduce weights for each. This is a management decision and should have been
made in the beginning of the project. First, we must measure both C and E in comparable
units and then introduce a relative weight. Let us assume that we have common units
already. Second, we then combine them into a single goal function
Goal = w_1 c + w_2 e
However, we have to consider the constraint that C and E are not independent but
rather constrained to each other and perhaps each constrained in their own right. There
may, for instance, be a maximal budget. The goal must now be maximized by finding the
right values of x, y and z such that goal is maximal subject to the constraints. This is a
problem for optimization theory.
8. It is very likely indeed that the final position of the optimum is not one of the
experimental support points tested in the experiment. For instance it may be that the
optimal answer has
X = regional, y = 30, z = quarterly.
This depends upon the outcome of the experiments.
9. Thus, we must consider the elements of design-of-experiment, constraints and multiobjective optimization theory in our efforts to locate the true optimal point. Before
engaging in an experiment, we should formulate the constraints as part of the problem
definition and then perform a design-of-experiment. Only then do we have the tools to
perform the (and only the) experiments needed and to be able to do the computations
leading to the optimum.
(Author of Discussion/Critique: Patrick D. Bangert, PhD)
36
Annotated Bibliography
Non World Vision publications
Hillier, F. S. & Lieberman, G. J. (2010). Introduction to OR. New York: MC Graw Hill Education.
For over four decades, this is the standard work on Operations Research. It includes a detailed
description of the fundamentals and the process of Operations Research and of the
mathematical models that are used in Operations Research.
Population Councils Frontiers in Reproductive Health Program, PATH & Government of Kenya
(2004). A Multi-Sectoral Approach to Providing Reproductive Health Information and Services to Young
People in Western Kenya: Kenya Adolescent Reproductive Health Project.
This is a project proposal and summary of an example for social OR that was conducted by the
Population Councils’ Frontiers in Health Program in collaboration with PATH and the
Government of Kenya.
Robson, Colin (2002). Real World Research.2nd Edition. United Kingdom: Blackwell Publishing.
A book on research approaches outside the laboratory in “real world” situations relevant for
behavioral and social science disciplines.
Stier Adler, E. & Clark, R. (2011). An Invitation to Social Research. Belmont, CA: Wadsworth, Cengage
Learning.
This is a book about the methods of social research. Describing the research process in all its
details this book provides guidance through the whole process of conducting social research.
USAID, Bureau for Global Health Office of Health, Infectious Disease, and Nutrition (2010) Guidance
for Detailed Implementation Plans (DIPs) Innovation & Tuberculosis Category Child Survival and Health
Grants
These are guidelines for project implementation and management with a detailed guidance for
a OR concept paper.
37
Online Sources
Fisher, A. A. & Foreit, J. R. (2002). Designing HIV/AIDS Intervention Studies: An OR Handbook. New
York: The Population Council. Retrieved April 20, 2011 from
http://www.popcouncil.org/pdfs/horizons/orhivaidshndbk.pdf
This is a Handbook on Operations Research especially for conducting it in health programs
related to HIV/AIDS. It provides guidance through the whole research process through a
detailed description of each step.
Gesellschaft für OR (GOR) e.V. GOR: OR. Retrieved April 20, 2011 from https://gor.unipaderborn.de/
This is the website of the German Society for Operations Research. It contains some basic
information on Operations Research.
International
Healthcare
Consultants.
inhcc.
Retrieved
May
10,
2011
from
http://www.inhcc.com/Operations_Research/operations_research.htm
International Healthcare Consultants is a Healthcare Information System Software Developer
and International Healthcare Management Consulting Company that offers cost-effective and
practical integrated solutions to a wide range of healthcare professionals and organizations.
Especially the information on a Primary Health Care Operations Research project is useful.
Reynolds, J. & Gaspari, K.C. (1985). Operations Research Methods: Cost-Effectiveness Analysis. PRICOR –
Primary Health Care Operations Research. Chevy Chase, Maryland: Center for Human Services.
Retrieved May 23, 2011 from http://pdf.usaid.gov/pdf_docs/PNAAX968.pdf
This document contains a description of the process of conducting a cost-effectiveness
Operations Research that is supplemented by an example of a cost-effectiveness Operations
Research in the field of primary health care.
Stat-Ease – statistics made easy. DOE for Six Sigma/Lean. Retrieved June 20, 2011 from
http://www.statease.com/sixsigma.html (Figure 1)
38
The Association of European Operational Research Societies. (2011). Operational Research: What is
Operational Research? Retrieved April 20, 2011 from http://www.euro-online.org/web/pages/197/whatis-or
This website of the Association of European Operational Research Societies provides general
information on Operations Research and the European Societies that are engaged in Operations
Research.
The European Working group on Operational Research Applied to Health Services. ORAHS: What is
Operational Research. Retrieved April 20, 2011 from http://orahs.di.unito.it/
This working group is specialized on Operations Research applied to health services and is
intended for practitioners in this field to share their ideas and experiences. The website contains
basis information on Operations Research conducted to problems in this field.
The Global Fund. Framework for Operations and Implementation Research in Health and Disease Control
Programs. Retrieved May 10, 2011 from
http://www.theglobalfund.org/documents/me/FrameworkForOperationsResearch.pdf
This is a framework for Operations Research in Health and Disease Control Programs and was
developed by the Global Fund to Fight AIDS, Tuberculosis and Malaria, the Special Program for
Research and Training in Tropical Diseases (TDR) and an inter-agency technical working group.
World Vision Sources
Chege, J. & Ghee, A. (2010). Draft Outline. A spectrum of inquiry: The role of operations research in World
Vision’s work. Retrieved April 15, 2011 from
https://www.wvcentral.org/cop/hnh/researchdme/Documents/Forms/AllItems.aspx?RootFolder=%2fcop%2fhnh
%2fresearchdme%2fDocuments%2fOperations%20Research%2fPreparation%20for%20July%204%2d6%20e
vent&FolderCTID=0x012000D8B5EEF256DDB84B9B3CF120A0B1A12D00B2F174B14066DD47B1E63A
961E004E39&View=%7b56EEBC5B%2dD5E0%2d4356%2dA79C%2dC500FE3E08C7%7d
This is a first draft on Operations Research that describes the characteristics of Operations
Research and the steps that need to be taken while conducting it.
39
Initiatives Inc. for The Healthcare Improvement Project (2010). Operations Research: Influence of CHW
Program Functionality Strengthening on CHW Engagement and Performance – Research Protocol. Retrieved
May 12, 2011from
https://www.wvcentral.org/cop/hnh/researchdme/Documents/Forms/AllItems.aspx?RootFolder=%2fc
op%2fhnh%2fresearchdme%2fDocuments%2fOperations%20Research%2fPreparation%20for%20July%
204%2d6%20event&FolderCTID=0x012000D8B5EEF256DDB84B9B3CF120A0B1A12D00B2F174B14
066DD47B1E63A961E004E39&View=%7b56EEBC5B%2dD5E0%2d4356%2dA79C%2dC500FE3E08C7
%7d
This is a document from Initiatives Inc. for the Healthcare Improvement Project that offers a
framework for the assessment of Community Health Workers (CHW) programs and for the
development of appropriate interventions to program problems.
Carboni, I. (2011). Guiding Principles for National Offices – Measuring and Reporting the Child Well-being
Targets. Retrieved May 12, 2011 from https://www.wvcentral.org/cwb/Documents/Child%20Wellbeing%20Partnership%20Targets/Principles%20for%20Measuring%20and%20Reporting%20CWB%20T
argets%20FINAL%20May%202011.pdf
This document outlines guiding principles for national offices to consider before measuring and
reporting on the Child Well-being targets and includes information about other forms of
research such as impact evaluations.
Monbiot, E. (2006). Agreed DRAFT standards for research.
This is document arose from a meeting held in 2006 and defines the purposes of research
within World Vision and describing the steps that need to be taken while conducting research.
40