Step 3: Establish Strategic Management
Partner with
government
Monitor
and evaluate
Develop the
program
Implement
special
services
Establish
strategic
management
Implement
service
delivery
Build site
capacity
will help you to elaborate and finalize the basic program design you have
established and place it within a comprehensive monitoring and evaluation
(M&E) framework. Having done that, you will be able to define a management
structure aligned with the program strategy. The resulting management structure should be able to
help you pursue the program objectives, ensure quality service delivery and manage the program’s
human, financial and technical resources.
Step 3
Objectives
n
Develop the M&E framework and finalize the program design.
n
Establish a strategic management structure.
22
In Step 2 the basic program design
began to take shape. In order to elaborate
it fully, it is recommended in Step 3 to
develop a comprehensive M&E framework.
In reality, developing an M&E framework is
a very effective and logical tool for designing
the entire program. However, it is useful
to have put in place the basic elements of
program design first as described in Step 2.
The elaboration of the M&E framework will
decisively establish the program objectives.
It will also solidify the inputs and activities
required to reach these objectives, as well as
the means of measuring the outputs, outcomes
and impact of these activities, in order to
determine to what extent the objectives are
ultimately attained.
Dr. Sebastian Wanless, BIPAI V.P. of Research and Program Evaluation lectures
about M&E to Romanian colleagues
Developing a Monitoring and Evaluation
Framework
Setting the Objectives
Objectives are specific, operationalized statements
detailing the desired accomplishment of the program or
project.
For example a new VCT program could have the
following objectives:
M&E is an essential element of a results based
program. It provides organizations with a tool to evaluate
how effectively they promote change. The purposes of
M&E can be summarized as follows:
n
n
n
n
n
n
n
n
n
By the end of the first program year, 98% of clients
tested for HIV will receive test results.
n By the 2nd quarter of the second program year,
65% of clients receiving HIV test results will
have developed and adhered to personalized riskreduction and treatment strategies.
n
Facilitating program design
Improving and supporting management
Determining if planned results are achieved
Generating shared understanding among personnel
and partners
Building the capacity of those involved
Motivating stakeholders
Ensuring accountability
Demonstrating results to funders
Fostering public and political support
These examples illustrate the SMART method for
defining objectives. SMART objectives are:
Specific:
They identify concrete events or actions
that will take place.
Measurable: They quantify the amount of resources,
activities or change.
Appropriate: They logically relate to the overall
problem statement and desired effects of
the program.
Realistic:
They provide realistic dimensions that
can be achieved with available resources.
Time-based: They specify the time within which the
objectives will be achieved.
The following guide will assist you in designing an
M&E framework for your program.
Finalize the Goals and Objectives of Your Program
In Step 1 you created overarching goals for your
program. At this point it is worth revisiting them again
and now clarifying the specific underlying objectives.
23
Use the simple table below to state your goal and
objectives.
n
Activities: the set of activities in which program
resources (human and financial) are used to achieve
the results expected from the program e.g., training
courses conducted, materials developed and dis­
seminated, workshops and conferences organized,
etc.
n
Outputs: are the immediate results obtained by the
program through the execution of activities, e.g.,
people are trained, guidelines on VCT have been
developed, and the target population’s knowledge
about HIV/AIDS prevention has increased.
n
Outcomes: are the intermediate results obtained by
the program through the execution of activities. They
typically incorporate a quality element: e.g., as a result
of the training and development of VCT guidelines,
the quality of VCT provided has improved.
n
Impact: is the long-term result obtained by the
program through the execution of activities, e.g.,
changes in disease incidence and prevalence and
changes in disease morbidity and mortality.
Table 3.1 Defining Program Goals and Objectives
The NAME project at the
NAME site aims to ______________________________
Its overall goal is to ______________________________
The specific underlying objectives are:
1. _____________________________________________
2. _____________________________________________
3. _____________________________________________
4. _____________________________________________
5. _____________________________________________
Develop the M&E Logic Model
Firstly, note the following definitions of key M&E
terms and concepts:
n
n
n
Monitoring:
– Tracks information relevant to program
planning, outputs, outcomes and impacts on an
ongoing and systematic basis.
– Provides the basis for episodic program
evaluation.
– Routinely collects data that allows measurement
of progress towards program objectives.
Impact: is the long-term result obtained by the program through the
execution
of activities,
changes3.2
in disease
incidence
andhelp
prevalence
The
analogies
in e.g.,
Table
below
will
you to
and changes in disease morbidity and mortality.
understand
these concepts better.
The analogies in table 3.2 below will help you to understand these concepts better.
Table 3.2: Example of M&E components, with
analogy
Table 3.2: Example of M&E components, with analogy
M&E components
Evaluation:
– Is an episodic rigorous and scientifically
based collection and analysis of information
about program activities, characteristics, and
outcomes to determine the merit of a specific
program. Evaluation typically occurs at various
predetermined times during a program and at
its completion.
– Is used to improve programs and inform
decisions about future resource allocations.
The Soup Analogy
HIV/AIDS M&E Pipeline
Inputs
Carrots, potatoes,
chicken broth, spices
Condoms, test kits, drugs, staff,
money, facilities, etc
Activities
Act of cooking the soup
Training
Delivery of services
Outputs
The finished soup
People receiving services (VCT,
clinical, care and support)
Condom availability
Trained staff
Knowledge of HIV transmission
Outcomes
Level of satisfaction
obtained from eating the
soup
Attitude changes
Appeasement of hunger
Changes in STI trends
Behavior change (increased condom
use, sexual abstinence, monogamy)
Increase in social support
Inputs: the various resources needed to run the
program, e.g., money, facilities, program staff, sup­
plies and equipment, etc.
Impact
Improved nutritional
status of person eating
the soup
Changes in HIV/AIDS morbidity and
mortality
Changes in social norms
Coping capacity of communities
Economic impact
The inputs, activities, outputs and expected outcomes and impact of your program
can be clarified and displayed in the form of a logic model, an example of which is
shown below. It is not really complicated. Use Tool No. 9 to develop your own
logic model. Start on the left hand sides by listing the inputs you have at your
24
The inputs, activities, outputs and expected out­
comes and impact of your program can be clarified
and displayed in the form of a logic model. It is not
really complicated. Use Tool No. 9 to develop your
own logic model. Start on the left hand sides by listing
the inputs you have at your disposal, and then describe
the activities you will undertake in the program. Under
outputs, outcomes and impacts list the anticipated or
at least hoped for short-term, medium-term and longterm results respectively.
Figure 3.1 is an example of such a logic model,
detailing a program of outreach of HIV/AIDS care and
treatment to children and their families living in rural
areas.
Determine the Relevant Performance Indicators
Figure 3.1 Example of a program logic model
At this point, you should elaborate the
performance indicators for your program.
Refer back to the logic model you have created Table 3.1: Performance indicators appropriate to the logic
inPerformance
Figure 3.1
Table 3.1:
indicators appropriate to the logic model in Figure 3.1
previously and decide which measurements you model
can make to determine how well the expected OUTPUTS
PERFORMANCE INDICATORS
doctors and other health
- No. of trained program doctors in the field
outputs, outcomes and impacts are attained. Program
- No. of other health professionals in the field
profess ionals trained and in the field
Tool No. 9 will also assist you to develop Satellites built and equipped
- No. of satellites built and equipped
- No. of other outreach centers providing
performance indicators for your program.
enhanced service
Also for reference, please find in Table 3.1, Community s ervices provided
- No. of each type of community service
provided
suggested performance indicators matching
- No. of satellite or outreach sites fully
serviced by community support
the logic model illustrated in Figure 3.1.
- No. of children treated at outreach sites
Remember that performance indicators C hildren treated in outreach s ettings
- No. of other family members treated at
outreach sites
represent the basic data you will be collecting
- % of clinical encounters at outreach sites
on an ongoing basis to monitor your program. C hange relative to outreach status
considered of best quality
They are clues, signs, and markers of how much
- % change in children tested at outreach sites
PERFORMANCE INDICATORS
progress a program is achieving and how much OUTCOMES
Increas ed access to high quality
- % of opportunistic infections successfully
deviation there is from the initial program treatment and care in outreach s ettings
treated
- Change in average age of first diagnosis of
design and the expected program results.
children (target < 6 months)
Once the performance indicators have
- % change in hospitalizations
- % change in uptake of VCT at antenatal
been finalized, you should design tools for Improved PMTCT services
clinics
collection of these data. Some of them,
- % change in MTCT rates
especially output indicators (e.g., the no. of
- % change in early infant HIV diagnosis
Communities mobiliz ed
- No. of people reached through community
children treated) can be simply tabulated.
outreach and mobilization techniques
- No. of children referred by community
Others may require calculation (e.g., after
organizations
implementing the outreach strategy, by what Local physicians and other health
- Evaluated by examination before program
percentage has school attendance improved?). profess ionals well trained in pediatric
presence and 6 months later
These calculations are probably best done by HIV care and treatment
IMPACTS
PERFORMANCE INDICATORS
an M&E officer.
Define M&E Roles and Responsibilities
The data collection associated with a
project usually necessitates the availability of
adequately trained staff at all levels of data
HIV morbidity and mortality decreas ed
-
Hospitalization rates
Efficacy (undetectable VL or composite
index based on CD4 and WHO stage
Nutritional status
School attendance by school age children
Mortality
MTCT rate
Remember that performance indicators represent the basic data you will be
collecting on an ongoing basis to monitor your program. They are clues, signs, and
markers of how much progress a program is achieving and how much deviation
25
gathering. It is ideal if the program is able to appoint
an M&E officer, who can be responsible for establishing
and maintaining M&E systems and mechanisms for all
project activities. The M&E officer ensures that problems
with data collection are dealt with in a timely fashion,
data are fed into National Health Information Systems
as required, that communication and coordination
about data issues, between all project stakeholders is
regular, that data analysis is done on a regular basis and
feedback is provided to all stakeholders and that data for
overall project indicators are reported routinely and are
of adequate quality.
Other than appointing an M&E officer, an important
task is to ensure that part-time or full-time staff have
been designated and assigned M&E responsibilities at
these various levels. The responsibilities should include
the data for which collection they are responsible and
the way in which they must report those data and to
whom.
In Step 7 of this toolkit, we shall return to M&E
in order to explain how to use the collected data.
Establish a Strategic Management Structure
Having fully enunciated the program design, it is
now time to develop a management structure which
is aligned with the design and which will give the best
chance of attaining the program goals and objectives.
To this end, you may wish to appoint specific individuals
tasked to carry out the activities associated with
individual objectives. Although the overall responsibility
for the program may well reside with a program
director, who might for example be a physician, it is
highly recommended that a trained and experienced
project manager be appointed. This means a person who
has specific training in project management, which is a
critical skill in itself. It does not matter if the project
manager comes from government, the community or
from an outside consulting firm—it is just essential that
the successful candidate treats all partners equally and is
qualified and experienced in project management.
The strategic management structure under the
program director and project manager is best determined
in a participatory fashion. The project manager could
introduce a proposal of the structure at a workshop of the
type described in Step 2. Broad discussion of the plan
by the stakeholders helps to ensure common ownership
and compliance. The resulting management structure
should specify the roles and reporting relationships
between partners. It should also specify the partners’
reporting responsibilities to stakeholders
The project manager should have sufficient authority
to carry out the job effectively. Project managers can also
rely on governance boards and management committees
composed of local leaders and government representatives
for strategic advice and support. These bodies can play an
important role in building government and community
support for the program, but they should not be involved
in daily management of the project. The project manager
might consider developing small technical groups for
each major function (e.g., clinical, laboratory, pharmacy,
community services, administration, finance etc.),
which in turn form a larger technical working group
for decisions on cross-functional activities. Technical
working groups can effectively supervise daily activities.
An example of a project management structure (that
of the Pilot Operational Research and Community-Based
Data Management, Analysis and Reporting
Data management, analysis and reporting are integral
parts of the monitoring process. The information
provided by the M&E staff at all levels should be input
in a database or entered in a filing system. Designated
staff should have the responsibility to ensure that the
information provided is accurate and reliable. This
process is referred to as quality assurance and may
include site visits and records reviews, interviews with
all staff involved in data collection, observation of their
work, auditing of the accounting and procurement
documentation etc. Quality assurance is an important
part of monitoring because it provides a mechanism
to verify and validate the information reported by the
coordinators and data clerks. Tool No. 10 will assist
you in establishing quality assurance systems.
Based on the collected data, designated staff at each
level should be asked to submit monthly, quarterly or
biannual reports to the project management or directly
to the M&E officer. Either upper management or the
M&E officer will do more intensive report writing that
will allow submission of monthly, quarterly or biannual
reports to other stakeholders such as donors and the
Ministry of Health.
The collected data should be examined on an ongoing
basis to determine if the project is on course. Significant
deviations from the expected results should prompt
investigation of potential issues and corrective and pre­
ventative action should be taken when appropriate.
26
F I G U RE 3.0:
MANA GEME NT ST RU CT U RE OF PORE CO PROJE CT AT THE B IP AI COE IN
SW AZIL AND
Programme – PMTCT Plus project in
Swaziland) is shown in Figure 3.2.
Note that both direct authority and
administrative reporting relationships
are detailed.
Lessons Learned
Develop an M&E framework
which collects data that will attest
to the quality of service provided
and which will assist in project
management as well as providing
evaluative date for outside parties.
n Using the M&E logic model can
effectively finalize the design of
your program.
n Identify a strong project manager
trained in project management
early in the planning stage and
assign that person adequate
authority.
n Establish a management structure
with clear roles, responsibilities,
and reporting relationships, which
is directly aligned with the goals
and objectives of the program.
n
Figure 3.2: Sample project management structure
27