Guide to Core Output and Outcome Indicators for
IFC Technical Assistance Programs
Revised January 26, 2006
Table of contents
I.
Introduction ...........................................................................................................................3
Purpose.............................................................................................................................3
Basic concepts ..................................................................................................................3
Outline...............................................................................................................................5
II. Core Indicators......................................................................................................................6
Output Indicators for IFC technical assistance projects ....................................................7
Outcome and Impact Indicators for Access to Finance.....................................................9
Outcome and Impact Indicators for Business Linkages ..................................................13
Outcome Indicators for Business Enabling Environment ................................................17
III.
Data collection.................................................................................................................21
Overview .........................................................................................................................21
Program records .............................................................................................................21
Surveys ...........................................................................................................................21
Secondary sources .........................................................................................................22
Budget guidelines............................................................................................................23
Appendix A. Program Logic Models...........................................................................................24
Appendix B. Definitions of Key Terms .......................................................................................28
Appendix C. A Note on Impact Assessment ..............................................................................31
2
I.
Introduction
Purpose
1. This guide presents indicators that can be used to assess technical assistance programs
that are geared to promoting private sector development, including but not limited to
programs directed toward small and medium enterprises.1 Specifically, the guide provides a
core set of output, outcome and impact indicators for three types of technical assistance
programs – access to finance, business linkages, and business enabling environment –
along with explanatory notes and guidance on data collection procedures.
2. The purpose of standard core indicators is three-fold:
•
Provide an objective basis for assessing performance of IFC projects;
•
Provide a solid foundation for management decisions ; and
•
Facilitate learning within IFC and external stakeholders.
3. The core indicators are based on discussions with program managers and M&E staff, a
review of program documents, and a review of literature on similar programs undertaken by
other organizations. In defining the indicators, emphasis has been placed on trying to
ensure that indicators are relevant to the types of technical assistance projects most
frequently undertaken by IFC in developing countries, are valid with respect to reflecting the
underlying concept that is meant to be measured, are reliable in terms of minimizing
potential measurement error, and are practical for facilities and regional offices to use given
the effort required to collect necessary data.
Basic concepts
•
Indicators are derived from program logic models. The indicators are based on
program logic models developed for each of the three major programs.2 These models
describe the sequences of cause and effect relationships that link IFC program activities
to intended impacts. Each model has five basic components as illustrated in Figure 1.
Inputs refer to the resources used in program activities. Activities are the actions taken
or work performed in particular projects using specified inputs. IFC technical assistance
projects include activities such as assessments, advisory services, training, and public
awareness campaigns. These activities are intended to result in outputs such as
reports, advice, training events, and media coverage. In turn, these outputs are
expected to yield certain outcomes in terms of changes in knowledge, behavior and
performance among beneficiaries in the target population. Finally, it is anticipated that
programs will generate development impacts including higher productivity, greater
income, and economic growth. The indicators presented in the guide focus on outputs
of IFC program activities as well as associated outcomes and impacts.
1
Initially the effort to define core indicators focused on programs directed to SMEs, but was subsequently
broadened to include programs to promote the development of private enterprises regardless of size.
2
Appendix A contains detailed program logic models for each of the three program areas.
3
Figure 1. Basic Program Logic Model
Target Population
If
Inputs
If
Then
Activities
If
If
Then
Outputs
Then
Outcomes
Then
Impacts
ASSUMPTIONS
•
The indicators account for the fact that IFC may provide services directly to a
specific set of beneficiaries or indirectly through intermediaries. In many cases,
IFC staff or third parties under contract to IFC provide technical assistance directly to
private enterprises. However, it is important to note that IFC also frequently works with
intermediaries such as financial institutions, government agencies, large multinational
corporations, and providers of business development services. IFC staff and/or third
parties under contract to IFC undertake program activities that are directed at these
intermediaries with the expectation that these intermediaries will in turn carry out
activities that will benefit private enterprises, including SMEs. As such, changes in the
magnitude and nature of activities undertaken by intermediaries are considered as
outcomes of IFC technical assistance projects. Whether a particular indicator is defined
as an output or outcome depends on how the project is carried out.
•
Some indicators may not be relevant to a specific facility given the facility’s
particular project mix. Program managers are meant to select indicators included in
this guide that are relevant to their programs and for which data are available at a
reasonable cost. Not all indictors need to be adopted by every facility. For example, if a
facility does not carry out projects that aim to reduce the number of days required to
register a business it would not need to track this indicator. However, facilities that do
carry out such projects should use the indicators included in the guide that deal with
improvements in the business registration process to ensure that measures are
consistent across IFC.
•
In decided who to measure, careful consideration needs to be given to the
definition of the target population and program participants. Outcome and impact
indicators are used to measure different characteristics of entities within specified
populations. The target population represents the intended beneficiaries of the program.
These could be individuals, financial institutions, BDS providers, business organizations
and/or private enterprises (including SMEs). As noted above, these entities can be
either direct or indirect beneficiaries of the intervention. Programs may be directed
toward one or more target populations. Program participants are members of the target
population that receive services or are otherwise affected by the program. Some
interventions are full-coverage programs in that every member of the target population is
involved. This is most likely in programs geared to reforming laws and regulations. In
full coverage program, it is important to assess the affects of the intervention on all
4
entities within the relevant population. However, most technical assistance projects
involve a sub-set of the target population that voluntarily agrees to participate in the
program. In partial coverage programs, it is important to assess the affect of the
intervention on the sub-set of the target population that actually participated in the
program.3
•
Indicators by themselves do not address the issue of attribution. For the most part,
the outcome and impact indicators presented in the guide represent gross program
effects, i.e., they do not provide an estimate of the net result of IFC interventions. As
such they are only intended for monitoring purposes to provide a sense of whether
members of the target population have been reached and have improved performance
on various dimensions. Attributing observed changes to the IFC intervention would
require more rigorous studies that involve comparison with a counterfactual – the
hypothetical situation that would have occurred in the absence of the program. Good
impact assessments use random assignment, constructed controls, and/or other
approaches to isolate the impact of the program from other extraneous factors affecting
outcomes.4 It is envisioned that IFC will undertake more comprehensive evaluations of
programs, including more rigorous impact assessment.
•
Multiple measures are needed to calculate changes over time. The outcome and
impact indicators included in the guide are expressed in terms of a level as of a specific
date. The intention is for programs to establish a baseline and assess changes in the
level of particular indicators over time. Changes can be calculated on an absolute or
percentage basis.
Outline
4. The guide is divided into three sections. Following this introduction, Section II presents core
indicators for each of the three programs. Section III provides guidance on data collection,
outlining requirements for program records, surveys, and secondary data.
5. Three appendices are attached. Appendix A contains the detailed program logic models.
Appendix B presents definition of key terms. Finally, a note on impact assessments is
presented in Appendix C.
3
Partial coverage programs provide an opportunity to compare participants to similar, non-participants as a means
of assessing net impacts.
4
A brief note on methods for assessing impacts is presented in Appendix C.
5
II.
Core Indicators
6. This section contains core output, outcome and impact indicators for IFC Technical
Assistance Projects. The section is divided into four parts:
•
Output indicators for technical assistance projects
•
Outcome and impact indicators for access to finance
•
Outcome and impact indicators for business linkages
•
Outcome and impact indicators for business enabling environment
6
Output Indicators for IFC technical assistance projects
7. Output indicators aim to measure the magnitude of the outputs produced directly by IFC
and/or third-parties under contract to IFC in technical assistance projects. These indicators
reflect outputs of activities carried out during a given quarter. Definitions of key terms are
included in Appendix B.
Activity
Assessments
Advisory
services
Training
Public
awareness
campaign
Indicator
Number of reports produced by IFC and third-parties under contract to IFC in
the quarter
Number of advisory service hours provided by IFC and third-parties under
contract to IFC in the quarter (by type of beneficiary)
Number of unique organizations to which advisory services were provided by
IFC and third-parties under contract to IFC in the quarter (by type of
beneficiary)
Number of unique organizations to which advisory services were provided by
IFC and third-parties under contract to IFC in the quarter that had not
received services previously, i.e., a new client (by type of beneficiary)
Number of unique organizations to which advisory services were provided by
IFC and third-parties under contract to IFC in the quarter that had received
services previously, i.e., a repeat client (by type of beneficiary)
Number of new laws/regulations drafted or contributed to the drafting.
Number of laws/regulations to which improvements have been
recommended.
Number of training events (courses, seminars, workshops, …) conducted by
IFC and third-parties under contract to IFC in the quarter
Number of participants in training events conducted by IFC and third-parties
under contract to IFC in the quarter (by type of beneficiary)
Number of participant-training hours provided by IFC and third-parties under
contract to IFC in the quarter (by type of beneficiary)
Number of unique organizations that sent individuals for training conducted
by IFC and third-parties under contract to IFC in the quarter (by type of
beneficiary)
Number of unique organizations that sent individuals for training conducted
by IFC and third-parties under contract to IFC in the quarter that had not sent
individuals previously, i.e., a new client (by type of beneficiary)
Number of unique organizations that sent individuals for training conducted
by IFC and third-parties under contract to IFC in the quarter that had sent
individuals previously, i.e., a repeat client (by type of beneficiary)
Number of substantive media reports produced in association with IFC and
third-parties under contract to IFC in the quarter
Number of press releases distributed
Number of TV/radio spots aired
Number of unique visitors to Web site developed by IFC and third-parties
under contract to IFC in the quarter (if website is a result of the project being
monitored)
Number of public information events held or sponsored by IFC and thirdparties under contract to IFC in the quarter
Number of participant at public information events held or sponsored by IFC
and third-parties under contract to IFC in the quarter
Data Source
Program
records
Program
records
Program
records
Program
records
Program
records
Program
records
Program
records
Program
records
Program
records
Program
records
Program
records
Program
records
Program
records
Program
records
Program
records
Program
records
Program
records
Program
records
Program
records
7
8. The output indicators for advisory services and training ask for a breakdown by type of
beneficiary, i.e., BDS providers, public institutions, business organizations, financial
institutions, and enterprises. With respect to the latter, if required, facilities should also keep
track of outputs by size of firm to determine the extent to which services have been
delivered to SMEs.
9. While advisory services and training are treated separately, a particular organization may
receive both types of services during the quarter. As such, adding the number of
organizations that receive advisory services to the number of organizations that sent people
for training would overstate the number of organizations that actually received services
during the period – some would be double counted.
10. The indicator “number of training participants” refers to the number of people that attended
training events in the quarter. Although, it would be preferable to count the number of
unique individuals that participated in training events this would require an information
system that assigns a unique code to individuals and an ability to determine whether the
same individual attended more than one training event in the quarter.
Client Satisfaction with Training or Advisory Services
11. Facilities are interested in measuring the level of satisfaction of organizations that received
services. While a five-point scale should be used in the survey, the indicator sums the
percentage “very satisfied” and “satisfied.”
Construct
Indicator
Satisfaction with
services provided by
IFC and third-parties
under contract to IFC
Percent of clients indicating being very satisfied or satisfied with
the service (Five-point scale where 5=very satisfied,
4=satisfied, 3=neutral, 2=dissatisfied, and 1=very dissatisfied)
Data
Source
Client
satisfaction
survey
8
Outcome and Impact Indicators for Access to Finance
12. Core outcome and impact indicators for the access to finance program aim to measure
characteristics of financial markets and financial institutions. With respect to the latter, some
projects may be directed at all financial institutions operating in the market; others may only
involve a sub-set of institutions that receive services directly from IFC and/or third parties
under contract to IFC. IFC facilities need to consider the purpose of particular projects in
determining the population to be measured.
13. Although some facilities expressed the need to assess access to financing for private
enterprises regardless of size, impact indicators should be broken down by size of loan as
proposed by the Financial Infrastructure & Institution Building Department of IFC.5
14. Measures of profitability of financial institutions such as the return on assets were excluded
from the list of core indicators due to perceived difficulty in obtaining necessary information.
5
“Access to Finance: Measuring Results to Improve Technical Assistance”, Nataliya Mylenko, May 10 2005.
9
Outcome and Impact Indicators for Access to Finance
Construct
Outcomes
Change in laws and
regulations related to FI
Learning in terms of
knowledge and skills
Introduction of new financial
products
Indicator
Data Source
Population
Number of laws and regulations promulgated (new or amended) during the
year where IFC or third-parties under contract to IFC played a role in the
process
Percent of participants strongly agreeing or agreeing with the statement that
they have obtained new knowledge and skills as a result of training (Fivepoint scale where 5=strongly agree, 4=agree, 3=neutral, 2=disagree, and
1=strongly disagree)
Number of participants reporting having obtained new knowledge and skills
as a result of training
Program
records
N/A
Training
Participant
Survey
Training participants
Training
Participant
Survey
FI survey
Training participants
Regulatory
authority or FI
survey
Financial institutions
Regulatory
authority or FI
survey
Regulatory
authority
Credit bureau
or FI survey
Financial institutions
New financial products (standard checklist) introduced into the market
during the year
Standard checklist:
•
Commercial loans
•
Leasing
•
Factoring
•
Microfinance products
•
Housing finance (mortgage and home equity)
•
Insurance
•
Other. Please specify __________
Competition in financial
markets
Number of financial institutions providing specific financial products
(standard list) as of year end
Financial institutions
/ Participating
financial institutions
Standard list:
•
Commercial loans
•
Leasing
•
Factoring
•
Microfinance products
•
Housing finance (mortgage and home equity)
•
Insurance
•
Other. Please specify __________
Number of active participants in relevant financial markets, e.g., commercial
lending, leasing, equity, etc.
Establishment of collateral
registries or credit bureaus
Use of collateral or credit
information
Number of collateral registries or credit bureaus legally registered as of year
end
Number of FIs that purchased credit reports during the year
N/A
Financial institutions
10
Number of new loans that were secured by registered capital during the
year (by financial product)
Process improvement in
participating financial
institutions
Number of participating FIs reporting change in business processes
(standard list) during the year
Collateral
registry or FI
survey
Financial institutions
FI survey
Participating
financial institutions
FI survey
Participating
financial institutions
FI survey
Participating
financial institutions
FI survey
Participating
financial institutions
Standard list:
•
Adopted new management practices
•
Adopted good corporate governance practices
•
Adopted new loan approval process
•
Adopted new technologies
•
Introduced new or improved products
•
Introduced new distribution channels
•
Entered new markets
•
Other. Please specify __________
Proportion of participating FIs reporting change in business processes
(standard list) during the year
Standard list:
•
Adopted new management practices
•
Adopted good corporate governance practices
•
Adopted new loan approval process
•
Adopted new technologies
•
Introduced new or improved products
•
Introduced new distribution channels
•
Entered new markets
•
Other. Please specify __________
Number of participating FIs reporting improved performance (standard list)
during the year
Standard list:
•
Higher quality of portfolio
•
Lower transaction costs
•
Faster turnaround on loan processing
•
Other. Please specify __________
Proportion of participating FIs reporting improved performance (standard
list) during the year
Standard list:
•
Higher quality of portfolio
•
Lower transaction costs
11
•
•
Faster turnaround on loan processing
Other. Please specify __________
Rate of non-performing
loans
Value (US$) of non-current enterprises loans / gross loan portfolio by
financial product as of year end
FI survey
Financial institutions
/ Participating
financial institutions
Impacts
Supply of credit to
enterprises
Value (US$) of outstanding loans by size of loan as of year end (by financial
product)
FI survey
Number of outstanding loans by size of loan as of year end (by financial
product)
FI survey
Value (US$) of loans disbursed by size of loan during the year (by financial
product)
FI survey
Number of loans disbursed by size of loan during the year (by financial
product)
FI survey
Financial institutions
/ Participating
financial institutions
Financial institutions
/ Participating
financial institutions
Financial institutions
/ Participating
financial institutions
Financial institutions
/ Participating
financial institutions
12
Outcome and Impact Indicators for Business Linkages
15. Outcome and impact indicators for business linkages should be applied to
community/cluster development and supply chain development projects (as detailed in the
program logic model). All projects undertaken under the business linkages program are
partial coverage with certain organizations in the target population electing to participate.
Indicators focus on measuring results among participants.
16. While not included in the list of core indicators, some facilities may also want to assess
changes in sales, employment and profitability among the large firms at the center of supply
chain projects.
13
Outcome and Impact Indicators for Business Linkages
Construct
Outcomes
Advisory services and/or
training delivered by BDS
providers to enterprises
Learning in terms of
knowledge and skills
Financing provided to
participating enterprises
Business process
improvement in participating
enterprises
Indicator
Number of unique enterprises assisted and trained by participating BDS
providers during the year
Percent of participants strongly agreeing or agreeing with the statement that
they have obtained new knowledge and skills as a result of training (Fivepoint scale where 5=strongly agree, 4=agree, 3=neutral, 2=disagree, and
1=strongly disagree)
Number of participants reporting having obtained new knowledge and skills
as a result of training
Number of unique participating enterprises that received external financing
facilitated through the program during the year
Proportion of unique participating enterprises that received external financing
facilitated through the program during the year
Value (US$) of loans disbursed by financial institutions or large firm to
participating enterprises during the year
Value of outstanding SME loan portfolio for SMEs that received financing
through the program
Number of unique participating enterprises reporting change in business
processes (standard list) during the year
Data Source
Population
BDS survey
Participating
BDS providers
Training
participant
survey
Training
participants
Training
participant
survey
Program
records
Program
records
Program
records
Program
records
Enterprise
survey
Training
participants
Participating
enterprises
Enterprise
survey
Participating
enterprises
N/A
N/A
N/A
N/A
Standard list:
•
Adopted new management practices
•
Adopted new process technologies
•
Adopted measures to improve working conditions
•
Adopted measures to reduce environmental impacts
•
Introduced new or improved products
•
Introduced new distribution channels
•
Entered new markets
•
Other. Please specify __________
Proportion of unique participating enterprises reporting change in business
processes (standard list) during the year
Standard list:
•
Adopted new management practices
•
Adopted new process technologies
•
Adopted measures to improve working conditions
•
Adopted measures to reduce environmental impacts
•
Introduced new or improved products
14
•
•
•
Introduced new distribution channels
Entered new markets
Other. Please specify __________
Number of unique participating enterprises reporting improved performance
(standard list) during the year
Enterprise
survey
Participating
enterprises
Enterprise
survey
Participating
enterprises
BDS survey
Participating
BDS providers
N/A
Standard list:
•
Higher quality
•
Faster order-to-delivery time
•
Higher rate of on-time delivery
•
Lower unit cost
•
Higher productivity
•
Other. Please specify __________
Proportion of unique participating enterprises reporting improved
performance (standard list) during the year
Standard list:
•
Higher quality
•
Faster order-to-delivery time
•
Higher rate of on-time delivery
•
Lower unit cost
•
Higher productivity
•
Other. Please specify __________
Sustainable BDS providers
Percentage of BDS total costs covered by fees paid by clients during the year
Establishment of Supply
Chain Development Program
in Large Firm
Number of supply chain development programs established during the year
Program
records
15
Impacts
Enterprise-wide performance
of participating enterprises
Total number of employees of participating enterprises as of year end
Total gross annual sales (US$) of participating enterprises during the year
Purchases by large firm from
local SME suppliers
Total value (US$) of gross annual sales of participating enterprises to large
firm during the year
Percent of gross annual sales of participating enterprises to large firm during
the year
Number of participating enterprises reporting an increase in gross annual
sales compared to previous year
Proportion of participating enterprises reporting an increase in gross annual
sales compared to previous year
Number of participating enterprises reporting an increase in net profit margin
compared to previous year
Proportion of participating enterprises reporting an increase in net profit
margin compared to previous year
Number of participating enterprises reporting an increase in employment
compared to previous year
Proportion of participating enterprises reporting an increase in employment
compared to previous year
Number of local suppliers from which large firm purchased goods and
services during the year
Number of local SME suppliers with active contracts with large firm
Purchases (US$) from local suppliers by large firm during the year
Enterprise
survey
Enterprise
survey
Enterprise
survey
Enterprise
survey
Enterprise
survey
Enterprise
survey
Enterprise
survey
Enterprise
survey
Enterprise
survey
Enterprise
survey
Large firm
survey
Large firm
survey
Large firm
survey
Participating
enterprises
Participating
enterprises
Participating
enterprises
Participating
enterprises
Participating
enterprises
Participating
enterprises
Participating
enterprises
Participating
enterprises
Participating
enterprises
Participating
enterprises
Participating
large firms
Participating
large firms
Participating
large firms
16
Outcome Indicators for Business Enabling Environment
17. Business Enabling Environment program includes a broad range of projects classified as
business registration, business licenses and permits, business inspections, contract
enforcement, and property registration. As noted below, indicators need to be tailored to the
particular type of project undertaken by IFC.
17
Outcome Indicators for Business Enabling Environment
Construct
Outcomes
Learning in terms of
knowledge and skills
Changes in the law and
regulations related to
business enabling
environment
Indicator
Data Source
Population
Percent of participants strongly agreeing or agreeing with the statement that they
have obtained new knowledge and skills as a result of training (Five-point scale
where 5=strongly agree, 4=agree, 3=neutral, 2=disagree, and 1=strongly
disagree)
Number of participants reporting having obtained new knowledge and skills as a
result of training
Training
participant
survey
Training
participants
Training
participant
survey
Program
records
Training
participants
Program
records
N/A
Doing Business
N/A
Enterprise
survey
Enterprises that
completed
procedures in
the last 12
months
Number of laws and regulations promulgated (new or amended) during the year
where IFC or third-parties under contract to IFC played a role in the process (by
BEE project type)
N/A
BEE project type:
•
Business registration
•
Business licenses and permits
•
Business inspections
•
Contract enforcement
•
Property registration
•
Other. Please specify ___________
Changes in relevant
administrative procedures
Number of administrative procedures changed during the year where IFC or thirdparties under contract to IFC played a role in the process (by BEE project type)
BEE project type:
•
Business registration
•
Business licenses and permits
•
Business inspections
•
Contract enforcement
•
Property registration
•
Other. Please specify ___________
Transaction cost for specific
procedures
Starting a business and dealing with licenses: Official cost of each procedure
(as a percentage of income per capita)
ƒ Registering Property: Official cost (as a percentage of the property value)
ƒ Enforcing contracts: Official costs (as a percentage of the debt value)
Average cost born by companies to complete specific procedures, including
government fees, payments to attorneys and other third-parties, and unofficial
payments to government officials. (by BEE project type)
ƒ
BEE project type:
•
Business registration
•
Business licenses and permits
18
•
•
•
•
Days required to complete
specific procedures
Business inspections
Contract enforcement
Property registration
Other. Please specify ___________
Average time, in calendar days, spent to complete each procedure (by BEE
project type)
BEE project type:
•
Business registration
•
Business licenses and permits
•
Business inspections
•
Contract enforcement
•
Property registration
•
Other. Please specify ___________
Corruption
Percentage of firms that say that corruption is a major or severe obstacle to the
operation and growth of their businesses
Percentage of firms that made unofficial payments to public officials during the
process (by BEE project type)
Doing
Business
Enterprise
survey
Investment
Climate
Survey
Enterprise
survey
BEE project type:
•
Business registration
•
Business licenses and permits
•
Business inspections
•
Contract enforcement
•
Property registration
•
Other. Please specify ___________
Value of unofficial payments made by firms to public officials
Policy advocacy by participating
business organizations
Number of participating business organizations playing an active role in policy
advocacy during the year
Proportion of participating business organizations playing an active role in policy
advocacy during the year
Sustainable business
organizations
Number of members paying membership fees during the year
Enterprise
survey
Business
organization
survey
Business
organization
survey
Business
organization
survey
N/A
Enterprises
that
completed
procedures in
the last 12
months
N/A
Enterprises
that
completed
procedures in
the last 12
months
Enterprises
that
completed
procedures in
the last 12
months
Participating
business
organizations
Participating
business
organizations
Participating
business
organizations
19
Proportion of members paying membership fees during the year
Percentage of business organizations total annual costs covered by membership
or service fees paid by enterprises during the year
Impacts
Rate of business registration
Number of businesses registered during the year
Private sector productivity and
growth
Percentage of enterprise reporting that the business enabling environment is
conducive to their productivity and growth
Business
organization
survey
Business
organization
survey
Participating
business
organizations
Participating
business
organizations
Regulatory
authority
Enterprise
Survey
N/A
Enterprises
that
completed
procedures in
the last 12
months
20
III.
Data collection
Overview
18. Successful measurement depends on the quality of data collected through program records,
surveys, and secondary sources. Data should be collected in a consistent manner using
agreed definitions and procedures, and stored in appropriate computer databases to
facilitate data access, analysis and reporting.
19. The following table lists the sources of data needed to calculate core indicators and the
recommended frequency of data collection efforts. Program records detailing the nature
and magnitude of activities undertaken by IFC and associated outputs should be
continuously updated. Surveys used to assess client satisfaction and learning
outcomes should be conducted upon project completion as needed. Other surveys
should be undertaken before programs are initiated to establish needed baselines and
repeated annually (as budget allows) in order to monitor changes. Data should also be
collected from secondary sources on an annual basis.
Data sources
Program records
Surveys
Client satisfaction
Training participants
BDS providers
Business organizations
Financial institutions
Enterprises
Secondary sources
Collateral registries
Credit bureaus
Regulatory authorities
World Bank - Doing Business
World Bank - Investment Climate
Frequency of data collection
Ongoing
Upon project completion
Upon completion of training
Baseline and annually thereafter
Baseline and annually thereafter
Baseline and annually thereafter
Baseline and annually thereafter
Baseline and annually thereafter
Baseline and annually thereafter
Baseline and annually thereafter
Baseline and annually thereafter
Baseline and annually thereafter
Program records
20. IFC facilities should maintain complete and accurate records. This should include data on
the characteristics of organizations receiving technical assistance from IFC or third parties
under contract to the IFC, including intermediaries and private enterprises. It should also
include data on particular projects, including the type of activity (assessment, advisory
services, training, and information dissemination), participants, service providers, date of
initiation and completion, and budget expenditures.
Surveys
21. As noted above, IFC will need to conduct a variety of surveys to collect requisite data. In
order to ensure the quality of data, the following procedures should be followed:
•
Questionnaires. IFC facilities should use instruments that contain questions needed to
obtain data required for relevant indicators. Questions should be worded in the same
21
manner with any translations checked to ensure that meanings have not been altered
inadvertently. (This will require the development of standard survey instruments.)
•
Sampling. The goal is to have a sample that is representative of the population and
therefore can be used to make valid generalization. Unless a census is appropriate,
IFC facilities should survey a random sample of organizations drawn from the
appropriate set of program participants or the target population as a whole.6 A
random sample is where each entity in the sample frame has a known and
independent probability of being selected for the sample.7 The size of the sample
should be large enough to provide sufficient statistical power.8 Although there are no
formal standards for statistical power, IFC facilities should aim to draw a sample that
would provide a power of 0.8 or greater.
•
Administration. Given the nature of the information sought, most surveys should be
administered in-person (as opposed to mail or telephone) with a strict promise to
protect the confidentiality of the respondent and their responses. Field personnel
should be trained to conduct the surveys. All survey should seek to achieve a high
response rate (at least 60 percent) to reduce potential response bias. To help ensure
a high response rate, IFC facilities should obtain the commitment of participants to
respond to surveys as a condition of program participation.
•
Data entry. IFC facilities should establish specific procedures for dealing with
completed surveys. This includes tracking responses so that individuals failing to
respond initially can be contacted and encouraged to complete the questionnaire.
The quality of data entry should be verified by checking all or a sample of
questionnaires for accuracy and by carefully examining data for responses that are
not consistent. All questionable entries should be checked for problems and verified.
Original copies of written questionnaires should be kept on file.
•
Survey schedule. March is generally a good time to administer surveys to
businesses in that it allows sufficient time for companies to close their books after
the end of the fiscal year. (Most fiscal years end December 31st.)
Secondary sources
22. Data required to calculate certain indicators will need to be obtained from secondary
sources such as collateral registries, credit bureaus, regulatory authorities, and surveys
conducted by other parts of the World Bank. IFC will need to work with these organizations
to ensure that data are accurate and provided in a consistent manner.
6
To facilitate surveys, agreements should be put in place with intermediaries to provide a complete list of
enterprises that have received services along with contact information.
7
Other sampling methods such as convenience sampling or purposive sampling are likely to be biased.
8
The power of a statistical test is the probability that the test will reject a false null hypothesis (Type II error).
Higher power increases the chance of obtaining a statistically significant result when the null hypothesis is false.
Statistical power depends on the degree of variance in the underlying data, the level of confidence desired, and the
effect size that researchers want to be able to detect.
22
Budget guidelines
23. The budget for data collection and analysis should be commensurate with the scale of the
project and/or its importance. As a general rule of thumb, M&E budgets should be in the
order of three to five percent of the total program budget.
23
Appendix A. Program Logic Models
PLM Access to Finance
PLM Business Linkages
PLM Business Enabling Environment
24
Activities
Outcomes
Impacts
Outside boundaries
of accountability
25
Activities
Outcomes
COMMUNITY
DEVELOPMENT
Development of
local
infrastructure
Impacts
Outside boundaries
of accountability
Lower cost of production for
related goods and services
Advisory
services and/or
training delivered
directly to
enterprises
within targeted
geographic/
industrial cluster
Financing
provided to
participating
enterprises
Brokerage deals
with financial
institutions
Establishment of
new BDS providers
Advisory
services and/or
training delivered
directly to BDS
providers
Change in business
processes in BDS
providers
Advisory
services and/or
training delivered
to participating
enterprises
Change in
business
processes
and/or
new capital
investment by
participating
enterprises
Improved performance
of participating
enterprises
Higher Quality
Faster Turnaround
Lower Cost
Greater Capacity
Sustainable BDS
providers
IFC invesment
Establishment /
expansion of
large firm
SUPPLY CHAIN
DEVELOPMENT
Improved enterprise-wide
performance of participating
enterprises
and/or
Increased demand for intermediate goods and services
Higher income
(value added)
in region
Productivity
Profitability
Sales
Employment
Increased sales of large firm
through local distributors
Adoption of targets
for local content by
large firm
Establishment of
supplier
development
program in large
firm (inc,
certification process
and criteria
Advisory
services and/or
training
delivered directly
to large firm
Increased purchases by large
firm from local suppliers
Identification of
non-core businesses
in large firm
Establishment of
franchise program
or decision to spinoff retail operations
(inc, certification
process and criteria)
Advisory
services and/or
training delivered
to suppliers and/
or distributors
Financing
provided to
suppliers and/or
distributors by
large firm
Change in
business
processes
and/or
new capital
investment by
participating
enterprises
Improved performance
of participating
enterprises
Higher Quality
Faster Turnaround
Lower Cost
Greater Capacity
Increased in number of
qualified suppliers and/
or distributors
Lower input /
distrbution costs
Increased
productivity in
large firm
Increased sales
of large firm
26
BUSINESS REGISTRATION
AND BUSINESS LICENSES /
PERMITS
Activities
Outcomes
Changes in laws and
regulations related to business
registration and business
licenses / permits
LEGAL AND
REGULATORY REFORM
Impacts
Outside boundaries
of accountability
Advisory
services
delivered to
Executive /
Legislative
entities
Greater awareness of
need for reform and
knowledge of specific
legal and regulatory
matters
Advisory services and/or
training delivered to
registration and business
licenses /permits agencies
Public awareness campaign
Assessment
Studies, e.g.,
Investment
Climate
Assessments /
Doing Business
surveys / SME
Maps / Etc.
Lower Fees
Implementation of new
administrative
procedures
Lower
transaction cost
Fewer required
days
Increased knowledge of
procedures and rights
Lower cost of
business
registration and
business
licenses /
permits
Increased rate of
business registration
(Lower rate of
informality)
Lower corruption
BUSINESS INSPECTION
Changes in laws and
regulations related to business
inspection
Fewer
inspections
Implementation of
new inspection
procedures
Advisory services and/or
training delivered to inspection
agencies
Increased knowledge of
procedures and rights
Public awareness campaign
Lower cost of
business
inspection
Lower
transaction cost
Lower corruption
CONTRACT ENFORCEMENT
Changes in laws and
regulations related to contract
enforcement
Implementation of
new contract
enforcement
procedures
Advisory services and/or
training delivered to legal
institutions
Public awareness campaign
BUSINESS ORGANIZATION
DEVELOPMENT
Advisory
services and/or
training delivered
to business
organizations
Increased knowledge of
procedures and rights
PROPERTY REGISTRATION
Establishment of
new business
organizations
Change in business
processes in
business
organizations
Advisory services and/or
training delivered to property
agencies
Public awareness campaign
Lower
transaction cost
Fewer required
days
Lower cost of
contract
enforcement
Increased
Investment
Lower corruption
Greater ability to
secure financing
Reliable title
Changes in laws and
regulations related to property
registration
Policy advocacy by
business organizations
Increased productivity
and growth of
enterprises
Lower court fees
Lower fees,
transfer taxes,
stamp duties
Implementation of
new property
registration
procedures
Increased knowledge of
procedures and rights
Lower
transaction cost
Fewer required
days
Lower cost of
establishing and
transferring title
Lower corruption
SECTOR-SPECIFIC
Sustainable
business
organizations
Changes in laws and
regulations related to specific
industries
Specific firm-level
outcomes
27
Appendix B. Definitions of Key Terms
Term
Activities
Advisory services
Advisory service hour
Administrative
procedures
Assessments
Beneficiaries
Employees
Enterprises
Gross loan portfolio
Impacts
Inputs
Loans disbursed
Loan size
New client
New products
Definition
Tasks performed to produce outputs using resources and methods.
Consulting services provided to an organization on an individual or group
basis.
Advisory services are measured in hours. If conversion from days to hours
is needed, an 8-hour day should be used. Advisory service time includes
time spend in all activities specified under the terms of reference, e.g. report
preparation, meetings (in-person or over telephone), and travel.
Steps required to comply with a particular regulation, e.g., business
registration, licensing and permitting, inspection, or contract enforcement.
Reports that examine issues related to particular regions, markets, and/or
industries. This includes market studies, needs assessment, and/or policy
studies. (Other terms may be used by particular facilities.)
Individuals or organizations that are intended to benefit directly or indirectly
from the intervention. Types of beneficiaries include: BDS providers, public
institutions, business organizations, financial institutions, and enterprises.
Full- and part-time workers as of a specific date or pay period, e.g., the week
of March 12th
Enterprises are businesses. To be classified within a specific category, an
enterprise must have at least two of the three characteristics (a,b,c):
ƒ Microenterprise
– (a) 10 or less Employees
– (b) US$ 100,000 or less in Total Assets
– (c) US$ 100,000 or less in Total Annual Sales
ƒ Small enterprise
– (a) 10 < Employees ≤ 50
– (b) US$ 100 000 < Total Assets ≤ US$ 3 million
– (c) US$ 100 000 < Total Annual Sales ≤ US$ 3 million
ƒ Medium enterprise
– (a) 50 < Employees ≤ 300
– (b) US$ 3 million < Total Assets ≤ US$ 15 million
– (c) US$ 3 million < Total Annual Sales ≤ US$ 15 million
The outstanding principal balance of all outstanding loans of a particular
financial institution. This includes current, delinquent and restructured loans.
Final intended outcomes with respect to benefits accruing to the target
population and/or broader economy.
Financial, human and material resources used to perform activities.
Value of all loans paid out during a specified period of time regardless of
their current condition (i.e. performing, non-performing or written off).
Loans are classified in three categories: micro (< US$ 10,000), small
(<US$100,000) and medium (<US$ 1 million for most countries and US$2
million for some more advanced countries, such as Argentina, Brazil, Chile,
China, Colombia, India, Korea, Mexico, Morocco, Peru, Russia, South Africa,
Thailand, Tunisia, Turkey, and all EU accession countries- Poland, Hungary,
Czech Republic, Slovakia, the Baltics and Slovenia).
An organization (e.g. enterprise, financial institution, business organization,
BDS provider, government agency) that received technical assistance from
IFC or third-party providers under contract with IFC in the period, but had not
received assistance previously.
In the case of Access to Finance projects, the introduction of new products
28
introduced to the
market
Non-performing loans
(NPL)
Outcomes
Outputs
Outstanding loans
Participant-training
hour
Participants
Policy advocacy
Population
Project
Public information
events
Region
Repeat client
SME loans
Substantive media
reports
Supply chain
development programs
Surveys
Target population
to the market denotes the introduction of leasing, factoring, mortgage or
other major financial instrument into a particular market for the first time.
(Also know as Portfolio at Risk). Loans outstanding that have one or more
installments of principal past due by at least 90 days. This includes the
entire unpaid principal balance, i.e. past due and future installments but not
accrued interest. (Restructured loans may be included in the calculation of
NPL since this type of loan carries higher risk than current loans.)
Changes in knowledge, behavior (new practices) and performance of direct
and indirect beneficiaries.
Products of program activities.
Loans that have not been fully repaid or written off.
It is an hour of training received by an individual during a specific training
event. Total participant-training hours for a single event are calculated by
multiplying the total number of participants times the number of hours of inclass instruction.
Individuals or organizations that have received assistance through the
program. Organizations include enterprises, government agencies, financial
institutions, business organizations, and BDS providers.
Activities undertaken to influence the adoption of particular laws and/or
regulations. Policy advocacy may include lobbing and public awareness
campaigns.
The group targeted by the intervention to which results could be generalized.
It is the group of individuals or organization from which a sample is drawn.
(Technically the sample is drawn from a sample frame.)
An intervention that includes a set of planned activities designed to achieve
established objectives within a given budget and specified period of time.
Activities undertaken as part of public awareness campaigns. IFC may
sponsor, organize, and/or participate in these events.
The geographic area that is the subject of the IFC intervention. A region
could be a country, state, city or other area.
An organization (e.g. enterprise, financial institution, business association,
BDS provider, government agency) that has received assistance on more
than one occasion.
As a proxy, loans to Enterprises are defined by the size of the loan: micro (<
US$ 10,000), small (<US$100,000) and medium (<US$ 1 million for most
countries and US$2 million for some more advanced countries, such as
Argentina, Brazil, Chile, China, Colombia, India, Korea, Mexico, Morocco,
Peru, Russia, South Africa, Thailand, Tunisia, Turkey, and all EU accession
countries- Poland, Hungary, Czech Republic, Slovakia, the Baltics and
Slovenia)
Stories or published interviews focusing on projects undertaken by IFC.
Substantive media reports can be published through radio, TV, newspapers,
or other media.
A program organized within a large firm to procure goods and services from
local providers in which the firm provides some form of assistance to the
suppliers. The intervention would take place at any point of the supply chain
to improve quality, costs and delivery issues.
Data collection techniques designed to collect standard information from a
large number of subjects. Surveys may include polls, mailed questionnaires,
telephone interviews, or face-to-face interviews.
Intended direct and/or indirect beneficiaries of programs. Target populations
include BDS providers, government agencies, financial institutions, business
organizations, and private enterprises.
29
Training events
Unique
Unique visitors to
Web site
An activity intended to build the knowledge or enhance the skills of
participants. Only training activities with duration of three or more hours
should be included. An event is defined by terms of reference for training.
The term unique refers to a particular company or organization. Unique
implies that the entity should be counted only once. This will require an
information system that includes a unique identifier for each company and
organization.
Represents an unduplicated (counted only once) visitor to a Web site over
the course of a specified period, based on hostname or numerical IP
address. The implementation of specialized software will be required to
track unique visitors.
30
Appendix C. A Note on Impact Assessment
24. Impact assessment are undertaken to find out whether a particular intervention actually
produced desired results. In this regard, impact assessments are concerned with
establishing causality. In demonstrating that a particular intervention resulted in a specific
outcome, certain conditions need to be met.9 First, changes engendered through the
intervention have to be shown to produce the effect – put another way, the outcome must be
responsive to the intervention. Second, plausible alternative explanations for the observed
outcome have to be ruled out – rival hypotheses must be disproved. Third, the mechanism
by which the outcome was produced has to be explained – a theory linking the intervention
to the outcome must be articulated. Finally, it must be possible to replicate the results in
similar settings. With proper research, apparent correlations can be translated into credible
causal explanations.
25. Impact assessments center on a comparison of the situation with the intervention to what
would have been had there been no intervention at all. The difference in resulting outcomes
between these two states constitutes the impact of the intervention as illustrated in the
following figure.10 While the counterfactual cannot be observed or known with complete
certainty, the concept of comparing observed outcomes to this hypothetical state underlies
all valid approaches to assessing impacts. Valid comparisons imply that the net effect of
interventions is isolated from all other extraneous or confounding factors that influence
defined outcomes.
Outcome
Observed outcome
with intervention
}
Impact
What would have happened
without intervention
Time
26. The major challenge in impact assessments is to estimate the effect of programs after
netting out extraneous confounding factors that affect outcomes. Threats to internal validity
are generally grouped under several broad categories such as external events, secular drift,
maturation, regression and attrition.11 Another threat to the validity of claims concerning the
impact of initiatives arises from the voluntary nature of programs. Some members of the
target population may be more inclined to participate due to greater interest, motivation or
other conditions. This self-selection process can bias results if the factors that lead
organizations to participate are related to the specific outcomes under study.
9
See Mosteller and Tukey (1977).
This is sometimes referred to as “additionality.”
11
Readers interested in exploring these concepts in more detail are referred to Cook and Campbell (1979).
10
31
27. The strength of causal inferences that can be drawn from an analysis of particular
intervention depends on how well the particular approach used in assessing impacts deals
with the threats to validity. While there are numerous variations, impact assessment can be
grouped under four basic headings based on the type of controls used to isolate program
effects from other confounding factors – experiments with random assignment, quasiexperiments with constructed controls, before-after studies with reflexive controls, and
participant judgment and expert opinion.12 Each of these is discussed below.
Method
Experiments with
random
assignment
ƒ
ƒ
ƒ
ƒ
ƒ
Quasi-experiments
with constructed
controls
ƒ
ƒ
ƒ
ƒ
Before-after studies
with reflexive
controls
ƒ
ƒ
ƒ
Participant
judgment and
expert opinion
ƒ
ƒ
ƒ
ƒ
Description
Entities in the treatment group participate in the program; those in the control group
receive alternative services or none at all.
Random assignment helps guarantee that the two groups are similar; extraneous factors
that influence outcomes are present in both groups.
Because of this comparability, claims that differences between the two groups are the
direct result of the program are more difficult to refute.
Experimental designs are used quite frequently to test the efficacy of new treatments in
health, social welfare and education.
However, this approach is often difficult to implement
– Political considerations make random assignment difficult
– The services provided to entities may not be standardized
– It is frequently hard to maintain experimental conditions
– Experiments tend to be costly and difficult to administer
The performance of participating entities is compared to other similar entities that have not
participated. However, instead of random assignment, a comparison group is constructed
after the fact.
Valid comparisons require that the two groups be similar with respect to key
characteristics, exposure to external events and trends, and propensity for program
participation.
To the extent that the two groups are similar, observed differences can be attributed to the
program with a high degree of confidence.
There are several types of designs that fall within this general category: regression
discontinuity, statistically equated controls, matched controls, and generic controls.
Many programs have been evaluated by comparing the performance of program
participants before and after the intervention, attributing all of the change to the program.
While widespread, results from studies that rely exclusively on reflexive controls should be
treated with substantial skepticism.
Before-and-after design cannot isolate the impact of the program from these extraneous
factors such as selection bias, maturational trends, secular drift and interfering events.
This approach relies on program participants or independent experts to make judgments
concerning impacts.
Individuals are asked to estimate the extent to which performance was enhanced as a
direct result of the program – in effect, to compare current performance to what would
have happened in the absence of the program.
This approach requires people to be able to determine the net effect of the intervention
based solely on their own knowledge.
It is often the only option available given data and budget constraints. When used, care
should be taken to make sure that people consider the counterfactual in their assessment
of impacts.
12
Case studies can also be used to examine the impacts of intervention. Unlike the other approaches, case studies
rely on extensive narrative descriptions and other evidence to assert that the intervention caused observed outcomes.
In general, case studies involve multiple sources of information including direct observation, interviews, documents,
and physical artifacts. While case studies can provide rich explanations of how and why the program affected
particular firms, it is difficult to generalize results beyond the firms studied.
32