Impact evaluation-approaches and methodologies
Pursuit for Effective Impact Evaluation Tools for
Market Development Programmes
Most Significant Change (MSC); Methodology
and Application
MSC Integration in Market Development
Programs; Rationale and Case for MEDA
Prosperity through Innovation Zambia Project
Results, Benefits and Lessons Learned in
integrating MSC in Market Development
Programs
Conclusion
It’s better to be approximately
right than precisely wrong.
John Maynard Keynes
Impact evaluation - as the component of a project cycle that seeks
to validate to what extent changes occurring to target groups can
be attributed to project interventions - is a topical issue in the
development sector.
 Impact evaluation is a necessary part of project planning and
monitoring and for improving implementation strategies
regardless of type of organization
 For evaluators the challenge now is how to adapt the basic
approaches of M&E (qualitative and quantitative) to develop an
innovative and cost effective M&E model
 For evaluators of market development projects developing
models is even more of a challenge due to the complexity of the
market itself (supply and demand side)

1. Market intervention indicators
-Relating to outputs, measuring the immediate results of project activities e.g
sales performance of products, increased stock volumes, size and expanse
of retail networks
2. Market development indicators
-Relating to outcomes, measuring results over space and time including access
to information on products location, price and benefits, impressions of
specified products to drive demand, supply side development and
adherence to business and marketing plans, suppliers capabilities in
product stocking and marketing. Others may includes adherence to
product quality, assessment of the users knowledge on how to use
technology, their purchasing power and demand.
3. Market sustainability indicators
-Relating to impacts, measuring results when the project intervention is withdrawn
Market development evaluators need to go
beyond measuring market intervention indicators
as the success of a market development project
depends on sustained product adoption
 Market development indicators are difficult to
track because they are qualitative
 The MEDA experience has shown that the
adoption of MSC as an evaluation technique
can be an effective tool in assessing market
development projects and specifically market
development indicators

•
Most Significant Change (MSC) is a participatory,
qualitative and complimentary approach to impact
evaluation.
•
MSC can be likened to newspaper development. Where
stories are collected and prioritised. In this case, however,
groups of beneficiaries are asked the MSC question and a
panel picks the “best” MSC story. These stories are then
compiled into a “catalogues/newspaper” and disseminated
to decision makers. There can be many levels of sifting
stories depending on the size or goal of MSC enquiry.
•
The MSC approach was originally developed in 1994 by
Rick Davies through his work on a rural development
project in Bangladesh with the Christian Commission for
Development Bangladesh (CCDB) and has been adopted
by a number of organizations in different countries
•
MEDA has applied MSC to measure market development
indicators to be specific issues around sustainable adoption
of SITT’s in the value chain.
What is Most
Significant
Change?
Steps of MSC
(Davies et al 2005)
• Using an ediscount
voucher as a
market
catalyst
• Building the
capacity of
water
technology
suppliers and
dealers to
serve farmers
better
Project Strategy
• Accelerating
the demand
and supply for
appropriate
and
affordable
water
technologies
Project Approach
Project Goal
The MEDA Project - Prosperity Through Innovation
• Pure market
development
facilitation
• Invisibility for
effective
market
sustainability
and project
exit strategy
Partner
Organisation;
Irrigation
Trainers
BUSINESS SUPPORT SERVICES
Organization interventions or
Member Services
Provision of Training on Irrigation
Business Transactions on
water technologies
Smallholder
Farmers
Water
Technology
Suppliers and
Retailers
MEDA
Supply side strengthening through capacity building on
marketing and cross cutting issues
Market Facilitation
Market Development






Lack of Baseline Survey from partners
Market Development program; requiring in-depth
understanding of the benefits to program
beneficiaries
To supplement training needs assessment tool and
client satisfaction survey
To validate sustainable adoption of technologies
with the view of accelerating demand. Are clients
buying into the technology as appropriate, aim of
the project
To check anticipated project impacts/ monitor
unintended impacts
Cost
3. Changes in Business
Income
4. Knowledge on
Technology, Available
Technologies and
Suppliers
5. Household Livelihood
Improvements
6. Changes in Risk
Aversion, Technology
Adoption and Technology
Appreciation
Annually
3.Collecting SC stories
2. Changes in Production
and Productivity
2. Defining the Reporting Period
1. Defining Domains of Change
1. Changes in Business
Practice and Spirit of
Entrepreneurship
1. The stories are
collected by the M & E
practitioner of the project.
2. The practitioner meets a
group of six farmers
(maximum) at a time
Then the M and E
practitioner develops a
newspaper of these
articles and circulates
them to the project
management at different
levels within the
organization
6. Verification
5. Feedback on the choices made
4. Selection of collected SC stories
1. The panel of stakeholder
used is the farmers
themselves.
To verify the stories the
project opted to conduct
the MSC approach in
groups of farmers in the
same locality


Stories were compiled and at the MEDA office the
number of stories on a particular domain of
change were weighed with those considered MSC.
In both scenarios increase in income was highest
placed impact.
The application of MSC enabled the MEDA Zambia
team to establish that the major impact of the
intervention i.e getting the SSIT’s out to the farmers
was an increase in gross income for farmers.
 The
validation of sustainable adoption
for most farmers pointed to benefits in
terms of finance accumulated versus
livelihood indicators that project
monitoring activities were focused on
mostly e.g health, education, housing

In project planning terms meaning that the main
information that other farmers should be furnished
with to accelerate demand should be the
potential for them to increase their business
income.
It is not a stand alone technique;
 Hit the nail on the head-get to the issue directly
 Its never too late too use it
 Gives a richer description of the context in which a
positive change is occurring, beyond the data from the
quantitative case studies
 Quality is added to the data collected as the stories are
collected in the beneficiaries voice(verbatim)
 It is more flexible as it is not based on predefined
indicators, you can measure unpredicted outcomes





MSC can be adapted to measure specific qualitative aspects of
a multi faceted project
Not all MSC domains will vary from indicators but can still
give a closer look at outcomes
During story telling other demand side players learnt from
their peers on best methods to employ the adopted
technology
The stories can be furnished with quantitative data for depth
If the MSC is conducted in a target group with participants in
the same areas they serve as a verification board to measure
the truth in a story.



1.
2.

Market development indicators are often a challenge to monitor due
to the complexity of the market and the qualitative nature they take.
MSC as an evaluation approach can be applied to capture specific
qualitative aspects of change for a multi faceted intervention.
For MEDA its application enabled the project validate sustainable
adoption of SITT’s as it related to income increase.
There is proof farmers have not only bought the SITT, but have
spotted benefits of it and realized its need (accelerating demand)
It was not complicated to apply and a larger number of farmers
were reached to gather this specific vital information
MEDA is sure what message should be fed to farmers to accelerate
demand as the supply chain is being developed even if it was not
what was prioritised initially.