“Capacity Development in Social Welfare”
END OF PROJECT EVALUATION
REPORT
Submitted to:
United Nations Development Programme (UNDP)
From:
Development Art Consultancy
Box 132
Lilongwe
Malawi
Billy Mayaya
Emmanuel Liwimbi
August 2011
Capacity Development in Social Welfare in Malawi- End of Project Evaluation Report
Page 1
Acknowledgements
The Capacity Development in Social Welfare End of Project Evaluation was
conducted with the financial assistance from UNDP Malawi and the guidance of
Mr. Peter Kulemeka, UNDP Evaluation Focal Point and Ms. Faith Kachimera,
Programme Analyst. The UNDP Assistant Resident Representative responsible for
Capacity Development, Mr. Ernest Misomali, gave useful insights of the project
while the Programme Manager at the Ministry of Gender, Children and
Community Development Mr. Harry Satumba provided key information on project
implementation. Valuable project design information was obtained from all UN
Volunteers both at the Ministry as well as those placed in all the districts. Many
individuals contributed ideas into this report and their contributions are
acknowledged.
Capacity Development in Social Welfare in Malawi- End of Project Evaluation Report
Page 2
Table of Contents
List of acronyms and abbreviations.......................................................................................... 4
Executive Summary ................................................................................................................. 6
INTRODUCTION .................................................................................................................... 8
1.1. Background Information ...............................................................................................8
1.2. Purpose and Objectives of End of Project Evaluation ..................................................9
1.3 Evaluation Methodology ..............................................................................................10
1.3.1 Data Sources ..........................................................................................................10
1.3.2. Ethical considerations...........................................................................................11
1.3.3 Limitations.............................................................................................................11
2. EVALUATION FINDINGS AND CONCLUSIONS ........................................................ 11
2.0 General Assessment .....................................................................................................11
2.1 Project Assessment based on Outputs ..........................................................................12
2.1.1 Project Output 1: strengthened capacity to implement, coordinate, monitor and
evaluate HIV and AIDS response at national and sub-national levels. ............................12
2.1.2. Project Output 2: establish a national OVC electronic management information
system. ................................................................................................................................16
2.1.3 Project Output 3: establish OVC database at the Ministry linked to all the
districts in Malawi……………………………………………………………………….17
2.2 Project Assessment based on the four UNDP Assessment criteria…………………..19
2.2.1 Relevance ..............................................................................................................19
2.2.2 Effectiveness .........................................................................................................20
2.2.3 Efficiency ..............................................................................................................22
2.2.4 Impact ....................................................................................................................23
2.2.5 Sustainability .........................................................................................................24
3. LESSONS LEARNED ...................................................................................................... 25
4. CONCLUSION .................................................................................................................. 25
5. RECOMMENDATIONS………………………. .............................................................. 26
6. ANNEXES ......................................................................................................................... 29
Annex 1: List of people consulted......................................................................................29
Annex 2: Evaluation questions ...........................................................................................31
Annex 3: Key Informant Interview Guide .........................................................................32
List of documentation reviewed .........................................................................................34
Capacity Development in Social Welfare in Malawi- End of Project Evaluation Report
Page 3
List of acronyms and abbreviations
AIDS
Acquired Immunodeficiency Syndrome
BPF
Best Practices Forum
CD
Capacity Development
CEYCA
Centre for Youth and Children’s Affairs
CHH
Child Headed Households
CONGOMA Council for Non Governmental Organisations
CRC
Convention on the Rights of the Child
CRS
Catholic Relief Services
CSO
Civil Society Organisation
CU
Concern Universal
CVSU
Community Victim Support Unit
DPA
District Plan of Action
DSWO
District Social Welfare Officer
EYC
Eye of the Child
FBO
Faith Based Organisation
FGD
Focus Group Discussions
FHI
Family Health International
GoM
Government of Malawi
HIV
Human Immunodeficiency
M&E
Monitoring and Evaluation
MDGs
Millennium Development Goals
MGCCD
Ministry of Gender, Children and Community Development
MGDS
Malawi Growth and Development Strategy
MIS
Management Information System
MIS
Management Information Systems
MoU
Memorandum of Understanding
MP
Member of Parliament
NAC
National AIDS Commission
NGO
Non Governmental Organisation
NICE
National Initiative for Civic Education
Capacity Development in Social Welfare in Malawi- End of Project Evaluation Report
Page 4
NPA
National Plan of Action
NSO
National Statistical Office
OVC
Orphaned, Vulnerable Children
PJM
Periodic Joint Meetings
SWO
social Welfare Office
SWOT
Strengths, Weaknesses, Opportunities, Threats
ToT
Training of Trainers
TWG
Technical Working Group
UNDP
United Nations Development Programme
UNICEF
United Nations Children’s Fund
UNV
United Nations Volunteer
VDC
Village Development Committee
VSU
Victim Support Unit
Capacity Development in Social Welfare in Malawi- End of Project Evaluation Report
Page 5
Capacity Development in Social Welfare in Malawi
End of Project Evaluation Report
Executive Summary
The specific objective of the project was to strengthen the capacity of the Ministry of
Gender, Children and Community Development as well as District Social Welfare
Offices (DSWO) to successfully implement the National Plan of Action and scale up
the national response for Orphans and Vulnerable Children. The outcome for the
project was to have improved national capacity to formulate policy, manage, monitor and
deliver services to protect the rights of vulnerable.
This report is an end of term evaluation of a three year project (2008-10) “Capacity
Development in Social Welfare in Malawi”. Among others, it highlights major
findings, challenges, lessons learnt and recommendations according to the five
assessment criteria of relevance, effectiveness, efficiency, impact and sustainability.
The evaluation report established that the project was relevant as its objectives were
within the mandate of the Ministry and in line with the countries overarching
national development policy, the Malawi Growth and Development Strategy
(MGDS). Nevertheless,
the project had inherent shortcomings arising from its
design. The project design had unclear and inconsistent outputs, it did not show
output indicators and had no baseline values for the indicators. This posed a
challenge in project monitoring and evaluation.
The findings of the report indicate that the project was effective as its project
outputs were to a larger extent achieved.
Following capacity development
initiatives such as skills improvement, targeted districts were able to submit timely
monitoring and evaluation and activities reports, better coordinate Orphans and
Vulnerable Children (OVC) activities and write proposal for resources mobilization
for the implementation of OVC-related activities in line with their Districts Action
Capacity Development in Social Welfare in Malawi- End of Project Evaluation Report
Page 6
Plans. At national level, the OVC data base was improved by making it more user
friendly and putting it on line to link to all Districts.
However, the project’s effectiveness was challenged by poor efficiency of the project
as there were delays in placement of United Nations Volunteers (UNVs) and
disbursement of funds which adversely affected the timeliness in which the project
objectives could be achieved.
The evaluation report noted that the project has the potential of being sustainable in
that its objectives are among the priorities of the Ministry. There is also a clear
institutional mandate and framework for implementation of social welfare activities
targeting vulnerable individuals and communities. However, the sustainability of
the project faces threats from inadequate financial resources and human resources
such that the project is likely be unsustainable if the external support ceases.
The report advances several recommendations that should inform the formulation
and implementation of future similar projects. In order to improve on the
effectiveness, efficiency, impact and sustainability of future similar project, the
report recommends that the project design should be clearly articulated in terms of
having a well-articulated programme theory, clear outcomes, outputs and
indicators. The role and functionality of the project’s board need to be clarified and
emphasised.
Notwithstanding the fact that the project sought to strengthen capacity at the
Ministry and District Social Welfare Offices, it is the considered view of the
consultants that the project should have also focus on addressing the HIV and AIDS
issues which is one of the major causes of increase in OVC and capacitating the
OVC themselves through formal and technical education bursaries and the most
affected communities through development of skills in Income Generating
Activities. This would have greater impact and more sustainable than is currently
the case.
Capacity Development in Social Welfare in Malawi- End of Project Evaluation Report
Page 7
INTRODUCTION
1.1. Background Information
1. UNDP in partnership with the Ministry of Gender, Children and Community
Development (MGCCD) and the District Social Welfare Offices (DSWO), has
implemented a three-year (October 2008 to September 2010) project with a total
budget of US $ 863, 556.12 entitled “Capacity Development in Social Welfare in
Malawi”.
2. The project was being implemented at the Ministry of Gender, Children and
Community Development and in the twelve Districts of the country namely:
Salima, Nkhota-kota, Dowa, Nkhata-Bay, Dedza, Ntcheu, Balaka, Zomba,
Mangochi, Nsanje, Chikhwawa and Chiladzulu.
3. The specific objective of the project was to strengthen the capacity of the Ministry
of Gender, Children and Community Development as well as District Social
Welfare Offices (DSWO) to successfully implement the National Plan of Action
and scale up the national response for Orphans and Vulnerable Children.
4. The outcome for the project was to have improved national capacity to formulate
policy, manage, monitor and deliver services to protect the rights of vulnerable.
5. This outcome was to be achieved by placing international United Nations
Volunteers (UNVs) as Technical Assistants/ counterparts in District Social
Welfare Offices and as Information Technology Specialists under the Ministry.
The District Social Welfare Offices
was a means of scaling up the OVC
programme response. The District Social Welfare Officer was meant to be the
counterpart to the UNV and the UNV was to ensure that the transfer of skills and
other areas of related capacity development is taking place within the district
through the provision of technical assistance.
Capacity Development in Social Welfare in Malawi- End of Project Evaluation Report
Page 8
6. The expected results (outputs) of the project included the following: (a)
strengthened capacity to implement, coordinate, monitor and evaluate HIV and AIDS
response at national and sub-national levels; (b) establish OVC database at the Ministry
linked to all the districts in Malawi; and (c) establish a national OVC electronic
management information system.1
7.
To achieve these results, the main strategies used were training workshops,
consultative meetings and field demonstrations.
8.
The direct beneficiaries of the project were expected to be local communities,
particularly orphans and vulnerable children, the physically challenged, elderly
caregivers and People Living With HIV and AIDS (PLWHA), the Ministry of
Gender, Children and Community Development and local government
structures.
Indirect benefits were expected to accrue to other government
departments, Non Governmental Organisations and UN agencies in the form of
capacity development achieved through training programmes.
9. Having the project come to an end, this report therefore, is an End of Project
Evaluation.
1.2. Purpose and Objectives of End of Project Evaluation
10. Basically, the purpose of this evaluation was to assess the achievements
towards
the
project’s
objectives
and
results
using
five
(5)
key
indicators/criteria, namely relevance, efficiency, effectiveness, impact and
sustainability and in agreement with the scope of work as specified in the
Terms of Reference (ToRs).
11. The specific objectives of this End of Project Evaluation were the following:
i.
Assess achievement towards the project’s objectives and results,
ii.
Assess the efficiency and cost-effectiveness of implementation,
iii.
Identify strengths and weaknesses in project design and implementation,
and,
1
Project Document “
Capacity Development in Social Welfare in Malawi- End of Project Evaluation Report
Page 9
iv.
Provide recommendations on design modifications and specific actions
that would increase the effectiveness and impact of future similar
initiatives.
12. The scope of the evaluation covered UNDP, the Ministry of Gender, Children
and Community Development, the DSWOs and the communities that benefited
from the project.
13. The findings and recommendations of this End of Project Evaluation will
inform future OVC programme/projects development.
1.3 Evaluation Methodology
14. The evaluation attempted to be as participatory as possible. Apart from the
document review, which provided the background and logic of the project,
there were consultative meetings with key stakeholders.
Besides, key
informants interviews and focus group discussions were conducted with
stakeholders knowledgeable about the project to solicit views on the relevance,
effectiveness, efficiency, sustainability and impact of the project.
1.3.1 Data Sources
15. The evaluation made use of both primary and secondary data sources. A
questionnaire was developed to solicit data and information from the following
stakeholders:
UNDP,
Ministry
of Gender, Children and Community
Development, National AIDS Commission, UN Volunteers, District Social
Welfare Offices and selected community leaders. An interview guide and a list
of interviewees is presented in the Annexes.
16. Secondary data was obtained from a review of documents in which detailed
information on project design, project achievements, success factors and
challenges during project implementation, among other things, were
documented. A list of documents reviewed is given in the Annexes.
17. Qualitative and quantitative data was collected mainly using a structured
questionnaire from the respondents and was analysed. The responses from the
questionnaires were verified using available secondary data from the document
Capacity Development in Social Welfare in Malawi- End of Project Evaluation Report
Page 10
review or in some cases verification was made based on field data, although
this was limited. In this way the accuracy of the responses was confirmed.
18. Drafts for this report were presented to UNDP review meetings at UNDP
offices and to Stakeholders meeting at the Ministry of Gender, Community and
Child Development. The evaluators circulated a full draft report to UNDP
programme office and the officials from the Ministry of Gender, Community
and Child Development officials who were involved in the project. Their
comments and suggestions inform this report.
1.3.2. Ethical considerations
19. The Evaluation was conducted in accordance with the principles outlined in
the UNEG ‘Ethical Guidelines for Evaluation’. The Evaluation team took every
measure to safeguard the rights and confidentiality of key informants in the
collection of data. The independence of the evaluation process and products
was safe guarded and upheld by making sure those evaluation findings, and
results are not tempered with or influenced in any way by those implementing
and managing the project.
1.3.3 Limitations
20. The documentation of the project was not readily available which negatively
affected the smooth assessment of the project. For example the consultants were
unable to access:
a) VSO to UNV transitional report
b) Financial Status Report
2. EVALUATION FINDINGS AND CONCLUSIONS
2.0 General Assessment
21. The general findings on the project outputs was that the project document did
not have output indicators hence made it difficult to assess the performance of
the project based on the outputs. Nevertheless, the evaluation used the
indicators contained in the delivery reports from the Ministry of Gender, Child
and Community Development.
Capacity Development in Social Welfare in Malawi- End of Project Evaluation Report
Page 11
22. The evaluation has also noted the absence of the baseline indicators as there
was no evidence that the project was implemented based on a baseline survey.
This made it difficult to effectively track the performance of the project.
23. The consultants are aware that not all projects start with a baseline survey as
some may use the indicators of the baseline surveys of the other previous
similar projects. Based on this understanding, the consultants were informed
that the project was based on the previous project that used the Volunteer
Service Oversees (VSOs).2 However, no documentation was made available to
the consultants regarding the baseline indicators of the previous project.
2.1 Project Assessment based on Outputs
2.1.1 Project Output 1: strengthened capacity to implement, coordinate, monitor and
evaluate HIV and AIDS response at national and sub-national levels.
24. Based on the delivery reports on output 1, the evaluation observed that Six
UNVs were attached to the Twelve selected District Social Welfare Offices
(DSWOs) and each UNV was attached to at least two DSWOs. This achieved
the project target that by the end of the project the selected Districts should
have one UNV working alongside the District Social Welfare Officers as
counters parts.
25. Three UNV Information Technology Specialists were to be recruited by the end
of the project. This indicator was achieved, whereby two out of the planned
three UNVs IT specialists were recruited at the Ministry level.
26. On the indicator of strengthened capacity to implement HIV and Aids responses at
national and sub- national level under output 1, the project had proposal writing
and resource mobilization as indicators. By the end of the project it was
expected that the targeted districts would be able to: (a) come up with District
Action Plans for OVCs, and (b) to write proposals for funding of activities in
the District Action Plans for OVCs.
2
Contribution by Mr. Willard Manjolo (Social Welfare Director) Ministry of Gender, at Stakeholders meeting
on 9th August, 2011
Capacity Development in Social Welfare in Malawi- End of Project Evaluation Report
Page 12
27. After the placement of the UNVs all the targeted districts managed to develop
their District Action Plans for OVCs. The aim of such plans was to enable
districts to mobilize resources for OVCs through proposal writing to other
partners as the Government funding was limited.
28. There is evidence that capacity for the districts has been strengthened in as far
as proposal writing is concerned. All district managed to develop proposals
and submitted to different partners. While
some districts have managed to get funding
be a success story as its proposal managed to
Ntcheu District was singled as the best
practice. Its proposal was funded under
the project name “ the Journey of Life.”
This project is a psycho- social support
initiative which involves community
mobilisation and awareness on issues
that affect children through training of
psycho-social support implementers.
get MK 20 Million funding from UNICEF
(Interview with the UNV for Ntcheu)
through their proposals, others are still
awaiting for response from partners.
29. Ntcheu District for example has stand out to
under the psycho-social support initiative.3
30. Out of the funding for Ntcheu District, the Ministry of Gender has also
benefited a tune of MK 7.2 Million which is being used to develop a psychosocial support guide manual for the OVC program in the country.4
31. Zomba District has trained the CBOs in the district in proposal writing to
enable them to source funding for OVCs activities on their own in an effort to
strengthen capacity at the community level.
32. Salima and Dowa Districts sourced funding from UNICEF for case management
system project whose funding was given to the Ministry of Gender Head
Quarters since the Ministry was implementing a similar project. This therefore
lead to the transfer of the UNV for Salima and Dowa to the Ministry
Headquarters to manage the project.5
33. Although it was commendable that such funding was sourced, in their
considered view the evaluators noted that the new role given to the
3
Interview with the UNV for Ntcheu and also confirmation by Mr Harry Satumba, Project Manager, Ministry
of Gender , Children and Community Development on 9 th August 2011
4
Interview with Mr Harry Satumba, Project Manager, Ministry of Gender , Children and Community
Development on 9th August 2011
5
Ibid
Capacity Development in Social Welfare in Malawi- End of Project Evaluation Report
Page 13
Salima/Dowa UNV at the Ministry Headquarters was outside the scope of the
project. In addition, her removal from the District created a gap in such
Districts. Such practices if encouraged do greatly affect negatively project
implementation.
34. Nkhotakota and Nkhata-bay Districts have also submitted their proposals to
UNICEF and they were awaiting for funding.6
35. The project was also intended to strengthen capacity to coordinate HIV and Aids
responses at national and sub- national level under output 1 through: (a)
establishment of Child Protection Committees at the District and Community
levels; (b) conduct coordination quarterly meetings for OVC stakeholders at
the District level.
36. As part of coordination under activity (a), the project planned to establish 12
Child Protection Committees at the District level (1 for each District) and 120
Child Protection Committees at the community level ( 10 for each District). By
the end of the project a total of 12 District Child Protection Committees were
established and only 90 committees were established at the Community level.
37.
Table 1: Summary of previous Status of Child Protection Committees:
District Child Protection Committee
Community
Child
Protection Committee
Districts
Nkhota kota
Had one but not active
None
Nkhata-bay
None
None
Salima
Had one which was working with the No clear evidence as
Family Health International (FHI) and FHI was in –charge.
not the District
Dowa
Had one not active, only its sub- None
committee on bursaries was active.
Dedza
Has one but meeting irregularly.
Ntcheu
Had one which was working with the No clear evidence as
None
NGO network but no coordination the NGO network was
6
Interview with the UNV for Nkhotakota and Nkhata-bay
Capacity Development in Social Welfare in Malawi- End of Project Evaluation Report
Page 14
from the District Office
Zomba
Had
many
committees
poorly
i.e
in
in-charge.
coordinated None
OVC,
ECD,
Education and labor
Machinga
Had one which was active but poorly None
coordinated
Chiradzulu
Had one which was active with No clear evidence as
support
from
Oxfam
but
no the NGO network was
coordination from the District Office
in-charge.
Nsanje
None
None
Chikhwawa
None
None
SOURCE: INTERVIEW WITH MR. SATUMBA, PROJECT MANAGER
38. The table for the previous status of children protection committee and the
current status of the same, shows that the project has made a commendable
achievement in that manner.
39. While a 100% achievement was attained at the District level, the project
managed to achieve 75% establishment of Child Protection Committee at the
community level. The reason for not attaining 100% achievement at community
level being inadequate funds as the requirement was that after every
establishment of a committee, there was supposed to be a training of committee
members, which required funds.
40. Quarterly coordination meetings were conducted in all Districts with partners
on OVC issues. This was evidenced from the Districts activity reports that were
quarterly submitted to Ministry Headquarters.
41. The project was also intended to strengthen capacity to monitor HIV and Aids
responses at national and sub- national level under output 1 through: (a)
submission of quarterly monitoring and evaluation reports; and (b) submission
of quarterly activity reports.
42. The evaluation established that both quarterly monitoring and evaluation
reports as well as quarterly activity reports were submitted by all Districts to
the Ministry after the placement of the UNVs in the Districts. This was an
Capacity Development in Social Welfare in Malawi- End of Project Evaluation Report
Page 15
achievement as previously the targeted Districts were not submitting such
reports. However, due to internet problems, some Districts such as Nsanje
failed to submit their reports in time. Nevertheless, this did not reflect one the
District’s capacity to report as it
was largely a technical problem.
43. Sometimes the Districts failed to
report in time due to the delays
in collecting data by the Child
Protection Workers from their
Zomba District represented a success story as
it managed to go round this problem by
training and using three Child Protection
Officers in the OVC data collection and entry
exercise. Through this approach, the project
managed to strengthen capacity beyond the
District to as far as community level.
communities for input to the
Districts. Such a delay was as a result of lack of capacity in data collection as
well as limited numbers of the Child Protection Workers (CPW) at the
Community level. The CPWs were yet to be trained by the OVC District Desk
Officers.
2.1.2. Project Output 2: establish a national OVC electronic management information
system.
44. Though this was one of the project outputs, it was observed during the
interviews that the Ministry had already in place an information management
system, however, it was offline.7
45. It was mentioned that the Ministry started registration of the OVC between the
period of 2006-2007 following the introduction of a country wide community
based OVC registration system.8 During that period most districts had
completed the manual registration of the OVCs.
46. In 2008, the Ministry engaged a consultant to design a database on which to
enter the data of OVCs which was collected manually between the period 20062007. The consultant completed the assignment.
7
8
Interview with Mr. Moyo, Ministry of Gender Official on 19 th August, 2011
Programme Document, Capacity Development in Social Welfare, page 20
Capacity Development in Social Welfare in Malawi- End of Project Evaluation Report
Page 16
47. However, the challenge was that, the database was offline, not user friendly
and most officials in the Ministry were not conversant with it as they were not
trained on how to use it.9
48. Given the scenario above, the consultants are of the view that project output 2
was wrongly framed, since prior to the project the Ministry had already an
information management system in which data was collected, could be
processed, analyzed, derived reports and could be disseminated although it
was not user friendly and offline.
2.1.3 Project Output 3: establish OVC database at the Ministry linked to all the districts
in Malawi.
49. Under this output the consultants established that, what the UNV IT Specialists
did was to improve the data base that was already there in the Ministry and not
to establish the OVC data base as reflected in project output 3.
50. Project output 3 gives the impression that there was no OVC data base in place
at the Ministry and the District which was not the case as there is evidence that
prior to the project in 2008 a data base was designed by a Consultant for the
Ministry.10
51. The UNV IT Specialists improved the data base by programming it with
specific indicators to make it user friendly, linked it to all Districts by putting it
on line.
52. The UNV IT Specialists also conducted trainings for Ministry officials on how
to use the data base, trained DSWOs and OVC Desk Officers on data collection
and computerization.
53. The OVC Desk Officers after being trained were expected to transfer the skills
to the Child Protection Workers who in turn were to train communities on how
to update data. There is no evidence that this has been achieved and the time
frame was not specified. The has resulted in the delay of data collection and
data computerization in most of the targeted districts. Out of the twelve
9
Programme Document, Capacity Development in Social Welfare, page 20
Interview with Mr. Moyo, Ministry of Gender Official, on 19th August, 2011
10
Capacity Development in Social Welfare in Malawi- End of Project Evaluation Report
Page 17
targeted districts only Dedza, Ntcheu and Zomba have completed the collection
and computerization of OVC data. While Chikhwawa district has made good
progress among the remaining districts.11
54. As part of strengthening capacity in data collection and computerization, the
UNV IT Specialist developed a manual for data collection and computerization.
This has been commendable initiative by the UNV IT Specialists to speed-up
the data collection and computerization process although some districts were
still dragging behind due among other factors such as large volumes of data to
be computerized and a few number of data entry clerks that were recruited
using UNDP funds.
55. One of the strategy to achieve the project objectives was use the database for
mapping out service required, offered, frequency of support and were to be
utilized as a decision making tool by all stakeholders. The database were also to
assist in establishing criteria for identifying OVC and enable stakeholders to
establish priority target groups for service provision. It were also to provide
information to donors for support areas.12
56. The potential for achieving the above mentioned strategy is evident. The most
significant achievement is that the data base is now on line. Furthermore, there
are reports and minutes reflecting that information on the improved OVC data
base has been disseminated to OVC stakeholders through quarterly OVC
Technical Group meetings and Annual OVC stakeholders meetings. There also
efforts of linking the data base to the Government Wide Area Network
(GWAN) and the National Statistical Office website as a way improving the
access to the data base.13
11
Interview with Mr. Moyo and Mr Benjamin Kawala (M&E Officer), Ministry of Gender Official on 19 th
August, 2011
12
Programme Document, Capacity Development in Social Welfare, page 7
13
Interview with Mr. Moyo and Mr Benjamin Kawala (M&E Officer), Ministry of Gender Official on 19 th
August, 2011
Capacity Development in Social Welfare in Malawi- End of Project Evaluation Report
Page 18
2.2 Project Assessment based on the four UNDP Assessment criteria
57. The evaluation of the project was also based on the four UNDP Assessment
criteria that were specified in the Terms of Reference (TOR) for the consultants.
These criteria were: Relevance, Effectiveness, Efficiency, Impact and Sustainability of
the project.
2.2.1 Relevance
58. The project was relevant as it dealt with enhancement of capacity in “Social
Welfare” which is within the mandate of the Ministry Gender, Child and
Community Development.
59. The project was also relevant in that it supported the aims of the Malawi
Growth and Development Strategy (MGDS), specifically, the Social Protection
Pillar (Protection of the Vulnerable).
60. Furthermore, the project was relevant in that it supported the outcomes of the
social welfare framework as outlined in the National Plan of Action for OVC
which were to do with the increasing the scaling up of the OVC response.
61. Based on these findings the consultants rated the project to be relevant to the
overarching national development of the country (MGDS), the mandate of the
Ministry of Gender and the National Plan of Action for OVC.
62. Despite being relevant, the evaluation noted that the project had inherent
problems in its design as follows:
a) The project design did not include base line indicators on which it was
based.
b) There was lack of consistency in outputs. For example on page 2 of the
project document the expected output was stated as “Strengthened
capacity to implement, co-ordinate, monitor, and evaluate HIV and AIDS
response at national and sub-national levels” while on page 8 of the same
document it was presented as “strengthened capacity of the Ministry of
Women and Child Development and District Social Welfare Offices to
successfully implement the NPA and scale up the national response to OVCs”.
This created confusion as it was not clear as to whether the project
expected output was on HIV and AIDS response or on OVC response.
Capacity Development in Social Welfare in Malawi- End of Project Evaluation Report
Page 19
c) Furthermore, the project document indicated only one project output,
while the terms of references for the consultancy indicated three outputs.
This raises questions as to what were really the outputs of this project.
d) As reflected in the project document, the project design did not include
the log frame. The log frame is critical as it clarifies the conceptual
understanding of the project and guides the implementation, monitoring
and evaluation of the project.
2.2.2 Effectiveness
63. This relates to the extent to which results contributed to achievement of the
project objectives.
64. One of the project objective was to strengthen the capacity of the Ministry of
Gender and the District Social Welfare Offices to successfully implement the
National Plan of Action for OVC.
65. Responsibility for the project achieving its designated outputs was ultimately
held by the Project Board. The Project Board was to be responsible for making
management decisions relating to the initiation, direction, review and closure of
this project. The Project Board was also to make key decisions at decision points
during the running of the project, or as necessary when critical issues were
raised by the National Project Manager. Notwithstanding the critical role of the
Project Board, there is evidence that the Project Board did not perform to its
expectation. While it was expected for the Project Board to be meeting quarterly
and review reports produced by the project manager, to the contrary it was
mentioned by the Project Manager that, throughout the project period, the
Project Board only met once. This obviously is an indication of inherent poor
coordination in terms of managing the project. Specifically it was observed that
while the project document specified the members of the board, it did not
clearly stipulate the roles of the individual members of the board. For example,
the document did not state who will be the board chair, who will conduct
secretariat functions and as such no one was ultimately responsible for the
performance of the board.
Capacity Development in Social Welfare in Malawi- End of Project Evaluation Report
Page 20
66. The targeted Districts were identified for the project due to their inability to
write proposals for funding, write and timely submit reports for their activities
and coordinate OVC activities.
67. As a way of strengthening capacity in the targeted Districts, the project
managed to place UNVs in all the targeted Districts who were partnered with
the DSWOs. The aim of this arrangement was to ensure transfer of skills from
the UNVs to the local officers. There is evidence that this was achieved:
a) All Districts managed to develop their District Action plans.
b) Based on the District Action Plans, all Districts were able to write
proposals and some even managed to get funding through their
proposal, for example Ntcheu, Dowa and Salima. While the rest were
still awaiting for response from the donors.
c) All Districts were able to timely submit OVC monitoring and evaluation
report and activity reports to the Ministry.
d) All Districts managed to establish District Child protection Committees
representing 100% achievement. However, there was an achievement of
75% establishment of Child Protection Committee at community level.
This was attributed to inadequate funds.
e) All Districts managed to conduct quarterly coordination meetings with
partners.
68. While the evaluation is of the view that the project was to a larger extent
effective in strengthening capacity it noted the following short comings:
a) Against the norm that capacity building involves both the
development of skills and numbers, it was noted that the project
focused much on skills development than numbers. This was a critical
anomaly given that at the time the project was running there was a
60% vacancy rate in the Ministry that was already negatively
impacting on the implementation of OVC activities.
b) There was less focus on capacity building at the community level
which negatively affected activities at the district level. For example,
there was a delay in data collection and computerisation at the District
Capacity Development in Social Welfare in Malawi- End of Project Evaluation Report
Page 21
level due to lack of capacity of Child Protection Workers involved in
the exercise at the community level.
c) Transfer of skills from UNVs to the local officers was sometimes
challenged by high differences in knowledge levels between these two
partners. For example, it was noted that the UNV IT Specialist was a
masters’ degree holder while the counterparts were diploma holders.
This caused a challenge for local counterparts to comprehend the
technical knowledge that was being passed on by the UNV IT
Specialist.
2.2.3 Efficiency
69. This relates to the timeliness with which the project objectives were achieved.
70. The evaluation noted that there were delays in the recruitment and placement
of UNVs which affected the efficiency of the project. The delays in recruitment
were due to the fact that some UNVs returned home soon after reporting for
work. For example during the initial stage of the project 3 UNVs IT Specialists
were hired. 2 UNVs out the 3 UNVs returned home barely after a month and a
year respectively. After this initial recruitment 2 more UNVs were hired as
replacements and only 1 reported for work. This delayed the implementation of
output 2.
71. Notwithstanding the above, it took six months for the UNVs to be deployed to
various Districts. This again was a drawback in the timely implementation of
the project.
72. Nearly all the implementing partners raised concerns with delays in UNDP funding
procedures. It was indicated that there was an initial funding gap of about six months
at the start of the programme. As a result, this brought about complications in terms of
reporting and accountability.
73. While the UNDP played a role in the delay of funding, the Ministry of Gender
as the implementing partner also shared responsibility for the delay. The
consultants established that, the Ministry of Gender was supposed to open an
account for the smooth transfer of the project funds from UNDP, this was not
yet done by the time of the evaluation. The absence of the project account
Capacity Development in Social Welfare in Malawi- End of Project Evaluation Report
Page 22
resulted in UNDP transferring money into personal accounts a thing which
raises questions in terms of efficiency and accountability.
74. Despite that the project was coming to an end, it was observed that some of the
Districts had not yet finalised computerisation of data due to among other
reasons the huge volume of data to be entered.
75. On the overall, due to the above problems, the efficiency of the project was
highly compromised.
2.2.4 Impact
76. This relates to the difference that the project has brought about on the targeted
beneficiaries.
77. There are indicators that the project has made some impact in terms of
strengthening capacity especially at the District level. The Districts are now able
to do some of the things which they were not able to before the project. For
example, they are able to write proposals for funding, write and submit reports
in time and coordinate OVC activities among others.
78. However, it cannot be 100% ascertained that the above elements of impact are
due to the occurrence of skill transfer from UNVs to the local officers. The
reason being that by the time of evaluation, the UNVs were still in the Districts
and still supporting the local officers. The achievement of skills transfer can
best be ascertained after the UNVs have completely left the local officers on
their own.
79. The evaluation came to this position based on the observation that there was a
rapid change on the targeted Districts; that is immediately after the placement
of the UNVs, the Districts were able to do that which they were not able to do
before. This raises the questions as to whether the change was due to the skills
transfer or mere placement of UNVs.
80. On the part of the data base, the project has made great improvement by
making it user friendly and putting it online. However, the evaluators did not
find evidence that stakeholders are using it to implement the OVC activities.
Nevertheless, based on the improvements made to the data base the consultants
Capacity Development in Social Welfare in Malawi- End of Project Evaluation Report
Page 23
are of the view that it has potential to make a positive impact in the
implementation of OVC activities if fully exploited.
81. It therefore calls for the elapse of a length of time in order to ascertain the real
impact of the project in terms of both skills transfer and operationalisation of
the data base.
2.2.5 Sustainability
82. Determines whether the positive outcomes of the project are likely to continue
after external support comes to an end.
83. The project was about strengthening capacity for the social welfare function in
the Ministry of Gender. Issues of social welfare are within the mandate of the
Ministry of Gender. For the Ministry to effectively and efficiently carry out its
functions it needs to strengthen the capacity of its staff among other things. It is
therefore in the interest of the Ministry to continue building capacity of its staff
to execute its functions. Based on this understanding there is high potential that
the Ministry will continue to pursue the objectives of the project even if after
the external support ceases.
84. However, there are threats to sustainability in terms of:
a) Finance – The Ministry operates on inadequate resources since Government
budget allocations are based on the ceilings given by the Treasury which are
determined by availability of resources and not need from respective
Ministries. As such the Ministry may be constrained to sustain the objective
of the project.
b) Human resource capacity – The Ministry has a high vacancy rate of 60%
which in itself is constraint on its capacity to pursue the objectives of the
project. Furthermore, due to issues of staff turn-over through transfers,
resignations and deaths a cross Government Ministries, there is no guarantee
that those who have been trained through this project will remain in the
Ministry and continue to support the objectives of the project.
85. Since the data base has been improved to the level of being user friendly and is
online and also that the members of staff have been trained on how to use it,
the Ministry will be able to exploit it.
Capacity Development in Social Welfare in Malawi- End of Project Evaluation Report
Page 24
86. However, there are equally threats to the sustainability of the data base arising
from such factors as inadequate resources to invest in the maintenance of the
data base as well as staff turn-over of technical personnel to operate the data
base.
87. Furthermore, it was observed that it would be challenging for the Ministry to
re-program the data base when such a need arises, since such an exercise need
high level expertise equal to that of the UNV IT Specialist, which the Ministry
does not currently have.
88. Opportunities: there is evidence of interest
from other partners to buy in the concept of
the project.
89. Despite the achievements of the project, the
consultants are of the view that, unless the
threats are minimised and the available
GTZ supported development of
OVC website Zomba District.
UNICEF funded proposals for
Salima and Tcheu. While
Nkhata-bay and Nkhotakota
were still awaiting for funding
from the same UNICEF
(Interview with Mr. Satumba)
opportunities are exploited, there is no
sustainability of the project.
3. LESSONS LEARNED
90. From the project evaluation a number of lessons have been learnt, which can
inform the design and implementation of future similar projects.
91. Capacity building is a long term process which requires long term projects if
real and sustainable impact is to be achieved.
92. Success of the project depends on the efficiency and effectiveness of the
placement of human resources and disbursement of financial resources. Thus is
vital since a project is implemented within a defined time frame.
93. Collaboration and good coordination of all stakeholders can expedite project
outcomes in a cost effective manner.
4. CONCLUSION
94. The consultants are of the view that the project to a larger extent managed to
achieve the project objectives despite the challenges that were experienced from
Capacity Development in Social Welfare in Malawi- End of Project Evaluation Report
Page 25
the design to implementation as already discussed in the report. If the
challenges are tackled based on the recommendations below, the consultants
are of the view the project can be more efficient, more effective, more
sustainable and can make a big impact. Given the potential that the project has,
it is justified that such a project should be extended.
5. RECOMMENDATIONS
95. Due to inherent shortcomings in the project design, the report recommends that
future projects should come up with project designs that are comprehensive,
thorough and that pay closer attention to the best practices for project designs.
96. Specifically the future projects in their project designs should include, baseline
indicators and in case that the project is using base line indicators from other
previous projects reference should be made to such indicators.
97. There should also be consistency in the wording of the outputs through-out the
project document. This gives a clear guidance on what the project intends to
achieve and a basis for assessing the performance of the project.
98. Designs of future projects should consider to include the log frame, specifying
the project goals, purpose, outputs, activities, objectives, means of verification
and assumptions. This would enhance the conceptual understanding of the
project and help in monitoring and evaluation.
99. It would also be helpful if the periodic reviews of the project should also
include the review of the log frame to check on the relevance of the
assumptions and risks.
100. In future similar projects, there should be clear stipulation of the roles of
individual project board members for which they could be responsible for the
overall performance of the board.
101. OVC issues are not confined to the capacity building of the Ministry of Gender,
Child and Community Development and District Social Welfare Offices only, as
such future similar projects should consider focusing on the OVCs themselves.
For example, efforts could be made towards empowering OVCs through
education bursaries (both for formal and technical education) so that they
Capacity Development in Social Welfare in Malawi- End of Project Evaluation Report
Page 26
eventually graduate from their OVC status. Project focus could also be
extended to the communities by capacitating them in income Generating
Activities so that they are made self-reliant in caring for OVCs.
102. Similarly, the future similar projects should recognize the cause and effect
relationship between OVC issues and HIV/AIDS. It is therefore recommended
that future similar projects should concurrently focus on tackling HIV/AIDSthe dominant cause of OVC crisis in the country in areas of prevention,
behavioral change, care and support.
103. Further, the inclusion of participatory assessment of the log frame by a cross
section of beneficiaries under all outputs is paramount to triangulating the
impact of the project.
104. It is recommended that a forum should be established where success stories
from good performing Districts could be shared with other Districts that are
lagging behind. This would help in replicating best practices among the
Districts that are implementing same projects.
105. In case of extension, the next phase of the project should also emphasize more
on the capacity building of the targeted communities for efficient and effective
implementation of project activities. For example training of Child Protection
Workers to speed up OVC data collection.
106. Since any project has a defined time frame within which to be implemented, it
is recommended that issues of recruitment of human resources and
disbursement of financial resources should be done efficiently and effectively.
This will avoid delays and ensure achievement of project objectives. For
example future projects should adopt a strategy of having a reserve list of
UNVs for immediate replacement of those who may quit the project along the
way. Again, bureaucratic procedures should be ironed out prior to the start of
the project to ensure smooth disbursement of funds.
107. Another strategy could be to consider the recruitment of local UNVs to cut on
the delays and cost in the recruitment process and exploit the locally available
skills, knowledge and experience.
Capacity Development in Social Welfare in Malawi- End of Project Evaluation Report
Page 27
108. Given that there is high vacancy rate in the Ministry, it is recommended that
future similar projects should explore on how to beef up the number of staff in
the Ministry to ensure that capacity is strengthened as capacity building
involves both skills development and beefing up the numbers of staff. The
UNDP has done this before through the UNDP Joint Support Program in which
it has recruited non-established staff to strengthen national M&E system in the
country and could adopt the same approach for the OVC programmes.
109. It is recommended that there should not be unnecessary departure from the
origin scope of the project in the course of implementation. A case in point is
the transfer of the UNV for Salima/Dowa to the Ministry Headquarters to run
the project which she was not recruited for.
110. For the sustainability of the data base, it is recommended that the Ministry
should make deliberate effort to either upgrade the current IT staff or recruit IT
personnel with high technical expertise like up to First Degree or Masters level.
111. In addition, for the sustainability of the achievements of the project in general,
it is recommended that the Ministry should strive to make more resources
available for capacity building and maintenance of the data base. Specifically,
the Ministry should prioritise allocation of resources to OVC activities or lobby
Parliament through relevant Committees such Social and Community Affairs
Parliamentary Committee which is the Ministry’s line Committee and Budget
and Finance parliamentary Committee which is responsible for scrutinising
National Budget allocation for increased funding of OVC activities. The
Ministry can also seek more resources by linking up with other development
partners who are interested in HIV and AIDS and OVC issues.
112. It is recommended that future project should has proper records management
system that could ensure availability of project documentation to ease the
monitoring and evaluation of the project.
Capacity Development in Social Welfare in Malawi- End of Project Evaluation Report
Page 28
6. ANNEXES
Annex 1: List of people consulted
Day
Place
10 November 2010
UNDP
Respondent
Faith Kachimera
Position
Programme
Analyst Capacity
Development in
Social Welfare
Phillimon
Programme
Tambala
Analyst Capacity
Development
Health
Ministry of
Harry Satumba
Gender, Children
Programme
Manager
and Community
Development
3 December 2010
Ministry of
Immaculate
UNV Social
Gender, Children
Guthera
Welfare
Leonard Greene
UNV IT Specialist
Claire
UNV Social
and Community
Development
Welfare
7 December 2010
Salima District
Nixon Chawinga
Council
District Social
Welfare Officer Salima
7 December 2010
Nkhotakota
Harrison Thindwa
District Social
Capacity Development in Social Welfare in Malawi- End of Project Evaluation Report
Page 29
District Council
Welfare Officer –
Nkhotakota
Ellen Paulo
UNV Social
Welfare
8 December 2010
Ntcheu District
Peter Mulinzi
Council
Zomba District
Welfare
Naftal Nyan’gwa
Council
9th December 2010
Chikwawa District
UNDP
UNV Social
Welfare
Nana Bafour
Council
12th January 2011
UNV Social
UNV Social
Welfare
Ernest Misomali
Assistant Resident
Representative,
Capacity
Development
Peter Kulemeka
Trust Fund
Manager
14 January 2011
Ministry of
Hyacintha
Director of
Gender, Children
Kulemeka
Children’s Affairs
Simon Chisale
Assistant Director
and Community
Development
14 January 2011
National AIDS
Commission
Capacity Development in Social Welfare in Malawi- End of Project Evaluation Report
Page 30
Dr Mwai
Policy Officer -
Mwakoka
Biomedical
Chaudley Msosa
Grants Officer
(Programmes)
19 August 2011
Ministry of
Mr Moyo
Child Affairs Officer
Mr. Benjamin
Economist/M&E
Coordinator
for
HIV/AIDS programs
Gender, Children
and Community
Development
Kayala
Annex 2: Evaluation questions
In pursuit of the overall objectives, the following key questions were addressed
during the End of Project Evaluation:
1. To what extent was the project relevant and the delivery mechanism relevant
in addressing the identified needs and problems?
2. To what extent did the project achieve its overall objectives?
3. To what extent were the results (impacts, outcomes and outputs) achieved?
4. Describe
the
management
arrangements
and
processes
and
their
appropriateness in supporting delivery.
Capacity Development in Social Welfare in Malawi- End of Project Evaluation Report
Page 31
5. Was the project design appropriate and did it respond to the needs?
6. To what extent did the project’s M&E mechanism contribute in meeting
project results?
7. What is the likelihood of continuation and sustainability of project outcomes
and benefits after completion of the project?
8. Describe key factors that will require attention in order to improve prospects
for sustainability of project outcomes and the potential for replication of the
approach.
9. Describe the main lessons that have emerged; and
10. Provide a set of recommendations for future cooperation between UNDP
and the Ministry of Gender, Children and Community Development,
including project design and arrangements.
Annex 3: Key Informant Interview Guide
Development Art Consultancy has been commissioned by UNDP Malawi to
conduct End of Project Evaluation of the Capacity Development in Social Welfare
Project. You are one of the stakeholders knowledgeable about the project. We are
requesting you to respond to the questions below to assist us come up with a fair
assessment of the project performance.
Your responses will only be used for
purposes of this evaluation. We thank you in advance for your cooperation.
Relevance
1.
What are the objectives of the project?
2.
Who are the beneficiaries of the project?
Capacity Development in Social Welfare in Malawi- End of Project Evaluation Report
Page 32
3.
Does project design address the identified issues?
4.
Do the project objectives meet the needs of the beneficiaries?
5.
Were the project beneficiaries properly selected?
6.
Did the project in its design and implementation take into account national
development priorities?
7.
Were the expected outcomes realistic given the time frame?
8.
Were the inputs and strategies identified realistic, appropriate and adequate
to achieve the results?
Effectiveness
1.
Did the project effectively meet the needs of the beneficiaries?
2.
What is the extent of achievements made with regard to the goals, purpose
and expected results?
3.
Were the planned outcomes achieved? Were they achieved within the
planned time frame?
4.
To what extent were planned outputs attained?
5.
What were the results in terms of promoting gender equity and equality
(planned/unplanned)?
Efficiency
1. Did the costs of inputs (financial, human and other resources) compare
favourably to outputs?
2. Was the implementation of activities cost-effective and are the results
justifiable in terms of the costs and amount of work?
3. Could more have been achieved with the same resources (financial, human
and other resources)?
4. Were time and resources allocated to the different project components and
activities in a planned, balanced, coordinated and justified manner?
Capacity Development in Social Welfare in Malawi- End of Project Evaluation Report
Page 33
5. Looking beyond the delivery process, did the program components achieve
or likely to achieve its outputs?
6. Did they contribute to achieving project outcomes and impact?
Impact
1. Did the interventions reach the intended target groups?
2. Were the beneficiaries sufficiently informed about the modalities of the
project?
3. Did the project coordinate and create complementarities with other
government, private sector or NGO programmes?
Sustainability
1. Did the project have a viable exit strategy?
2. Will beneficiaries continue to benefit from the project interventions and
activities now that the planned project period is over?
3. Do the beneficiaries have sufficient capacity and resources to undertake the
provision of services after the project is phased out?
4. What particular factors or events affected the project results? Were these
factors internal or external to the project?
List of documentation reviewed
District Orphans and Other Vulnerable Children (OVC) Strategic Plan, Chiradzulu
District Council
District Orphans and Other Vulnerable Children (OVC) Strategic Plan, Nkhotakota
District Council District Orphans and Other Vulnerable Children (OVC) Strategic
Plan, Zomba District Council
Government of Malawi, Project Document Capacity Development in Social Welfare
in Malawi 1 October 2008 – 30 September 2010
Monitoring the Situation of Children and Women in Malawi, National Statistical
Office and UNICEF
Capacity Development in Social Welfare in Malawi- End of Project Evaluation Report
Page 34
National Plan of Action for OVC
National Strategic Plan for Early Childhood Development 2009 -2014
OVC fact sheet
OVC IEC materials
Quarterly, annual reports and work plans for the duration of the project
Stakeholders’ Meeting Report, Zomba 10th December 2009
Capacity Development in Social Welfare in Malawi- End of Project Evaluation Report
Page 35