WIPO - IAOD
EXTERNAL QUALITY ASSESSMENT OF THE EVALUATION FUNCTION
FINAL REPORT
Ian C Davies, CE, Consultant
June 29, 2014
Page 1 of 45
Executive Summary
This report presents the findings, analysis, conclusions and recommendations
of the external quality assessment of the evaluation function of the Internal
Audit and Oversight Division (IAOD) of the World Intellectual Property
Organization (WIPO).
Purpose
The primary purpose of the external assessment is to provide assurance on
IAOD evaluation conformity with the United Nations Evaluation Group (UNEG)
Norms and Standards. The secondary purpose is to provide
recommendations to improve the effectiveness of the IAOD evaluation
function including the integration of Human Rights and Gender Equality in
evaluations. Questions relative to evaluation resources and quality assurance
are also addressed.
Conformity with UNEG norms and standards
The review concludes that overall, IAOD’s evaluation function is generally
consistent with UNEG norms and standards. The review concludes that there
are specific areas that partially conform and require the attention of IAOD in
order to strengthen their consistency with UNEG norms and standards and to
contribute to the performance of the evaluation function. Although taken
individually or collectively the assessed gaps in conformity do not threaten the
fundamental integrity of the evaluation function, they are sufficiently important
that they warrant action in each of the three assessed areas of independence,
credibility and utility. Most importantly:
 Publish,
disseminate and communicate publicly all evaluation reports1.
 Address
bottlenecks in the conduct of evaluations that threaten their
timeliness.
 Develop
quality standards specific to the IAOD evaluation function and
put in place an explicit quality control process to manage for quality each
of the processes of programming, planning, implementation,
communication and follow up.
 Address
the evaluation function’s ability to fulfil its learning mandate and
contribute to organisational knowledge.
1
It should be noted that the Proposed Revisions to the WIPO Internal Oversight Charter (May 22,
2014) states: The Director, IOD shall publish internal audit and evaluation reports on the WIPO
website within 30 days of their issuance. In exceptional cases, if required to protect security, safety
or privacy, the Director, IOD may, at his/her discretion, redact or withhold a report in its entirety.
Page 2 of 45
Human rights and gender equality (HRGE)
To the question “to what extent are principles and implicit standards related to
Human Rights and Gender Equality (HRGE) integrated in evaluations and
what revisions to the Evaluation Policy and Strategy should be considered to
further that integration?” the review concludes that principles and implicit
standards related to Human Rights and Gender Equality (HRGE) are not yet
fully integrated into IAOD evaluations.
The review recommends at the operational level:
 Maintain
regular professional development sessions for IAOD evaluation
staff on gender responsive evaluation.
 As
part of to-be-developed quality control standards for evaluation in
IAOD, quality control and quality assurance processes, operationalise
and integrate HRGE.
 Access
specialised HRGE resources so that IAOD evaluations provide
assessments and recommendations to the stakeholders, programmes,
organisations and networks on HRGE levers for effectiveness and
sustainability.
At the organisational level:
 Articulate
a specific policy component the integration of HRGE in all
IAOD evaluations
 Develop
annual operational plans for the integration of HRGE integration
in evaluation.
Effectiveness
To the questions “to what extent does the Evaluation Function contribute to
WIPO accountability and learning and to what extent are evaluation results
incorporated and used in follow-up activities, how they are monitored and
what is their actual impact?” the review concludes that:
 IAOD
evaluation contributes to WIPO accountability however could
contribute more to learning.
 Although
there is a system in place for monitoring and following up on
evaluation recommendations, there is as yet little in the way of
observable or reported impact.
 The
observed application of a predominant audit frame of reference to
oversight may constrain the effectiveness of the IAOD Evaluation
Function and may limit its ability to exploit fully opportunities for adding
value to WIPO and its stakeholders.
Page 3 of 45
The review recommends that IAOD evaluation should:
 Consider
improving the quality and nature of its engagement with WIPO
management .
 Systematically
include evaluability assessments at the programming
stage of work planning for evaluations. .
 Focus
more systematically on intended users as part of programming and
planning processes.
 Identify
and address systematically the bottlenecks in the evaluation
process.
 Consider
diversifying its portfolio of evaluations to conduct short cycle
evaluations.
 Revisit
the purpose and use of reference groups for evaluation.
Resources
The question that was put to the review is: To what extent are the resources
available to the Evaluation Function adequate for it to achieve its mandate?
The review concludes that in terms of quality IAOD evaluation staff resources
are adequate however the quantity of qualified evaluation staff may not be
sufficient.
The review recommends IAOD monitor its level of allocation of qualified
evaluation staff to its evaluation function and the feasibility of its intended
workplan, based on a systematic and data based analysis of estimated
workload.
Quality assurance
The question that was put to the review is: How adequate is the quality
assurance system for the Evaluation Function and in particular the periodicity
of reviews?
The review concludes that there is not a sufficiently developed quality
assurance system for the Evaluation Function of IAOD.
The review recommends that IAOD should:
 Give
priority to developing and instituting a quality control process for
evaluation based on explicit quality standard.
 Develop
and institute a quality assurance process using professionally
qualified evaluation experts external to IAOD.
Page 4 of 45
TABLE OF CONTENTS
EXECUTIVE SUMMARY
2
1.
INTRODUCTION
7
2.
DATA COLLECTION
7
3.
ASSURANCE ON CONFORMITY
8
3.1 Findings
3.1.1
Independence
3.1.2
Credibility
3.1.3
Utility
8
9
12
16
3.2
Conclusion
19
3.3
Recommendations
19
4.
HUMAN RIGHTS AND GENDER EQUALITY
20
4.1
Findings
20
4.2
Analysis
21
4.3
Conclusion
21
4.4
Recommendations
22
5.
ASSESSMENT OF EFFECTIVENESS
23
5.1 Findings
5.1.1
The view from WIPO management
5.1.2
Understanding of evaluation
5.1.3
Programming
5.1.4
Evaluation process
23
23
24
25
25
5.2 Analysis
5.2.1
Evaluation and audit
5.2.2
Organisation
5.2.3
Operations
26
26
27
27
5.3
Conclusion
29
5.4
Recommendations
30
6.
ADEQUACY OF RESOURCES
32
6.1
Findings
32
6.2
Conclusion
32
Page 5 of 45
6.3
Recommendation
32
7.
ADEQUACY OF QUALITY ASSURANCE PRACTICES
32
7.1
Findings
33
7.2
Conclusion
33
7.3
Recommendations
33
8.
WIPO-IAOD EVALUATION FUNCTION SURVEY
34
8.1
Relevance and use of evaluations
34
8.2
Evaluation process
34
8.3
Communication of evaluation findings
35
8.4
Quality of evaluations
35
8.5
Additional comments
35
8.6
Observations
35
9.
ADDITIONAL REMARKS
36
10.
SUMMARY OF RATINGS
37
Page 6 of 45
1. Introduction
This report presents the findings, analysis, conclusions and recommendations
of the external quality assessment of the evaluation function of the Internal
Audit and Oversight Division (IAOD) of the World Intellectual Property
Organization (WIPO).
The assessment was conducted by the external professional consultant Ian C
Davies, Credentialed Evaluator (CE) over the period from April to June 2014.
The assessment is first and foremost an assurance engagement to assess
the extent of the Evaluation Function’s conformity with the United Nations
Evaluation Group (UNEG) Norms and Standards for evaluation in the UN
system.
The secondary purpose of the assessment is to provide recommendations to
IAOD on improving guidance for, and effectiveness of, the evaluation function
including the integration of Human Rights and Gender Equality in evaluations.
Questions relative to evaluation resources and quality assurance are also
addressed.
2. Data collection
The assessment was conducted based on data collected through systematic
document review, semi-structured interviews, meetings with IAOD staff and
a web-based survey questionnaire administered to WIPO management
respondents.
The document review included all key guidance documents pertaining to the
evaluation function in WIPO, full documentation on IAOD, comprehensive
documentation of the evaluation function of IAOD including, but not limited
to, working papers, evaluation reports, relevant job descriptions and staff
CVs, comprehensive complement of evaluation documents from
programming through to planning, conduct, reporting, follow up as well as
documentation pertaining to external consultants.
The review comprised a one week intensive onsite visit to WIPO
headquarters for meetings, interviews, file reviews and the detailed
examination of working papers and web based systems.
A descriptive summary of the results of the web based survey is included at
the end of this assessment report.
Page 7 of 45
3. Assurance on conformity
This chapter looks at the extent to which the organisation, systems and
processes of IAOD and its Evaluation Section are consistent, i.e. in
conformity, with the norms and standards of the United Nations Evaluation
Group (UNEG).
Conformity with norms and standards is an ad minima proposition, i.e. it
means that the organisation is set up, has procedures and behaves in a
manner consistent with what are considered to be minimum criteria.
Examining and assessing conformity allows for flagging areas that may be
below standard, i.e. do not conform, and that require immediate attention and
correction as they are essential to the functioning of the organisation, as well
as areas that conform partially and call for including or strengthening aspects
that may be already in place or planned for. It is however key to understand
that conformity with norms and standards does not imply nor does it equate
with good performance. Conformity is a necessary but not sufficient condition
for good performance the question of which is dealt with in chapter 5.
3.1 Findings
The assessment undertook to address the following question:
To what extent do the evaluation services and products of the Evaluation
Function, in particular the conduct of evaluation processes and the quality of
evaluations, conform to UNEG Norms and Standards for Evaluation?
The review drew key findings from the data that were collected and assessed
these against the full set of UNEG norms and standards with a focus on
UNEG’s normative2 framework for Independence, Credibility and Utility of
evaluation function.3 The normative framework is the basis on which UNEG
Peer Reviews are conducted of UN organisations with a normative mandate
as is the case with WIPO.
The UNEG Annual General Meeting, 2012, defined normative work as:
“The support to the development of norms and standards in conventions, declarations,
regulatory frameworks, agreements, guidelines, codes of practice and other standard setting
instruments, at global, regional and national level. Normative work also includes the support to
the implementation of these instruments at the policy level, i.e. their integration into legislation,
policies and development plans, and to their implementation at the programme level. (UNEG,
2012, p. 5)”
3 Please see http://www.uneval.org/papersandpubs/documentdetail.jsp?doc_id=1484
2
Page 8 of 45
3.1.1
Independence
Independence is a function of statutory and organisational dispositions, as
well as explicit and adequate support to behavioural independence.
Accordingly the findings on independence are based to a large extent on the
examination of WIPO’s key guidance and organisational documents for
IAOD’s evaluation function, of working papers and reports for evaluations, of
IAOD evaluation staff CVs and position descriptions as well as interviews with
IAOD staff.
The following presents key findings for independence against relevant UNEG
norms:
N2.3: The governing bodies/heads of organizations are responsible for ensuring
that adequate resources are allocated to enable the evaluation function to
operate effectively and with due independence.
Responsibility with respect to the allocation of resources to IAOD, including
the Evaluation Function, rests with the Director General of WIPO (the head of
the organisation). The Program and Budget of WIPO is presented to the
General Assembly (the governance level) for approval.4
Status: Conforms
N.5.1: Impartiality is the absence of bias in due process, methodological rigor,
consideration and presentation of achievements and challenges. The requirement
for impartiality exists at all points in the process: planning, conduct, reporting
The requirement for impartiality is found in the IAOD’s guidance and
operational documents and is assessed as consistent with generally accepted
practice in the UN system and more generally with evaluation principles such
as those found in OECD-DAC guidance.
In addition, the review’s in depth examination of working documents of IAOD’s
Evaluation Function does not show evidence of bias or perception thereof.
Status: Conforms
It is important to distinguish the level at which lies responsibility for the allocation of resources,
which is what the norm refers to, from the question of adequacy of resources of the Evaluation
Function to fulfil the mandate given to it, which is addressed in 3.4
4
Page 9 of 45
N6.1: The evaluation function has to be located independently from the other
management functions so that it is free from undue influence and that unbiased
and transparent reporting is ensured. It needs to have full discretion in
submitting directly its reports for consideration at the appropriate level of
decision-making pertaining to the subject of evaluation.
The evaluation function is located in IAOD and, as such, is independent from
management functions of WIPO. The independence of IAOD, and therefore of
its Evaluation Section, is enshrined in the WIPO Internal Oversight Charter,
which states in section D, article 4, : “The Director, IAOD, enjoys functional
and operational independence from the Secretariat and Management in the
conduct of his/her duties”. It also states that “The Director IAOD is
responsible to the Director General (…)” and further states that “ He/she has
the authority to initiate, carry out and report on any action, which he/she
considers necessary to fulfil his/her mandate.”
Based on the authority and prerogatives conferred to the Director of IAOD by
the Charter as well as a systematic review of evaluations completed since
2012 and their transmission, the assessment considers that the norm is
respected.
Status: Conforms
N6.2: The head of evaluation must have the independence to supervise and
report on evaluations.
Although there is a position of Head of Evaluation Section, it is part of IAOD
for which designated responsibility for oversight, including evaluation, rests
with the Director of IAOD. The review finds that the independence to
supervise evaluations is well defined in guidance documentation and effected
based on operational documentation and interviews with staff.
The WIPO evaluation policy states: “The IAOD Evaluation Section will ensure
that all evaluation reports are disclosed to the public at the completion of the
evaluation process and disseminated widely through the print and electronic
media in accordance with WIPO’s disclosure policy”.5
The review found that evaluation reports are not always systematically made
available to the public and in cases the practice appears to be to provide them
to Member States on request through “Brainloop”6, to make them available on
WIPO’s intranet in some cases and to publish public summaries in some
instances.
This finding indicates potential inconsistencies with the Revised WIPO
Evaluation Policy which states that : “All evaluation reports are posted on the
WIPO website and are accessible to the public.”7
IAOD Revised Evaluation Policy – May 4, 2010 – article 59
A secure password protected cloud based storage facility
http://www.brainloop.com/en/products/brainloop-secure-dataroom.html
7 Revised Evaluation Policy (May 4, 2010) paragraph 23. Protection from outside interference.
5
6
Page 10 of 45
The review did not examine the extent to which WIPO’s disclosure policy had
a bearing on the above finding nor is this issue clearly addressed from a
public information perspective. Because public perception of independence,
and transparency, are important to the credibility of the evaluation function, its
products and more broadly WIPO’s activities, the relationship, if any, between
WIPO’s disclosure policy and public reporting of evaluations should be clearly
addressed. More importantly public reporting should be consistent with WIPO
policy and the intent expressed in the proposed revised Charter.8
Status: Partially conforms
N6.3: To avoid conflict of interest and undue pressure, evaluators need to be
independent, implying that members of an evaluation team must not have been
directly responsible for the policy-setting, design, or overall management of the
subject of evaluation, nor expect to be in the near future.
From a review of the documentation made available to it, including CVs of
staff of the evaluation function, as well as interviews with said, the norm as
stated above is met.
Status: Conforms
N6.4: Evaluators must have no vested interest and have the full freedom to
conduct impartially their evaluative work, without potential negative effects on
their career development. They must be able to express their opinion in a free
manner.
The review finds that the WIPO Internal Oversight Charter9, Evaluation Policy
as well the quality control processes in place in IAOD at each stage of the
evaluation and related processes ensure that the evaluators do not have
vested interests and are able to conduct their activities impartially. This finding
extends to the external consultants that are retained for IAOD evaluations.
Status: Conforms
It should be noted that the Proposed Revisions to the WIPO Internal Oversight Charter (May 22,
2014) states: The Director, IOD shall publish internal audit and evaluation reports on the WIPO
website within 30 days of their issuance. In exceptional cases, if required to protect security, safety
or privacy, the Director, IOD may, at his/her discretion, redact or withhold a report in its entirety.
9 WIPO Internal Oversight Charter
8
Page 11 of 45
3.1.2
Credibility
Credibility is to a large extent based on the perception that stakeholders have
of evaluation. While there are factors that affect perception and fall outside the
reasonable control of the evaluation function, good practice as reflected in the
following norms shows that there are key aspects of the way in which
evaluation is conducted that affect positively and constructively credibility.
Credibility is important because, among other things, it increases the
likelihood of evaluation use.
The following presents key findings for credibility against relevant UNEG
norms:
N4.2: The evaluation plan can be the result of a cyclical or purposive selection of
evaluation topics. The purpose, nature and scope of evaluation must be clear to
evaluators and stakeholders. The plan for conducting each evaluation must
ensure due process to ascertain the timely completion of the mandate, and
consideration of the most cost-effective way to obtain and analyze the necessary
information.
The review finds that evaluation planning at the organisational level is both
cyclical and purposive based on the WIPO Evaluation Section: 2010-2015
Strategic Plan. Generally, from a planning perspective, the strategic plan is
consistent with good practices in evaluation and coherent with the Charter
and the Evaluation Policy.
At the level of specific evaluations, and on the basis of document review and
interviews, the review finds that, generally, their purpose, nature and scope is
well understood by both the evaluators of IAOD and by stakeholders who
have participated in them. The review was not able to ascertain with a
sufficient degree of confidence the extent to which the purpose, nature and
scope of specific evaluations was clear to externally contracted consultants,
however with responsibility for the evaluation retained by IAOD in all cases
risks are appropriately mitigated.
The systematic review10 of three final evaluation reports11 based on the
UNEG Quality Checklist for Evaluation Reports (2010)12 shows weaknesses
in section 3. Evaluation Purpose, Objective (s) and Scope with an average
rating across all three reports of 1.513 (on a four point rating scale where 1=
poor and 4=excellent).
As well, based on the examination of working papers for evaluations and
interviews, the review finds that the processes for reaching agreement on
terms of reference, inception and on reporting with primary stakeholders may
put at risk the timely completion of the mandate.
The review of evaluation reports was effected separately by two independent reviewers.
Country Portfolio Evaluation Thailand. Program 1 (Patent Law). Improvement of National, Subregional and Regional IP Institutional and User Capacity.
12 http://www.uneval.org/papersandpubs/documentdetail.jsp?doc_id=607
13 A maximum 3 was given only once for the Program 1 report.
10
11
Page 12 of 45
Although generally the factors contributing to the credibility of evaluation
identified in this norm are addressed by the IAOD evaluation function there
are areas requiring strengthening for the evaluation reports and the timeliness
of completion.
Status: Partially conforms
N5.2: Impartiality increases the credibility of evaluation and reduces the bias in
the data gathering, analysis, findings, conclusions and recommendations.
Impartiality provides legitimacy to evaluation and reduces the potential for
conflict of interest.
Building on the findings in N5.1 and based on evaluation reports and working
papers, the review examined the methodology used in each of the three
evaluations the reports of which were assessed, with a focus on data
collection methods and sources, findings, conclusions and recommendations.
The review found no evidence of bias.
In addition, data provided through interviews with stakeholders involved in
IAOD evaluations as well through a web based survey questionnaire14,
provide no indication of bias. To the survey question “To what extent are
evaluations conducted transparently, independently and impartially?”,
respondent ratings averaged at 3.73 (on a five point rating scale where 1= not
at all and 5=fully)
Provisions for avoiding conflict of interest exist for external consultants
involved with evaluations.
Status: Conforms
N8.1: Each evaluation should employ design, planning and implementation
processes that are inherently quality oriented
Based on its examination on working documents related to evaluations,
interviews with evaluation staff as well with stakeholders involved in
evaluations, the review finds that there is systematic effort by the IAOD
Evaluation Section to carry out key steps in the evaluation process in a
manner that is consistent with general principles of good practice.
Reference is made to UNEG Norms and Standards however the review did
not find an explicit quality control process in place to manage for quality each
of the processes of design, planning and implementation. The review notes
that IAOD uses the UNEG Checklist for reports when conducting its peer
review quality checks.
Status: Partially conforms
The survey consisted of 10 questions with 5 point Likert scales and one open ended question.
Invitations were sent to 34 recipients with 14 respondents (including partial collections) for a
response rate of 41%.
14
Page 13 of 45
N9: The head of the evaluation function, evaluation staff and evaluators should
have proven competencies to manage and/or conduct evaluations 15
The review examined the job descriptions, curriculum vitae, of IAOD
evaluation function staff and finds that, given the European context and the
corresponding stage of development of professional evaluation and related
training, they have collectively and experientially proven competencies to
manage and conduct evaluations.
Status: Conforms
N10.1: Transparency and consultation with the major stakeholders are essential
features in all stages of the evaluation process. This improves the credibility and
quality of the evaluation. It can facilitate consensus building and ownership of the
findings, conclusions and recommendations.
Based on interviews with WIPO stakeholders who had participated in
evaluations, the results of the survey16, the examination of working papers
and the assessment of evaluation reports, the review finds that, generally, key
stakeholders are consulted as part of the evaluation.
Stakeholder participation could be strengthened through more systematic
involvement in the programming phase of evaluation. Stakeholder analysis
should be undertaken systematically for all evaluations. Evaluation reports
should improve their reporting on describing stakeholder consultation
processes as well as the rationale for these. Full access to all evaluation
reports would further increase transparency.
Status: Conforms
The formulation of Norm 9 – Competencies for Evaluation is a synthesis of its components 91,
9.2, 9.3
16 From the survey summary: The extent to which evaluation processes engage with the survey
respondents and their stakeholders in ways that make evaluations useful received an average
rating of 3.15 with the extremes of the rating scale only being used by one respondent each.
15
Page 14 of 45
N11: Evaluators must have personal and professional integrity, respect people
and their rights, confidentiality of data and sources, and be sensitive to beliefs as
well as issues of gender and human rights.17
From its examination of working papers and interviews with WIPO
stakeholders and IAOD staff the review considers the professional deportment
of the IAOD Evaluation Function staff to be consistent with the above.
Status: Conforms
N8.2: Evaluation reports must present in a complete and balanced way the
evidence, findings, conclusions and recommendations. They must be brief and to
the point and easy to understand. They must explain the methodology followed,
highlight the methodological limitations of the evaluation, key concerns and
evidenced-based findings, dissident views and consequent conclusions,
recommendations and lessons. They must have an executive summary that
encapsulates the essence of the information contained in the report, and facilitate
dissemination and distillation of lessons.
Notwithstanding that some areas of the evaluation reports18 that the review
examined in detail offer opportunities for quality improvements, the review
finds that, generally, reports adhere to the principle of fair and balanced
reporting. 19
In addition, the survey found that on balance, respondents indicated that the
findings were communicated in a constructive manner.
Status: Conforms
N10.2: documentation on evaluations in easily consultable and readable form
should also contribute to both transparency and legitimacy
Please see observations under N6.2
Status: Partially conforms
The formulation of Norm 11 – Evaluation Ethics is a synthesis of its components 11.1, 11.2,
11.3, 11.4, 11.5
18 Country Portfolio Evaluation Thailand. Program 1 (Patent Law). Improvement of National, Subregional and Regional IP Institutional and User Capacity.
19 Item 6.4 - UNEG Quality Checklist for Evaluation Reports (2010)
17
Page 15 of 45
3.1.3
Utility
Utility recognises the fundamental purpose of evaluation as a discipline that
contributes to learning and progress through the provision of reliable and
meaningful information that can be used. In other words, evaluation that is not
useful and used does not have value. In this sense evaluation is different from
social science research which does not necessarily have to be used to be
considered valuable. As such the concept of utility is central to sound
evaluation.
The following presents key findings for utility against relevant UNEG norms:
13.1 Evaluation contributes to knowledge building and organizational
improvement. Evaluations should be conducted and evaluation findings and
recommendations presented in a manner that is easily understood by target
audiences.
The review finds that, overall, in guidance documents for the Evaluation
Function the learning aspect of evaluation, i.e. as a component of knowledge
building, is put forward. In particular the WIPO Evaluation Policy highlights the
role of the IAOD Evaluation Section in contributing to the Organisation’s
learning20.
However, based on the examination of the selected evaluation reports 21, the
feedback provided by WIPO stakeholders who had participated in IAOD
evaluations and to a lesser extent the responses to the survey, the review
finds that in practice the contribution of evaluation to knowledge building and
organizational improvement is still weak and of little consequence.
From an evaluation output perspective the review finds that the “lessons
learned” tend either to be identified “shortcomings” to which solutions are
proposed. These solutions are too specific to the evaluand and its context to
be considered lessons learned.
Or they are re-statements of well know principles of good practice, e.g. “The
key lesson here is that the Project design and management style/approach
have an important effect on the outcomes of Projects. It is a matter that
should therefore be taken seriously.”
The review further finds that in most cases recommendations tend to be
general and insufficiently specific to be actionable.
Part 3 section 4 paragraph (p.12):
i)
Promoting and supporting best practice in developmental evaluation and
developing appropriate and user-friendly mechanisms for the collection, publication
and dissemination of lessons learned.
ii)
Creating for a with internal and external stakeholders to share and discuss findings
and feeding into future decision-making
iii)
Developing and managing a public WIPO web site dealing with evaluation.
21 Country Portfolio Evaluation Thailand. Program 1 (Patent Law). Improvement of National, Subregional and Regional IP Institutional and User Capacity.
20
Page 16 of 45
Interviews with WIPO stakeholders highlight concerns with the lack of
substantive contribution of evaluation to the “business of IP” together with a
lack of understanding of IP and insufficient knowledge of the administrative,
programmatic and management issues of the organisation. As a result, and
although there is generally a positive appreciation of the work of the IAOD
evaluation function, there is little perception, understanding and recognition of
its contribution to organisational improvement and knowledge.
Status: Partially conforms
N2.6: The Governing Bodies and/or Heads of organizations and of the evaluation
functions are responsible for ensuring that evaluation contributes to decision
making and management. They should ensure that a system is in place for
explicit planning for evaluation and for systematic consideration of the findings,
conclusions and recommendations contained in evaluations. They should ensure
appropriate follow-up measures including an action plan, or equivalent
appropriate tools, with clear accountability for the implementation of the
approved recommendations.
WIPO instituted a web based recommendation management system,
TeamCentral, in March 2013, migrating from an Excel open recommendation
spreadsheet. The review finds that there is a systematic process whereby the
Director of IAOD provides to the Director General of WIPO regular updates on
the status of the Oversight Recommendations. The updates are simple
statistical descriptions of the status of implementation of recommendations
according to variables such as programmes and importance. The update
encompasses all oversight recommendations. The updates do not present
analysis.
The review did not examine the extent to which specific recommendations
from evaluations contribute to decision-making and management. Based on
data collected through interviews with WIPO stakeholders and through the
survey, the review finds that the recommendations and their follow up tend to
be perceived by management more as accountability and compliance checks
rather than substantive inputs to management decision making.
Based on summaries of ratings from Client Satisfaction Questionnaires
provided to it by IAOD, the review notes that, to the statement “The lessons
learned from this audit/evaluation have a positive impact on your
management processes and will help you achieve the objectives of your
program/unit/project” the average rating was 4 “I agree” (for 7 evaluations on
a rating scale from 5 – Totally agree, to 1 – Totally disagree)
Status: Partially conforms
Page 17 of 45
N4.1 Proper application of the evaluation function implies that there is a clear
intent to use evaluation findings. In the context of limited resources, the planning
and selection of evaluation work has to be carefully done. Evaluations must be
chosen and undertaken in a timely manner so that they can and do inform
decision-making with relevant and timely information. Planning for evaluation
must be an explicit part of planning and budgeting of the evaluation function
and/or the organization as a whole. Annual or multi-year evaluation work
programmes should be made public.
Building on the findings presented in N2.3, the 2010-2015 Evaluation Strategy
as well as consultations by the Director of IAOD with senior management of
WIPO as part of the evaluation programming process, the review finds that
there is clear intent to make the evaluations relevant and useful to
management. As well care is taken in the programming of evaluations to
make clear its dependence on allocation and levels of resources available to
the evaluation function. Planning for evaluation is an explicit part of IAOD’s
programme and budget planning which itself feeds into WIPO’s overall
program and budget on a cyclical basis. The 2010-2015 Evaluation Strategy
is publicly accessible on WIPO’s website.
Status: Conforms
N4.2 The evaluation plan can be the result of a cyclical or purposive selection of
evaluation topics. The purpose, nature and scope of evaluation must be clear to
evaluators and stakeholders. The plan for conducting each evaluation must
ensure due process to ascertain the timely completion of the mandate, and
consideration of the most cost-effective way to obtain and analyze the necessary
information.
Please see N4.1
Status: Conforms
N2.7: governing bodies and/or heads of organizations and of the evaluation
function are responsible for ensuring that there is a repository of evaluation and a
mechanism for stilling and disseminating lessons to improve organizational
learning and systemic improvement
The review finds that there is an archive for evaluations. Please see N2.6 and
N13.1.
Status: Partially conforms
Page 18 of 45
3.2 Conclusion
The review concludes that overall, IAOD’s evaluation function is generally
consistent with UNEG norms and standards.
The review concludes that there are specific areas that partially conform and
require the attention of IAOD in order to strengthen their consistency with
UNEG norms and standards and to contribute to the performance of the
evaluation function.
Although taken individually or collectively the assessed gaps in conformity do
not threaten the fundamental integrity of the evaluation function, they are
sufficiently important that they warrant action in each of the three assessed
areas of independence, credibility and utility.
3.3 Recommendations
IAOD should take action required to:
 Publish,
disseminate and communicate publicly all evaluation reports22.
 Address
bottlenecks in the conduct of evaluations that threaten their
timeliness.
 Develop
quality standards specific to the IAOD evaluation function and
put in place an explicit quality control process to manage for quality each
of the processes of programming, planning, implementation,
communication and follow up.
 Address
the evaluation function’s ability to fulfil its learning mandate and
contribute to organisational knowledge (the more specific aspects of
which are dealt with further in this report).
22
It should be noted that the Proposed Revisions to the WIPO Internal Oversight Charter (May
22, 2014) states: The Director, IOD shall publish internal audit and evaluation reports on the WIPO
website within 30 days of their issuance. In exceptional cases, if required to protect security, safety
or privacy, the Director, IOD may, at his/her discretion, redact or withhold a report in its entirety.
Page 19 of 45
4. Human rights and gender equality
Human rights and gender equality are a cornerstone of the UN’s mission and
values. As such they have garnered priority and system wide attention. In
particular these issues are recognised as key to economic and social
development, among other things, and they should be woven into the fabric of
UN organisations, their interventions and their evaluation. It is on this latter
aspect that evaluation in WIPO has an opportunity through its evaluations and
recommendations to support the inclusion of HRGE considerations in the
WIPO interventions it evaluates.
4.1 Findings
The assessment undertook to address the following question:
To what extent are principles and implicit standards related to Human Rights
and Gender Equality (HRGE) integrated in evaluations and what revisions to
the Evaluation Policy and Strategy should be considered to further that
integration?
The review drew key findings from the data that were collected and assessed
these against UNEG norms and standards with a focus on HRGE and as well
the principles, tasks and checklist highlighted in the UNEG guidance
document: Integrating Human Rights and Gender Equality in Evaluation23
The review examined the three evaluation reports24 that were submitted to it
and found that in none of these was there reference to gender and human
rights25 . The review further examined the evaluation report “Technology and
Innovation Support Centers (TISCs)” and the summary of the evaluation
report26 “Support Services to the Intergovernmental Committee on Intellectual
Property and Genetic Resources, Traditional Knowledge and Folklore (IGC)”
and found no reference to gender and human rights.
From the examination of evaluation working papers and interviews with staff
the review found little evidence of systematic integration of HRGE in
evaluation.
The review found that the WIPO Evaluation Policy contains only two
references to gender, one as an example of a theme that may be addressed
within a defined sector and the other as part of the section of Evaluation
Ethics.
http://www.unevaluation.org/HRGE_Guidance
Country Portfolio Evaluation Thailand. Program 1 (Patent Law). Improvement of National, Subregional and Regional IP Institutional and User Capacity.
25 Section 8 of the UNEG Quality checklist for evaluation reports http://www.uneval.org/papersandpubs/documentdetail.jsp?doc_id=607
26 The review was unable to access publicly the full evaluation report
http://www.wipo.int/about-wipo/en/oversight/iaod/evaluation/
23
24
Page 20 of 45
It further found that the WIPO Evaluation Strategy refers to gender as part of
its explanation of coverage criterion and goes onto state: “For all its
independent evaluations, the Evaluation Section will uphold the UN System
obligations and commitments to gender equality and human rights, making
sure that these are increasingly mainstreamed into all of its independent
evaluation work.”
The review did take note of the appointment of an IAOD internal audit staff
person as focal point on gender within WIPO and to the UNEG Taskforce on
Gender Equality and Human Rights working with, inter alia, the UN SWAP Meta-Review/Evaluation Scoring Tool27 .
4.2 Analysis
The findings indicate that principles and implicit standards related to Human
Rights and Gender Equality (HRGE) are not yet integrated fully in IAOD
evaluations.
The UNEG Gender related Norms and Standards provide a framework for
gender responsive evaluations which are wholly applicable to the various
evaluands of IAOD. Considerations of HRGE aspects is relevant to the
evaluation of WIPO interventions and may be significantly important in areas
such as Intellectual Property and Genetic Resources, Traditional Knowledge
and Folklore.28
4.3 Conclusion
The review concludes that principles and implicit standards related to Human
Rights and Gender Equality (HRGE) are not yet fully integrated into IAOD
evaluations. Because of the importance of HRGE it should be integrated fully
to evaluation guidance, i.e. the Oversight Charter and Evaluation Policy, as
well as to the Evaluation Strategies.
http://uneval.org/papersandpubs/documentdetail.jsp?doc_id=1452
For example see: Gender dimensions of intellectual property and traditional medicinal
knowledge
http://www.undp.org/content/dam/aplaws/publication/en/publications/environmentenergy/www-ee-library/biodiversity/gender-dimensions-of-intellectual-property-andtraditional-medicinal-knowledge/Gender_IP_and_TKM_Final_27_Apr07.pdf
27
28
Page 21 of 45
4.4 Recommendations
At the operational level:
 Maintain
regular professional development sessions for IAOD evaluation
staff on gender responsive evaluation.
 As
part of to-be-developed quality control standards for evaluation in
IAOD, quality control and quality assurance processes, operationalise
and integrate HRGE.
 Access
specialised HRGE resources so that IAOD evaluations provide
assessments and recommendations to the stakeholders, programmes,
organisations and networks on HRGE levers for effectiveness and
sustainability.
At the organisational level:
 Articulate
a specific policy component the integration of HRGE in all
IAOD evaluations
 Develop
annual operational plans for the integration of HRGE integration
in evaluation.
Page 22 of 45
5. Assessment of effectiveness
Different from the thrust and nature of the examination and assessment
conducted on conformity by the review29, this chapter addresses the question
of performance of IAOD’s evaluation function relative to its goals, the mission
of WIPO and its contribution to development in Member states. The question
of performance is an ad maximis proposition, i.e. to what extent is the
evaluation function managed to achieve its goals and what can be done to
improve it.
Because use of evaluation is integral to its effectiveness and impact on the
organisation and its stakeholders, it is also addressed in this chapter.
5.1 Findings
The assessment undertook to address the following questions:
Taking into account IAOD capacity and policy, to what extent does the
Evaluation Function contribute to WIPO accountability and learning, as set
forth in the Internal Oversight Charter and expressed in the expectations of
WIPO management?
To what extent are evaluation results incorporated and used in follow-up
activities, how they are monitored and what is their actual impact?
The review drew its findings from interviews with WIPO management staff,
IAOD staff, the survey of WIPO management staff as well as documentation
such as the Oversight Charter, the proposed revisions to the Oversight
Charter, the Evaluation Policy, the 2010-2015 Evaluation Strategy, evaluation
reports and working papers.
As well, key aspects relating to use, follow up and monitoring are addressed
in Chapter 3, 3.1.3 Utility, N4.1.
5.1.1
The view from WIPO management
Overall the review finds that WIPO management are receptive to and
supportive of evaluation in WIPO. Generally WIPO management would like
more evaluations of their programmes and areas of work, and consider the
work carried out by the evaluation function to be relatively independent,
impartial and constructive.
29
Please see chapter 3.
Page 23 of 45
On the other hand, there is a view that the evaluation function does not reach
out enough to management to well understand its needs and that of the
organisation as a whole. The fact that IAOD has the ability to “oversee” the
entire organisation puts it in a unique position which provides it with a
comparative advantage that is not seen to be sufficiently exploited. There are
opportunities to provide input and support at the policy level of WIPO which
do not appear to be recognised and acted on at present.
The management view is formulated that the evaluation function in IAOD has
still not reached maturity. It is seen as insufficiently distinct from internal audit
and from a control approach, and has yet to establish firmly its substantive
competence and credibility. Its niche remains unclear and there is a still a way
to go in engaging with management before it can capture the sense of value
that evaluation deserves.
A majority of interview and survey respondents consider evaluation in WIPO
to be primarily about fulfilling accountability requirements. And although there
is an understanding of the learning aspect of evaluation, respondents
generally consider that evaluation does not provide significantly for learning
and for improving practices.
Evaluation recommendations are on balance considered too vague or general
to be meaningful, and for some, reflect a lack of substantive knowledge of the
evaluand, i.e. the subject areas that are evaluated.
The system for follow up of recommendations is considered to be more of a
control and compliance exercise, with little value added in terms of operational
management or improvement of practices, than a learning function.
5.1.2
Understanding of evaluation
There are differences in level of appreciation of the evaluation function
according to the degree of direct implication of respondents in an evaluation.
Generally the greater the implication the more positive the appreciation. This
may suggest, among other things, that the evaluation process itself has
inherent value that is equal to if not superior to the perceived value, and
hence the utility and use, of the evaluation report including its
recommendations.
This said, evaluations that are carried out “at the request” of management in a
specific area tend to be benefit from what could best be termed a “positive
pre-disposition” of managers who know about evaluation, albeit in general
terms, understand its distinction from control and audit, and are interested in
the added value of external assessment of performance and the learning
potential evaluation brings.
Page 24 of 45
On the other hand, the review found that for those management respondents
less directly involved in evaluation, the distinction between evaluation and the
other functions of IAOD, in particular internal audit but as well control
approaches in general, was not clear.
As a result, management less familiar with evaluation tend to have a less
positive view of evaluation, are not particularly interested in requesting it,
consider it a control exercise and do not see much added value from
evaluation to their management and to the performance of their programmes.
5.1.3
Programming
The positive correlation between appreciation for evaluation and extent of
direct participation in it is also found to carry through to the appreciation of the
relevance of the topic that is evaluated.
In all cases however there is a clear expectation and oft formulated request
from management for more systematic and direct involvement with IAOD’s
Evaluation Section in the programming and planning of evaluations, i.e. with
the evaluators themselves. There is a view that the evaluation function does
not engage sufficiently in dialogue with management and that, as a result, it
does not meet well the needs and requirements of the management of WIPO.
Generally the reasons motivating more involvement in programming of
evaluations tend to fall into two categories. One set of reasons has to do with
evaluating programmes or components thereof that are known not to be
functioning well or that face significant challenges, in order to better
understand why, identify possible solutions and actions to implement, monitor
and adjust these.
The other set of reasons falls into a category which could be best summed up
as learning for improvement, i.e. where a programme is functioning
acceptably and management is looking for ways to improve and scale up.
5.1.4
Evaluation process
For WIPO management involved more directly in evaluations there is a
common view that evaluation processes tend to take too long and incur
transaction costs that are not always commensurate with the benefits to
management, the programme, the organisation and related stakeholders.
While appreciative of the thoroughness and attention given by the evaluation
function to the conduct of evaluations and effort to engage constructively
during the process, there is a view that evaluation should be “quick and light”
to be timely, responsive and relevant to management and organisational
issues in WIPO.
Page 25 of 45
The view and appreciation from WIPO management on external consultants
and their use is at best mixed. Although they tend to be retained by IAOD’s
evaluation function for their IP expertise it does not appear that their input
translates into substantive added value for management. As well the role and
extent of responsibility for the evaluation of external consultants does not
always appear clear to management.
5.2 Analysis
Overall IAOD evaluation is conceived, planned, carried out, perceived and
received as an accountability rather than a learning function. This despite the
fact that learning is put forward as a key objective in WIPO evaluation
guidance documents as well as in IAOD evaluation related documentation.
As such, the unique added value that evaluation can provide as a discipline
and profession by drawing on its extensive and deep body of theory and
practice, remains largely under-utilised. Instead the evaluation function in
WIPO is limited to a significantly and relatively restricted conception and
application as an “accountability tool”, with accountability understood in the
classic sense of “The obligation to account for a responsibility conferred”.
Quite aside from the truism that just because it is written does not mean that it
will necessarily get done, different perspectives on this reality provide some
measure of explanation for the state of the evaluation function’s effectiveness,
use and impact in WIPO.
5.2.1
Evaluation and audit
The fundamental distinction and differences between evaluation and audit is
not reflected in a manner that is sufficiently clear, systematic and appropriate
in the organisation, systems and processes for evaluation in WIPO.
This said, the fact that IAOD’s examination and validation of WIPO’s Program
Performance Report (PPR) has moved from the evaluation function to the
audit function is for example wholly consistent with the fundamental
assurance remit of audit rather than that of evaluation.
While audit may be considered as a type of “evaluative” exercise, it is a
process that is superimposed on an accountability relationship for the purpose
of providing assurance. Audit considers the “what” question, e.g. what was
done, against expectations.
Evaluation addresses the range of questions from the “what”, to the “why”, the
“how” and, most importantly, the “so what”, for “who”, under “what
circumstances”. While evaluation as a discipline provides a almost limitless
array of approaches, methodologies and techniques for determining value, it
does not provide the best configurations for assurance and control purposes.
This is the specialised purview of audit.
Page 26 of 45
5.2.2
Organisation
Position
The organisational position of evaluation in an oversight function brings with it
known and specific challenges, but may also bring benefits depending on
overall context. In the case of WIPO a clear benefit is the independence from
operational management of IAOD in which the evaluation function is located.
The basic challenge is one of identity of evaluation, both within the oversight
function and for stakeholders external to it. Within the oversight function
“evaluation” as conceived, programmed, managed, conducted and followed
up is sometimes difficult to distinguish from performance audit.
Guidance
This is to some extent the case with the current and proposed guidance
frameworks for IAOD’s evaluation function. The definitions of evaluation in the
Oversight Charter, and moreover those of the proposed revisions to the
Charter, correspond to those applicable to a performance audit, i.e. examining
the “what”, more than focussing on the determination of value.
From a management and stakeholder view point external to the oversight
function, evaluation is also often difficult to understand as something more
than an accountability, and specifically a control, mechanism. As such, that
perception tends not to be conducive to the type of engagement that allows
for and nurtures mutual learning. Furthermore that perception conditions the
way in which recommendations and their follow up are treated, i.e. as control
requirements rather than as added value to management.
5.2.3
Operations
Programming and planning
The overarching audit approach within IAOD is also reflected in the use of risk
assessment as the initial frame of reference for prioritising its annual work
plan. The revised evaluation policy states in article 25: “Planning priorities will
be based on risk-assessment, size and needs for evaluation.” While risk
assessment is suited to assurance and performance audit, it is not the most
useful frame for evaluation.
As such, and in addition to consultation with management on selection of
evaluation topics, a framework that is anchored in a value proposition permits
considerations beyond achievement of objectives. A evaluation specific value
oriented frame of reference allows for a greater choice of evaluation
approaches and methodologies, in particular those more suited to nature of
the work WIPO in the domain of IP, e.g. systems based, network approaches,
rights based, equity focussed, capacity assessment, gender lens, social and
intellectual capital, public good, normative, complexity approaches,
developmental evaluation, etc.
Page 27 of 45
Questions of evaluability should also be addressed systematically as part of
programming and work planning stages, rather than once the evaluation has
been agreed to and is being planned. Systematic stakeholder analysis can
form part of programming and planning of evaluations.
Efficiency
With respect to the efficiency of the evaluation process, more direct
engagement and dialogue with management from a learning perspective
would likely reduce the time spent and effort expended in what appears to a
be a relatively formal exchange of documentation and correspondence to
arrive at agreements at planning and reporting stages.
From the perspective of staff of IAOD the back and forth appears sometimes
more like “a pattern of passive resistance” than a genuine effort to craft jointly
a process that will provide value added learning. Again this might be due to
the perception some management staff have of evaluation as a control and
accountability tool, a perception however that may be reinforced by the way
evaluation is defined and presented in guidance documentation.
Learning Resource Group
The Learning Resource Group (LRG) is a kind of reference committee for
evaluations intended to “ (…) increase ownership, transparency and learning
effect of evaluations”.30 For the LRG ToR that the review examined31 there
was majority participation of six WIPO management staff and three external
members.
Although the review was not able to assess systematically the extent to which
generally the LRG contributed to learning and impact, interviews with
management indicated a positive assessment of participation in LRG and its
utility for providing a practical understanding of the evaluation function in
WIPO.
What appears less clear however, both from an examination of the ToR and
from the different perspectives of management on the LRG, is its primary
purpose, e.g. whether it’s for validation of the evaluation, expert input,
governance, quality assurance, facilitating logistics, or a combination thereof.
As is often the case with reference groups in evaluations for which the
purpose might not always be clearly and consistently understood among
participants, independence and efficiency of process may become issues.
External consultants
The use of external consultants by the IAOD evaluation function is systematic
in all cases. This practice is understandable in view of the limited staff
resources that IAOD may affect to evaluations and their associated
requirements of expertise in various domains of IP that IAOD should not be
expected reasonably to possess.
30
31
Country Program evaluation (CPE) – Thailand – TORs Learning Resource Group (LRG)
Ibid
Page 28 of 45
As well, although external consultants typically fulfil a significant role in
evaluations, IAOD through its evaluation function retains full ownership and
responsibility for the evaluation32.
The primary role of the external consultant is not however consistently clear,
e.g. whether it is for IP content expertise, evaluation expertise, additional
human resources, negotiating local arrangements, knowledge of local context,
etc.
While each of the above examples may be justified for hiring external
resources, it is important to be clear in each evaluation about the primary
purpose of the external consultant for reasons of credibility, efficiency and
effectiveness of the evaluation and its process.
At present the approach of the evaluation function appears to be to find
external consultants who combine both IP content expertise with evaluation
expertise, the upshot of which is that the evaluation function sees itself in the
position of “tapping into the same limited pool of evaluators”.
The findings of the review, most notably the appreciation of WIPO
management that evaluations may lack substantive value as well as of
understanding and knowledge of the areas that are being evaluated, together
with the positive appreciation of the constructive way in which the evaluation
function conducts evaluation, suggests that the primary purpose of the
external consultants should be specific IP expertise related to the evaluand.
The evaluation expertise resides with the staff of the evaluation function of
IAOD and it would appear less necessary to resort to additional evaluation
expertise through external consultants unless there is a clear and explicit
reason to do so.
5.3 Conclusion
The review concludes that the Evaluation Function contributes to WIPO
accountability however could contribute more to learning as expressed in the
expectations of WIPO management and in view of the potential that
evaluation offers for organisational learning.33
The review further concludes that although there is a system in place for
monitoring and following up on evaluation recommendations, there is as yet
little in the way of observable or reported impact.
Which is different from some other evaluation functions of public institutions in where the
evaluation report is written by the external consultant and considered as such, i.e. not a product
of the evaluation function per se but that of the external consultant.
33 Please refer to the work of Chris Argyris, Donald Schön, Ron Heifetz, Gregory Bateson and
Peter Senge for an understanding of single, double and triple loop learning.
32
Page 29 of 45
The review concludes that the observed application of a predominant audit
frame of reference to oversight may constrain the effectiveness of the IAOD
Evaluation Function and may limit its ability to exploit fully opportunities for
adding value to WIPO and its stakeholders.
5.4 Recommendations
The distinction between audit and evaluation, in which the primary purpose of
audit is to provide assurance and that of evaluation to determine value, has
critical implications for evaluation at all levels, from organisational to functional
to operational.
The distinction is of a fundamental nature such that it has implications for
foundational guidance documents of WIPO such as the Oversight Charter and
the Evaluation Policy.
As well, it has implications for the way in which the evaluation function is
resourced and managed in relation to the other functions of the oversight
division so as to promote complementarity and synergy.
Finally, it has implications for programming and conducting evaluations
particularly with respect to approaches and methodologies, including quality
control and assurance processes.
The review has not acquired knowledge of the organisational context of WIPO
sufficient and comprehensive enough to understand and appreciate the extent
to which recommendations of a fundamental nature at the organisational level
are realistically feasible and actionable.
For this reason the review is not in an informed position to provide practical
recommendations at organisational and policy levels consistent with its
assessment that the application of a frame of reference more appropriate to
evaluation will increase significantly its effectiveness, learning impact and
value for WIPO.
The recommendations that follow focus therefore on the operational level
recognising that they should assist in improving the evaluation function within
the existing organisational and policy frameworks.
 IAOD
should consider improving the quality and nature of its engagement
with WIPO management in line with its expressed desire for more direct
and meaningful involvement with the evaluation function and with
particular attention to programming of evaluations.
 The
evaluation function should systematically include evaluability
assessments at the programming stage of work planning for evaluations.
It should also use stakeholder analysis systematically and when useful as
part of programming and planning.
Page 30 of 45
 As
part of programming and planning processes the evaluation function
should focus more systematically on intended users of evaluation by
applying utilisation-focussed approaches and frameworks such as the
Utilisation Focussed Checklist of Michael Quinn Patton34.
 The
evaluation function should identify systematically the bottlenecks in
the evaluation process that hinder its efficiency, some of which are noted
in the discussion, and take steps to address them such as: more direct
and collaborative engagement at the planning stage, an agreed
communication plan for a “no surprises” approach, a clear and explicit
distinction between findings and conclusions both in terms of process for
preparing the report and in its published form.
 The
evaluation function should consider diversifying its portfolio of
evaluations in order to conduct where it judges worthwhile, short cycle
evaluations such as real time evaluation.
 The
evaluation function should revisit the purpose and use of reference
groups for evaluation, i.e. such as the Learning Resource Group, to
determine and make explicit their primary purpose while avoiding “multipurpose” mandates that may affect independence of the evaluation or
otherwise create confusion as to respective roles and responsibilities of
participants. For example a steering committee, the function of which is to
govern, i.e. support the overall orientation of, the evaluation, is different
from a reference group which acts as a sounding board based on the
specialised knowledge it has of the evaluand and its context, both of
which are in turn different from a management group which acts as the
primary project and contract management interface between the
commissioner and the evaluator.
In all cases there should an appropriate understanding and application of
the procedures and respective roles required to protect evaluation
independence, i.e. respecting the evaluator’s independence in exercising
judgment to reach and formulate conclusions without interference,
pressure or influence from the LRG or other groups.
 The
evaluation function should, in each case in which it resorts to
external consultants, make clear and explicit the primary purpose of the
consultant’s contribution, i.e. whether it is for IP content expertise and/or
for another reason. The use of external consultants should be consistent
with the recognition that evaluation expertise rests within the evaluation
function of IAOD and that IAOD is not, and should not, be expected to
possess IP content expertise. It follows that external consultants with IP
content expertise do not also need to have evaluation expertise in all
cases.
34
http://www.wmich.edu/evalctr/archive_checklists/ufe.pdf
Page 31 of 45
6. Adequacy of resources
The question that was put to the review is: To what extent are the resources
available to the Evaluation Function adequate for it to achieve its mandate?
The review addressed this question from two perspectives, i.e. the quality and
quantity of staff resources allocated to the evaluation function.
6.1 Findings
The question of quality is addressed in Chapter 3 – Assurance on conformity
– 3.1.2 Credibility N9 and is found to be adequate.
On the question of quantity the review based its assessment on available
comparative data, including that which is in Annex 3 of the WIPO revised
Evaluation Policy, as well as the staff allocation to the evaluation function of
comparable oversight services that also include audit such as OECD, i.e. 2
full time staff, and the Council of Europe, i.e. 2 full time staff.
More importantly the review assessed the adequacy of staffing levels to
realise the workplan described in the WIPO 2010-2015 Evaluation Strategy:
Independent strategic evaluations: 2 in period of 6 years
Independent country level evaluations: 10
Thematic: 6
Program: 5
6.2 Conclusion
Taking into account the model on which the evaluation function of IAOD is
based, i.e. leadership and full participation in the evaluation with the
additional contribution of contracted external consultants, the specialised
competency requirements for evaluators and in light of the work programme
that is proposed, the review concludes that in terms of quality IAOD
evaluation staff resources are adequate however the quantity of qualified
evaluation staff may not be sufficient.
6.3 Recommendation
IAOD should monitor its level of allocation of qualified evaluation staff to its
evaluation function and the feasibility of its intended workplan, based on a
systematic and data based analysis of estimated workload.
7. Adequacy of quality assurance practices
Page 32 of 45
The question that was put to the review is: How adequate is the quality
assurance system for the Evaluation Function and in particular the periodicity
of reviews?
The review addressed this question primarily from the perspective of
principles for quality control and assurance of specific evaluations, consistent
with UNEG Norms and Standards for Evaluation. The review considered the
periodicity of review as determined in the Evaluation Policy.
7.1 Findings
The review did not find in place a quality assurance process consistent with
accepted principles. The review did not find explicit quality standards for
quality control of evaluation however did note that the 2010-2015 Evaluation
Strategy indicates that by end of the strategy, evaluation quality assurance
standards will have been developed and that 80% of WIPO evaluations will
have applied them.
The review further found that there appears to be a lack of clarity and
understanding with respect to the distinction between the supervisory function
of the hierarchical superior and quality assurance, e.g. in the case of the
country programme evaluation for Thailand it is stated that “The Director,
IAOD, will conduct quality assurance”.
The review noted that the Evaluation Policy calls for a review every 3 years.
7.2 Conclusion
The review concludes that there is not a sufficiently developed quality
assurance system for the Evaluation Function of IAOD.
The 3 year periodicity of reviews appears reasonable.
7.3 Recommendations
 IAOD
should give priority to developing and instituting a quality control
process for evaluation based on explicit quality standard. Quality control
should be the responsibility of the evaluation team leader.
 IAOD
should develop and institute a quality assurance process using
professionally qualified evaluation experts external to IAOD to provide
assurance and advice at key points in the evaluation process, e.g.
planning based on approach papers, terms of reference and inception
reports; analysis and assessment based on findings and preliminary
conclusions; final reporting.
Page 33 of 45
8. WIPO-IAOD Evaluation Function Survey
The survey consisted of 10 multiple choice questions and one open question.
The link to the online survey was sent to 34 WIPO staff members. It was
completed by 14 people, making for a response rate of 41%. Out of the 14
respondents, 11 answered all multiple choice questions. All multiple choice
questions used a rating scale from 1 (lowest) to 5 (highest).
8.1 Relevance and use of evaluations
The majority of respondents indicated that WIPO evaluations focus on
subjects that are most useful to them. 14% of the respondents rated this
question with a 5, 64% with a 4 and 21% with a 3. With an average of 3.93,
this is the question which received the highest average score.
The responses indicate that WIPO evaluations are perceived at being slightly
better at contributing to accountability requirements than at contributing to
learning requirements. The average rating given for the evaluations’
contribution to accountability requirements is 3.50, while it is 2.93 for the
evaluations’ contribution to learning requirements. Three respondents
indicated that the evaluations did not at all contribute to their learning
requirements.
There was some disagreement on the extent to which evaluations contribute
to establishing best practice. According to 3 respondents this was not at all
the case, while for 1 respondent this was fully the case and 3 others rated this
question with a 4. With an average score of 2.83, this question is among
those that received the lowest average score.
8.2 Evaluation process
The extent to which evaluation processes engage with the survey
respondents and their stakeholders in ways that make evaluations useful
received an average rating of 3.15 with the extremes of the rating scale only
being used by one respondent each.
The question to what extent evaluations were conducted a) transparently, b)
independently, and c) impartially mostly received average to positive feedback
with transparency receiving a lower average score (3.33) than independence
(3.64) and impartiality (3.73).
Page 34 of 45
8.3 Communication of evaluation findings
Respondents were asked to what extent evaluation findings are
communicated to them and their stakeholders in a a) useful, b) constructive
and c) timely manner. The responses were on average slightly on the positive
side with the “constructive” aspect of communication receiving a higher
average score (3.67) than the two other aspects (3.17 for both). For each
aspect, one person responded that this was “not at all” the case.
On the question to what extent evaluation findings were communicated in an
impartial way with adequate levels of technical and political credibility, most
respondents (42%) gave a medium rating (3 points), with the average being
slightly lower (2.83).
8.4 Quality of evaluations
On the question whether evaluations are technically sound, there was an
interesting division between 42% of the respondents who rated this question
rather negatively with a 2 and 50% of the respondents who rated it rather
positively with a 4. One person gave an average rating (3) to this question,
while the extremes of the rating scale (1 and 5) were not used.
None of the respondents had a strong opinion on the extent to which
evaluation reports are of high quality as all responses range from 2 to 4 points
(average of 3.09).
8.5 Additional comments
The opportunity to provide additional comments was only used by two
respondents. One of them recommends a better integration of M&E into the
process of program planning, further capacity building of staff on M&E and a
stronger participation of staff members in evaluation.
The other comment refers to the lack of understanding by IAOD of the
respondent’s substantive area of work and of the division of responsibilities
therein, while stressing that the overall experience of working with IAOD has
otherwise been positive.
8.6 Observations
Most ratings are around the medium value, with relatively few strong
(negative/positive) opinions. The range of average ratings per question goes
from 2.83 to 3.93, while the average of the averages is at 3.29. The
responses on the evaluations’ contribution to learning and to the creation of
best practice go into the same direction and indeed underline that the
evaluations are seen as less useful for knowledge creation or changing
practices than for accountability purposes.
Page 35 of 45
9. Additional remarks
Although it is not within the agreed scope of the review to assess evaluation in
WIPO outside of the IAOD evaluation function, the review notes that the
requirements of programmes for evaluations over their life cycle do not
appear to be covered explicitly in WIPO.
As noted by a respondent: “Program and Budget and Program Performance
Reporting provide a framework for planning and monitoring. In this context,
there is a vacuum on evaluation which is neither budgeted for nor otherwise
conceptualized, let alone planned; this vacuum cannot be filled by IAOD’s
evaluation function in its current state, nor is it sure whether it should be”.
To the extent that evaluations in WIPO may be commissioned and conducted
outside the remit of IAOD, there should be a clear and explicit WIPO policy
covering non-IAOD evaluations.
Such a policy should address conformity of all WIPO evaluations with UNEG
Norms and Standards, including inter alia those related to HRGE, quality
control and assurance processes, competencies of commissioners and
evaluators, communication and dissemination, as well as relationship and
coordination with IAOD.
Page 36 of 45
10.
Summary of ratings
SUMMARY OF RATINGS
IAOD conformity with UNEG Norms
Conforms
Independence
N2.3: The governing bodies/heads of organizations are responsible for ensuring that adequate
resources are allocated to enable the evaluation function to operate effectively and with due
independence.
N.5.1: Impartiality is the absence of bias in due process, methodological rigor, consideration and
presentation of achievements and challenges. The requirement for impartiality exists at all points in the
process: planning, conduct, reporting
N6.1: The evaluation function has to be located independently from the other management functions so
that it is free from undue influence and that unbiased and transparent reporting is ensured. It needs to
have full discretion in submitting directly its reports for consideration at the appropriate level of decisionmaking pertaining to the subject of evaluation.
N6.2: The head of evaluation must have the independence to supervise and report on evaluations.
N6.3: To avoid conflict of interest and undue pressure, evaluators need to be independent, implying that
members of an evaluation team must not have been directly responsible for the policy-setting, design,
or overall management of the subject of evaluation, nor expect to be in the near future.
N6.4: Evaluators must have no vested interest and have the full freedom to conduct impartially their
evaluative work, without potential negative effects on their career development. They must be able to
express their opinion in a free manner.
Credibility
N4.2: The evaluation plan can be the result of a cyclical or purposive selection of evaluation topics. The
purpose, nature and scope of evaluation must be clear to evaluators and stakeholders. The plan for
conducting each evaluation must ensure due process to ascertain the timely completion of the mandate,
and consideration of the most cost-effective way to obtain and analyze the necessary information.
Page 37 of 45
Partially
conforms
Does not
conform
N5.2: Impartiality increases the credibility of evaluation and reduces the bias in the data gathering,
analysis, findings, conclusions and recommendations. Impartiality provides legitimacy to evaluation and
reduces the potential for conflict of interest.
N8.1: Each evaluation should employ design, planning and implementation processes that are
inherently quality oriented
N9: The head of the evaluation function, evaluation staff and evaluators should have proven
competencies to manage and/or conduct evaluations
N10.1: Transparency and consultation with the major stakeholders are essential features in all stages of
the evaluation process. This improves the credibility and quality of the evaluation. It can facilitate
consensus building and ownership of the findings, conclusions and recommendations.
N11: Evaluators must have personal and professional integrity, respect people and their rights,
confidentiality of data and sources, and be sensitive to beliefs as well as issues of gender and human
rights
N8.2: Evaluation reports must present in a complete and balanced way the evidence, findings,
conclusions and recommendations. They must be brief and to the point and easy to understand. They
must explain the methodology followed, highlight the methodological limitations of the evaluation, key
concerns and evidenced-based findings, dissident views and consequent conclusions,
recommendations and lessons. They must have an executive summary that encapsulates the essence
of the information contained in the report, and facilitate dissemination and distillation of lessons.
N10.2: documentation on evaluations in easily consultable and readable form should also contribute to
both transparency and legitimacy
Utility
13.1 Evaluation contributes to knowledge building and organizational improvement. Evaluations should
be conducted and evaluation findings and recommendations presented in a manner that is easily
understood by target audiences.
N2.6: The Governing Bodies and/or Heads of organizations and of the evaluation functions are
responsible for ensuring that evaluation contributes to decision making and management. They should
ensure that a system is in place for explicit planning for evaluation and for systematic consideration of
the findings, conclusions and recommendations contained in evaluations. They should ensure
appropriate follow-up measures including an action plan, or equivalent appropriate tools, with clear
accountability for the implementation of the approved recommendations.
N4.1 Proper application of the evaluation function implies that there is a clear intent to use evaluation
findings. In the context of limited resources, the planning and selection of evaluation work has to be
carefully done. Evaluations must be chosen and undertaken in a timely manner so that they can and do
inform decision-making with relevant and timely information. Planning for evaluation must be an explicit
part of planning and budgeting of the evaluation function and/or the organization as a whole. Annual or
multi-year evaluation work programmes should be made public.
N4.2 The evaluation plan can be the result of a cyclical or purposive selection of evaluation topics. The
purpose, nature and scope of evaluation must be clear to evaluators and stakeholders. The plan for
conducting each evaluation must ensure due process to ascertain the timely completion of the mandate,
Page 38 of 45
and consideration of the most cost-effective way to obtain and analyze the necessary information.
N2.7: governing bodies and/or heads of organizations and of the evaluation function are responsible for
ensuring that there is a repository of evaluation and a mechanism for stilling and disseminating lessons
to improve organizational learning and systemic improvement
Page 39 of 45
EVALUATION FUNCTION QUALITY ASSESSMENT
TABLE OF RECOMMENDATIONS
No RECOMMENDATION #
(§ reference in the report)
1
3
Management comment and action
plan
3
4
Deadline for
implementation
Director,
IAOD
Thierry
Rajaobelina
Q 3 2014 for
publication
Q4 2015 for
dissemination
and
communication
IAOD
Evaluation
Section
Claude
Hilfiker, Head
IAOD
Evaluation
Section
Claude
Hilfiker, Head
Q 3 2014
IAOD
Evaluation
Q 2 2014
Conformity Assurance
3.3.1 Publish, disseminate and
communicate publicly all evaluation
reports.
2
Responsible
Unit
Manager
A revised Oversight Charter will be
proposed to the General Assembly
in September 2014, allowing
publication of audit and evaluation
reports. Later on, IAOD will prepare
standard operating procedures
(SOP) for dissemination.
3.3.2 Address bottlenecks in the
We will pursue and formalize after
conduct of evaluations that threaten
action reviews, to identify i.A. time
their timeliness.
constraints and introduce these in
our rolling planning for upcoming
evaluations.
3.3.3 Develop quality standards specific We have committed to the UNEG
to the IAOD evaluation function and
standards for evaluation processes
put in place an explicit quality control and products. We will further
process to manage for quality each
formalize quality control (QC)
of the processes of programming,
processes in a customized internal
planning, implementation,
SOP on Quality Control, defining the
communication and follow up.
various steps and responsibilities for
QC in planning, field work, final
release and follow-up.
3.3.4 Address the evaluation function’s We will pursue and formalize after
ability to fulfil its learning mandate
action reviews, to identify i.A.
Page 40 of 45
Q 3 2014
No RECOMMENDATION #
(§ reference in the report)
and contribute to organizational
knowledge.
5
6
7
8
Management comment and action
plan
Responsible
Unit
Manager
Section
Claude
Hilfiker, Head
IAOD
Evaluation
Section
Claude
Hilfiker, Head
Deadline for
implementation
We will further formalize QC
processes in the internal SOP on
Quality Control, which will
incorporate the UNEG HRGE
minimal standards as applicable for
IAOD evaluations.
On contract assignments:
depending on availability of funds,
include in evaluation teams experts
who (also) have HRGE experience.
IAOD
Evaluation
Section
Claude
Hilfiker, Head
Q 4 2014
IAOD
Evaluation
Section
Claude
Hilfiker, Head
Q 3 2014
What is to be done by IAOD is to
provide guidance and strategy for
these aspects in the context of
revising its Policy and Strategy
along with extending discussions
with the WIPO Gender Specialist on
IAOD
Evaluation
Section
Claude
Hilfiker, Head
learning components (events and
products) and planning for these in
upcoming evaluations.
4.
Human Rights and Gender
Equality
At the operational level:
4.4.1 Maintain regular professional
development sessions for IAOD
evaluation staff on gender responsive
evaluation.
4.4.2 As part of to-be-developed
quality control standards for evaluation
in IAOD, quality control and quality
assurance processes, operationalize
and integrate HRGE.
4.4.3 Access specialized HRGE
resources so that IAOD evaluations
provide assessments and
recommendations to the stakeholders,
programs, organizations and networks
on HRGE levers for effectiveness and
sustainability.
At the organizational level:
4.4.4 Articulate a specific policy
component the integration of HRGE in
all IAOD evaluations.
Possible action point: have all IAOD
evaluation professionals trained on
the adhered HRGE frame of
reference.
Page 41 of 45
To be
incorporated in
the 2015
oversight plan
Q 4 2014
No RECOMMENDATION #
(§ reference in the report)
9
4.4.5 Develop annual operational
plans for the integration of HRGE in
evaluation.
10
5.
12
how to incorporate evaluation as a
strategic component within the
Policy.
This will be part of Evaluation
Planning within the IAOD annual
Work Plan.
Responsible
Unit
Manager
Deadline for
implementation
IAOD
Evaluation
Section
Claude
Hilfiker, Head
Q 4 2014
Director,
IAOD
Thierry
Rajaobelina
Implemented.
On-going.
IAOD
Evaluation
Section
Claude
Hilfiker, Head
Q 4 2014
IAOD
Evaluation
Section
Claude
Q 4 2014
Assessment of effectiveness
5.4.1 IAOD should consider improving
the quality and nature of its engagement
with WIPO management in line with its
expressed desire for more direct and
meaningful involvement with the
evaluation function and with particular
attention to programming of evaluations.
11
Management comment and action
plan
5.4.2 The evaluation function should
systematically include evaluability
assessments at the programming stage
of work planning for evaluations. It
should also use stakeholder analysis
systematically and when useful as part
of programming and planning.
5.4.3 As part of programming and
planning processes the evaluation
function should focus more
systematically on intended users of
Director IAOD prepares its oversight
plan in consultation with Senior
Management, Management,
Member States and other oversight
bodies. This will continue. At the
engagement level, initial meetings
allow discussion with Management
to get their views and concerns
before finalizing the plan.
Following systematic assessment of
evaluation needs, IAOD Evaluation
Section will conduct systematic
evaluability assessments according
to criteria yet to be decided (see e.g
criteria that were developed for
deciding on program evaluations)
Following systematic assessment of
evaluation needs, IAOD Evaluation
Section will apply state-of-the art
selection criteria for utilization-
Page 42 of 45
No RECOMMENDATION #
(§ reference in the report)
13
evaluation by applying utilizationfocused approaches and frameworks
such as the Utilization Focused
Checklist of Michael Quinn Patton.
5.4.4 The evaluation function should
identify systematically the bottlenecks in
the evaluation process that hinder its
efficiency, some of which are noted in
the discussion, and take steps to
address them such as: more direct and
collaborative engagement at the
planning stage, an agreed
communication plan for a “no surprises”
approach, a clear and explicit distinction
between findings and conclusions both
in terms of process for preparing the
report and in its published form.
14
5.4.5 The evaluation function should
consider diversifying its portfolio of
evaluations in order to conduct where it
judges worthwhile, short cycle
evaluations such as real time evaluation.
15
5.4.6 The evaluation function should
revisit the purpose and use of reference
groups for evaluation, i.e. such as the
Learning Resource Group, to determine
and make explicit their primary purpose
Management comment and action
plan
focused evaluation.
For the “bottlenecks”, and “no
surprise approach”, this is partly
addressed by the responses to the
two previous points.
On the communication plan part of
this recommendation, we already
agreed that we need to develop
some ideas on how to frame
evaluation products and processes
for communicating to people about
evaluation results in the public
domain. This should be started
asap, but finalized in the second half
of this year.
We are doing this already when we
have capacity to respond to
requests (e.g. IGC and RRP). If we
are to do all evaluations in shorter
cycles, we will need to refer to
measures that are feasible on the
resourcing part.
Develop clear standard ToR for
LRGs.
Page 43 of 45
Responsible
Unit
Manager
Hilfiker, Head
Deadline for
implementation
IAOD
Evaluation
Section
Claude
Hilfiker, Head
Q 4 2014
IAOD
Evaluation
Section
Claude
Hilfiker, Head
Implemented
On-going
IAOD
Evaluation
Section
Claude
Hilfiker, Head
Q 3 2014
No RECOMMENDATION #
(§ reference in the report)
16
17
while avoiding “multi-purpose” mandates
that may affect independence of the
evaluation or otherwise create confusion
as to respective roles and
responsibilities of participants.
5.4.7 The evaluation function should, in
each case in which it resorts to external
consultants, make clear and explicit the
primary purpose of the consultant’s
contribution, i.e. whether it is for IP
content expertise and/or for another
reason. The use of external consultants
should be consistent with the recognition
that evaluation expertise rests within the
evaluation function of IAOD and that
IAOD is not, and should not, be
expected to possess IP content
expertise. It follows that external
consultants with IP content expertise do
not also need to have evaluation
expertise in all cases.
6.
Adequacy of resources
6.3.1 IAOD should monitor its level of
allocation of qualified evaluation staff to
its evaluation function and the feasibility
of its intended workplan, based on a
systematic and data based analysis of
estimated workload.
Management comment and action
plan
Responsible
Unit
Manager
Deadline for
implementation
Develop a vision for the IAOD
evaluation function way of working
while engaging external experts by
clarifying their roles and
responsibilities within the various
assignments, enhancing the
evaluation technical expertise of
IAOD and thus reflecting it in the
design of ToR for the different types
of expertise and deliverables
required.
IAOD
Evaluation
Section
Claude
Hilfiker, Head
Q 3 2014
Ex-post calculations based on levels
of effort for the 2012-2013
assignments have been used for
preparing the annual oversight plan.
This will continue. The work plan
will be adapted accordingly if
needed.
Page 44 of 45
Director,
IAOD
Thierry
Rajaobelina
2015 and future
year’s oversight
plan
No RECOMMENDATION #
(§ reference in the report)
18
19
Management comment and action
plan
Responsible
Unit
Manager
Deadline for
implementation
Include internal QC based on the
UNEG Quality Standards for
evaluation in the customized internal
SOP.
IAOD
Evaluation
Section,
Claude
Hilfiker, Head
Q 3 2014
Decide on feasibility of including
additional external experts for the
current and future biennia. This will
be contingent on the availability of
resources.
Director,
IAOD
Thierry
Rajaobelina
Q 3 2014
7.
Adequacy of quality assurance
practices
7.3.1 IAOD should give priority to
developing and instituting a quality
control process for evaluation based on
explicit quality standard. Quality control
should be the responsibility of the
evaluation team leader.
7.3.2 IAOD should develop and
institute a quality assurance process
using professionally qualified evaluation
experts external to IAOD to provide
assurance and advice at key points in
the evaluation process, e.g. planning
based on approach papers, terms of
reference and inception reports; analysis
and assessment based on findings and
preliminary conclusions; final reporting.
Page 45 of 45