E
CDIP/15/3
ORIGINAL: ENGLISH
DATE: FEBRUARY 12, 2015
Committee on Development and Intellectual Property (CDIP)
Fifteenth Session
Geneva, April 20 to 24, 2015
EVALUATION REPORT FOR THE PROJECT ON OPEN COLLABORATIVE
PROJECTS AND IP-BASED MODELS
prepared by Mr. Daniel Keller, Consultant, Leubringen, Switzerland
1.
The Annex to the document contains an external independent Evaluation Report for the
Project on Open Collaborative Projects and IP-Based Models, undertaken by Mr. Daniel Keller,
Consultant, Leubringen, Switzerland.
2.
The CDIP is invited to take note of
the information contained in the Annex to this
document.
[Annex follows]
CDIP/15/3
ANNEX
LIST OF ACRONYMS ................................................................................................................ 2
EXECUTIVE SUMMARY ............................................................................................................ 3
1.
INTRODUCTION ................................................................................................................. 6
(A)
PROJECT BACKGROUND AND DESCRIPTION ........................................................ 6
(B)
SCOPE, PURPOSE, METHODOLOGY AND LIMITATIONS OF THIS EVALUATION.. 7
(i)
(ii)
(iii)
(iv)
2.
Scope ........................................................................................................................... 7
Key purpose ................................................................................................................. 7
Methodology ................................................................................................................. 8
Main limitations to this evaluation ................................................................................. 8
FINDINGS AND ASSESSMENT ......................................................................................... 9
(A)
PROJECT PREPARATION AND MANAGEMENT ....................................................... 9
(i) Project preparation ....................................................................................................... 9
(ii) Use of project planning tools ......................................................................................... 9
(iii) Project Management................................................................................................... 11
(B)
RELEVANCE ............................................................................................................. 11
(i) Policy relevance ......................................................................................................... 11
(C)
EFFECTIVENESS ...................................................................................................... 12
(i) Output 1: Taxonomy-Analytical Study ......................................................................... 12
(ii) Output 2: Open-ended Meetings with Member States................................................. 12
(iii) In-depth evaluation study ............................................................................................ 13
(iv) Expert meeting ........................................................................................................... 13
(vi) Initial outcomes observed ........................................................................................... 14
(vii) Impact......................................................................................................................... 14
(D)
(i)
(ii)
(iii)
(iv)
(E)
EFFICIENCY .............................................................................................................. 14
Financial implementation ............................................................................................ 14
Assessment of approach ............................................................................................ 14
Assessment of quality of outputs ................................................................................ 14
Synergies with other activities conducted by the Secretariat ....................................... 15
LIKELIHOOD OF SUSTAINABILITY OF RESULTS ................................................... 15
3.
CONCLUSIONS ................................................................................................................ 16
4.
RECOMMENDATIONS ..................................................................................................... 17
LIST OF APPENDIXES............................................................................................................ 19
CDIP/15/3
Annex, page 2
LIST OF ACRONYMS
CDIP
CHF
DA
DAC
DACD
IP
IPR(s)
LDCs
OECD
SMART (indicators)
TISCs
ToRs
UN
WIPO
Committee on Development and Intellectual Property
Swiss Francs
Development Agenda
Development Assistance Committee (of the OECD)
Development Agenda Coordination Division
Intellectual Property
Intellectual Property Rights
Least Developed Countries
Organization for Economic Co-operation and Development
Specific, Measurable, Ambitious, Relevant and Time-bound
Technology and Information Support Centers
Terms of Reference (of this evaluation)
United Nations
World Intellectual Property Organization
CDIP/15/3
Annex, page 3
EXECUTIVE SUMMARY
This independent final evaluation commissioned by the Secretariat covers the “Project on Open
Collaborative Projects and IP-Based Models” (Project Code: DA_36, “the Project”) under the
Development Agenda (DA). The Project was approved by the 6th session of the Committee on
Development and Intellectual Property (CDIP) in November 2010. Project objectives were
framed by the DA Recommendation 36 (Cluster D), which calls for exchanging experiences on
Open Collaborative Projects such as the Human Genome Project, as well as on intellectual
property (IP) models. Deliverables included two studies, an open-ended meeting with Member
States, an expert meeting, the establishment of an interactive platform and the incorporation of
resulting recommendations into relevant WIPO programs after approval by Member States. The
work on the platform is still ongoing. The Project started on January 1, 2011 and formally
ended on June 30, 2014, following a non-cost extension of 12 months approved by the 12th
session of the CDIP. According to the final progress report presented to the 14th session of the
CDIP, 27% of the total budget of 895,000 Swiss francs was spent.
The evaluation work was conducted from December 10, 2014 to April 24, 2015 by an
independent evaluator in close coordination with the Development Agenda Coordination
Division (DACD) and resulted in the following conclusions:
Conclusion 1 on relevance: Strengthening the use of Open Collaborative Projects and IP
based models continues to be of high relevance
Open Collaborative Projects offer opportunities for exchanging intellectual assets worldwide on
a voluntary basis in mutual benefit, regardless of a possible North-South divide. This is of high
interest for both the developing and the developed world. Through sharing the insight gained
on models used, WIPO’s work provides a unique value added to foster new North-South
partnerships. The model of Open Collaboration is applicable beyond what is commonly
understood as “innovation” and extends also to creative industries. Open Collaborative Projects
are an effective tool to enhance the “valorization” of intellectual assets, including but not limited
to IP in the public domain. In this sense, project objectives are highly relevant to facilitating the
use of IP for development, which is an international priority reflected in Strategic Goal III of the
Program and Budget for the Biennium 2014/2015. The conclusions of the Project are
potentially relevant for WIPO’s work in many fields, including promoting innovation, capacity
building for IP service providers, training, the cooperation with universities, and policy advice.
Conclusion 2 on project preparation and management: The Project addresses DA
Recommendation 36 in a creative and appropriate way. Activities were well organized.
The application of standard project planning and monitoring tools was however weak.
The project was significantly over budgeted and implemented at a slow pace.
The Secretariat translated the relatively open DA Recommendation 36 into a clearly articulated
delivery strategy and methodology. The type and planned sequencing of activities was
conducive to achieving objectives. Researching on the use of open collaborative models,
identifying good practices, and subsequently offering a platform for discussing results and
sharing experiences among experts and other key stakeholders, is an appropriate way to foster
the exchange of experience. Those activities that had been completed were well prepared and
organized.
The application of standard tools for result-based management in project planning and
monitoring was however weak. Progress reports list activities rather than comparing achieved
against planned results using specific, measurable, ambitious, relevant and time-bound
(“SMART”) indicators. Assumptions and risks were neither identified nor monitored. Due to a
lack of a realistic and detailed cost calculation, the Project was significantly over budgeted.
Inaccurate budgeting blocks resources, which would otherwise be available to generate benefits
CDIP/15/3
Annex, page 4
for Member States. Furthermore, there is a risk that projects are “perpetuated” for the sole
reason that resources are still available. A budget and financial report presenting type of costs
for each objective is not available.
Project implementation advanced slowly. The originally planned 30 months should have been
sufficient to implement the limited number of activities. Even after the CDIP granted a project
extension, no measures were taken to accelerate implementation. Delays are costly and cause
a loss of momentum. If not explainable by the impact of unforeseeable external factors,
implementation delays erode the confidence of Member States into the Secretariat’s project
implementation capability.
Conclusion 3: Outputs were of high quality, but not all of them were delivered. The
Secretariat generally made economic use of resources. Without a follow-up, initial
promising results are likely to be lost.
The Project did not reach all of its planned objectives. Through the delivery of four out of six
planned outputs, the Project contributed to the identification of successful Open Collaborative
Projects, in particular in developing countries, to deriving best practices from them and to
discussing the results within a limited audience. The not yet completed Interactive Platform
(Output 5) is pivotal for sharing good practices with a wider audience and to make practical tools
to support open innovation available. Due to the delays in completing the platform, WIPO’s
work in the field of Open Collaboration risks to lose momentum. No steps have been taken
towards an integration of project results into WIPO’s existing Program activities. The wording of
“output” 6 indicates that Member States would expect a specific proposal of the Secretariat.
Without a follow-up, the Project’s reach remains limited and initial promising results are likely to
be lost.
RECOMMENDATIONS
Recommendation 1 (from conclusion 3) to the WIPO Secretariat on finalizing the
interactive platform (planned output 5 of the Project)
(a)
(b)
(c)
(d)
Complete a beta version of the interactive platform;
Test-run the interactive platform and obtain feed-back from users;
Integrate the feed-back from users;
Present a final version of the platform to the 17th session of the CDIP in November 2015;
and
(e) Assign a clear responsibility and allocate resources to maintain and update the
interactive platform regularly.
Recommendation 2 (from conclusions 1 and 3) to the WIPO Secretariat on preparing a
proposal to the CDIP on how to facilitate Open Innovation through WIPO’s existing
Programs (planned Output 6 of the Project)
(a) Continuing identifying, collecting and sharing best practices in the field of Open
Collaborative Projects (studies);
(b) Offer practical capacity building (including tool kits) tailored to specific target users;
(c) Offer capacity building to IP and/or innovation service providers in developing countries,
such as for example IP Offices, Technology Transfer Centers etc.;
(d) Support specific open collaborative pilot projects in developing countries; and
(e) Advise Member States in creating an enabling environment for open collaboration in
their IP policies.
CDIP/15/3
Annex, page 5
Recommendation 3 (from conclusion 3) to the WIPO Secretariat on strengthening WIPO’s
presence in open collaboration conferences and fora
A regular presence and visibility of WIPO in international conferences on Open Innovation
(including but not limited to events organized by other UN organization) may help WIPO to
position itself as a “competence center” in the field of Open Collaborative Projects, to create
visibility and to benefit from additional experience of a wide array of conference participants.
Recommendation 4 (from conclusion 2): To the Secretariat on ensuring the application of
planning and monitoring tools in project cycle management
(a) Strengthen the quality control of new projects submitted to the CDIP in regards to proper
application of WIPO’s existing project tools for project cycle management;
(b) Strengthen the quality control of progress reports submitted to the CDIP to ensure that
WIPO’s existing project tools for project cycle management are properly applied;
(c) Consider introducing the logical framework as a basis for project cycle management;
(d) Consider the introduction of compulsory courses for future project managers on project
cycle management; and
(e) Ensure regular coaching of project managers on a demand basis.
CDIP/15/3
Annex, page 6
1.
INTRODUCTION
1.
This independent final evaluation commissioned by the Secretariat covers the “Project on
Open Collaborative Projects and IP-Based Models” (Project Code: DA_36, “the Project”) under
the Development Agenda (DA) and is guided by the Terms of Reference (ToRs) dated
December 9, 2014 (Appendix 2). An inception report operationalized the ToRs.
2.
The work was undertaken between December 10, 2014 and April 24, 2015 by an
independent external evaluator1 in close coordination with the Development Agenda
Coordination Division (DACD).
(A)
PROJECT BACKGROUND AND DESCRIPTION
3.
The Project was adopted by the 6th session of the Committee on Development and
Intellectual Property (CDIP) in Geneva (November 22 to 26, 2010). The first thematic project
proposal of the Secretariat (CDIP 6_6 dated October 10, 2010) underwent minor amendments
during this session (see CDIP 6_6_rev, November 26, 2010).
4.
Project objectives are framed by the DA Recommendation 36 (Cluster D), which calls
specifically for exchanging experiences on Open Collaborative Projects such as the Human
Genome Project as well as on intellectual property (IP) models.
5.
Open Collaborative Projects intend to bring into play innovative solutions from a wide
range of inventors and problem-solvers through the sharing of best practices among several
entities. Open collaborative innovation may be defined as an exchange of knowledge between
an organization or community and its environment. It may be promoted through a variety of
arrangements, such as licensing of IP, subcontracting, R&D collaborative contracts, joint
ventures and patent pools, Internet-enabled trends that foster customer driven innovation
include crowd-sourcing, ideas competitions etc.2
6.
Open Collaborative Projects and IP-based models are not a purpose in itself, but one of
many tools to stimulate innovation and facilitate the valorization of IP, including but not limited to
IP in the public domain.
7.
The Project’s approach3 was to research on and present successful Open Collaborative
Projects and IP-based models, in particular in developing countries. Through a web-based
“interactive platform” (web-forum), the Project intended to facilitate the sharing of best practices
and experiences and making IP tools and online training kits from WIPO available.
8.
Target beneficiaries included: Governments, universities and research institutions, the
private sector, international organizations, intergovernmental organizations, non-governmental
organizations, and independent individuals forming the network.
9.
The following outputs (deliverables) were planned:
(a) Output 1: A Taxonomy-Analytical Study with the aim of mapping, clustering,
analyzing and correlating different open collaborative initiatives and the respective IP
models they are based on;
(b) Output 2: An Open-ended Meeting with Member States for a constructive debate on
the essence, logic and stages of the approach of Open Collaborative Projects;
Daniel P. Keller (EvalCo GmbH/Sàrl, Evilard/Leubringen, Switzerland). The evaluator is independent and has not
been involved into the preparation or implementation of the Project.
2 Summarized from CDIP_6_6_rev., page 2
3 The delivery strategy is described in detail in section 2.3 of CDIP_6_6_rev. (pages 5 an 6)
1
CDIP/15/3
Annex, page 7
(c) Output 3: An experts meeting to exchange best practices on existing Open
Collaborative Projects;
(d) Output 4: An in-depth evaluation study establishing pros and cons of existing Open
Collaborative Projects by extracting inherent IP models for successful open collaborative
environments;
(e) Output 5: The establishment of an interactive platform for exchange of experiences
consisting of two components: a web site informing on the studies and proposing possible
IP tools, and a web forum for receiving feedback from/on experiences in Open
Collaborative Projects and IP-based models;
(f)
Output 6: The incorporation of resulting recommendations into relevant WIPO
programs after approval by Member States.
10. At the outcome level4, the Project aimed at contributing to the stimulation of local
innovation, in particular in developing countries. Project-specific impact objectives5 were not
defined.
11. According to the final progress report submitted to the CDIP6, which was updated and
validated through interviews in January 2015, Outputs 1 – 4 have been fully delivered, as it will
be described in more detail in section 2.C below (assessment of effectiveness). By the time of
the evaluation, the Project Manager was in the process of completing Output 5 (Interactive
Platform), while no work had been undertaken on Output 6. The reported budget utilization rate
per August 31, 2014 was 27%.
(B)
SCOPE, PURPOSE, METHODOLOGY AND LIMITATIONS OF THIS EVALUATION
(i)
Scope
12. According to the ToRs, the evaluation covered the period from January 1, 2011 to
August 31, 2014. Relevant subsequent developments were taken into account until the end of
the field mission in January 2015.
(ii)
Key purpose
13. The main purpose of this evaluation given by the ToRs was to assess whether the Project
as a whole provided the right type of support to achieve its key objectives in the right way, with
the main purpose to draw lessons learned for possible further WIPO activities. Future WIPO
activities might in particular refer to the originally envisaged incorporation of activities into
relevant WIPO programs (Output 6), which will require formal approval by the Member States.
14. Balancing the need for organizational learning with the purpose of ensuring accountability
of the Secretariat towards the Member States, the specific evaluation objectives were two-fold:
(a) To ensure learning from experiences during the Project’s implementation, what
worked well and what did not work well for the benefit of continuing activities in the field of
Open Collaborative Projects and IP-Based Models.
(b) To provide an evidence-based assessment to support the CDIP’s decision making
process.
Outcomes are results generated by the use of the project’s deliverables (outputs).
Impact refers to positive and negative, intended and non-intended, directly and indirectly, long term effects
produced by a development intervention.
6 The last implementation report available and presented to the Member States is dated August 28, 2014
(CDIP_14_2, Annex III).
4
5
CDIP/15/3
Annex, page 8
(iii)
Methodology
15. The evaluation framework is provided by WIPO’s Evaluation Policy7, which reflects the
key principles of the evaluation criteria and quality standards issued by the Development
Assistance Committee of the Organization for Economic Co-operation and Development
(OECD-DAC)8. The ToRs requested an assessment of project quality, including its design and
management. In line with the ToRs and standard evaluation practices, the assessment was
conducted based on the following five criteria9:
(a) Project preparation and management: Did project preparation and management
followed good practices, including applying the tools of results-based management?
(b) Relevance: The extent to which project objectives were consistent with beneficiaries’
requirements, the Member Countries’ needs, global priorities and policies.
(c) Efficiency: How efficiently inputs (e.g. funds, expertise, and time) were converted
into results. The evaluation mainly looked at the Project’s approach.
(d) Effectiveness: The extent to which objectives were achieved or are expected to be
achieved, taking into account their relative importance.
(e) Sustainability: The likelihood of continuation of project benefits after the assistance
has been completed.
16. The evaluation combined different evaluation tools to ensure an evidence-based
qualitative and quantitative assessment. A particular emphasis was placed on cross-validating
data and on assessing the plausibility of results obtained. The methodological mix included
desk studies, individual interviews and direct observation. The list of persons interviewed and
documents consulted are presented in Appendixes 3 and 4 to this report.
17. While maintaining his independence, the evaluator applied a participatory evaluation
approach by seeking the views of representatives of all key project stakeholders. The
evaluation process itself was designed in a way to facilitate organizational learning. The
evaluator attempted to enroll WIPO staff members into the process and where possible to
obtain their alignment on key findings, conclusions and recommendations.
18. The Secretariat supported the evaluation process actively. The evaluator was able to
work freely and without interference. Factual information obtained during data collection was
clear, comprehensive and consistent.
19. The presentation of the evaluation report at the 15th session of the CDIP in April 2014
aims at ensuring the dissemination of information, providing input to the CDIP’s decision making
process, and contributing to the accountability of WIPO towards its Member States.
(iv)
Main limitations to this evaluation
20. Experience shows that it takes time, before the content of studies and the conclusions of
conferences, through their use, translate into measurable effects. The planned “interactive
platform” as a tool to reach a wider public is not yet operational. It is not plausible to assume
sufficient causality between the Project’s outputs and changes observed. An attempt to assess
their outcomes of even broader impact would thus be premature.
7
WIPO, Revised Evaluation Policy, May 2010, in particular Annex 1 on evaluation criteria, which makes reference to
the DAC Criteria of evaluating development assistance.
8 DAC Guidelines and Reference Series, Quality Standards for Development Evaluation, OECD-DAC, OECD 2010.
9 The ToRs only requested an assessment of effectiveness and sustainability
CDIP/15/3
Annex, page 9
21. For the same reason, the evaluator focused in his fact finding on desk study and those
actors who were directly involved into activities, including the Secretariat, experts who drafted
the studies, and conference participants.
22. A consolidated financial report that links expenditures to budget lines and results for the
whole duration of the Project was not available. A detailed analysis of financial efficiency, which
would require information types of expenditures for each of the outputs, was thus not possible.
23. The findings and assessment in section 2 below should be read in considering that these
constraints necessarily limited the scope and depth of the evaluation.
2.
FINDINGS AND ASSESSMENT
This section presents the findings of the evaluation and provides an assessment of project
quality against the evaluation criteria.
(A)
PROJECT PREPARATION AND MANAGEMENT
(i)
Project preparation
24. The Secretariat did a good job in translating the relatively open DA Recommendation 36
into a clearly articulated delivery strategy and methodology, which outlines the following three
step approach:
(a) To map existing paradigmatic open collaborative initiatives and their relations with IP
models through a taxonomy-analytical study.
(b) To exchange of views and best practices from Member States and experts, an
evaluation study will evaluate pros and cons of existing projects and identify lessons
learned (two meetings).
(c) The creation of an “Interactive Platform on Open Collaborative Projects and IPBased Models” to allow for a broad exchange and dissemination of technical information
and experiences with all stakeholders.
(d)
25.
The integration of findings and conclusions into WIPO’s activities.
The sequencing of activities was appropriate and conducive to achieving objectives.
26. The Project was significantly over budgeted. The basis for the cost calculation is not
clear. Comparing the activities with the budget, it is rather obvious that it would never be
possible to disburse the entire budget without compromising on project efficiency. Over
budgeting blocks resources that could otherwise be spent for the benefits of Member States.
Furthermore, there is a risk that projects are “perpetuated” for the sole reason that resources
are still available. The evaluator received confirmation that the budgeting process within the
Secretariat has been improved in a way to prevent significant errors in financial forecasting.
(ii)
Use of project planning tools
27. The application of standard tools for result-based management was weak. The project
document does not include a logical framework, which are commonly used as a tool for project
planning, monitoring and evaluations of development assistance projects.
CDIP/15/3
Annex, page 10
28. Objectives at output level10 are reasonably clear, although some of the “outputs” defined
would rather be considered as “outcomes”. Output and outcome levels are partially mixed (e.g.
consensus of Member States is not an output). Objectives at outcome level11 were only
defined for the interactive platform. Without an in-depth analysis of the project document it is
not possible to clearly understand the intervention theory (causal chain), especially for those
who are not specialized in the field.
29. The project document does also not clearly outline the broader positive changes to which
the Project is expected to contribute (overall development objectives, impact level).12
30. Good practice in result-based planning and monitoring requires linking objectives at all
levels to indicators, which should be specific, measurable, relevant and time-bound (SMART).
Means of verification should be defined and budgeted for if data collection requires resources.
31. Most indicators defined in the project document do not fulfill one or several of these
requirements, for example:
(a) “(...) 60% positive feedback on the project process from participants (on the
evaluation questionnaires) (...)” as an indicator does not say anything on the achievement
of the output they intend to measure (open ended meeting with Member States).
(b) The generation of more open collaborative projects through the use of the platform
or increased demand for WIPO training (is not achievable within the duration of the
Project; there is an insufficient causal link between project outputs and training demand);
(c) Although theoretically measurable, retrieving the information needed to assess
some outcomes, such as the number of collaborative projects, would possibly require
extensive data collection (surveys), for which no resources were budgeted.
32. Risks and assumptions for each of the objectives were not identified at the planning stage
(with the exception of “Output 6”). Assumptions and risks should refer to external factors that
are relevant for translating outputs into outcomes and outcomes into impact13. In order to help
project managers to focus more closely on monitoring those risks that need to be controlled in
order to achieve objectives, they should also be categorized according to the likelihood they
materialize and the potential degree of negative impact. Identifying assumptions and risks
(including defining mitigation strategies) is also part of a sound project preparation.
33. Overall, the application of standard project planning tools at the design stage was weak.
As a result of this, reporting was mainly “activity based” rather than comparing planned with
achieved results using “SMART” indicators.
34. Ensuring a proper project planning and monitoring requires compulsory training for project
managers as well as a stringent quality control of project documents and progress reports.
10
Output: The products, capital goods and services, which result from a development intervention; may also include
changes resulting from the intervention, which are relevant to the achievement of outcomes (OECD, Glossary of Key
Terms in Evaluation and Results Based Management, 2010).
11 Outcome: The likely or achieved short-term and medium-term effects of an intervention’s outputs (OECD, Glossary
of Key Terms in Evaluation and Results Based Management, 2010).
12 Impact: Positive and negative, primary and secondary long-term effects produced by a development intervention,
directly or indirectly, intended or unintended (OECD, Glossary of Key Terms in Evaluation and Results Based
Management, 2010).
13 Hypotheses about factors or risks, which could affect the progress or success of a development intervention
(OECD, Glossary of Key Terms in Evaluation and Results Based Management, 2010)
CDIP/15/3
Annex, page 11
(iii)
Project Management
35. As discussed in more details under the assessment of effectiveness and efficiency below,
those activities that had been completed were well prepared and organized.
36. Project implementation in general however advanced at a slow pace. Even after the first
project extension, no measures were taken to accelerate implementation. In 2014, the
responsibility for the Project was, together with the Project Manager, relocated from the former
Innovation and Technology Transfer Sector14 to the Patents and Technology Sector (PCT
Business Development Division). Allocating the Project to a division with little expertise in the
field further weakened its technical supervision.
37. More than four years after the Project’s start, the finalization of its key output (interactive
platform) is still ongoing. The originally planned 30 months and even more the extended project
duration of 42 months (until June 2014) should have been sufficient to commission two studies,
organize two meetings in Geneva and to establish a web-based interactive platform. Unless
caused by the impact of unforeseeable factors beyond WIPO’s control, delaying project
implementation erodes the confidence of Member States into the Secretariat’s project
implementation capability.
38. Delays reduce project efficiency, because some of the running costs of projects are
generally fixed and incur regardless of whether any benefits are generated. Furthermore, slow
implementation may also cause a loss of project relevance and momentum, a problem that was
raised by several of the stakeholders interviewed.
(B)
RELEVANCE
Relevance assesses the extent to which project objectives were consistent with beneficiaries’
requirements, member countries’ needs, global priorities and WIPO’s policies.
(i)
Policy relevance
39. Policy relevance for Member States at the macro level is evidenced by the CDIP’s
approval of the Project through consensus.15
40. Recommendation 36 of the DA to which the Project aims at responding is located in
Cluster D (Assessment, Evaluation and Impact Studies). Stimulating innovation as the
expected project outcome is however equally relevant to many of the DA recommendations
located in Cluster C.
41. Project objectives were well aligned with WIPO’s Strategic Goals and Programs. Expected
results at the output level link primarily into the objectives of Program 16 (Strategic Goal V:
economic studies and statistics on the impact of IP). The enhanced use of practical tools for
open innovation was a specific objective of Program 18 under the Program and Budget for the
2010/2011 Biennium. The Program and Budget for the Biennium 2014/2015 still promotes the
application of open innovation structures to address Global Challenges (Program 18)16, while no
resources are allocated anymore under the current Program 16.
14
Now: SMEs and Entrepreneurship Division, Department for Transition and Developed Countries (TDC)
Comments of Member States to a first technical proposal prepared by the Secretariat were taken up and
integrated into the approved version of the Project document.
16 See paragraph 18.5: “(...) the core element of the Program’s strategy rests on the development, implementation
and sustainability of multi-stakeholder platforms that facilitate effective collaborative networks and technology transfer
relevant to health and climate change. Initiatives will build on partnerships and collaborations using open innovation
structures, networked innovation, and other forms of partnerships to accelerate their impact. (...)”
15
CDIP/15/3
Annex, page 12
42. The relevance of Open Collaborative Projects goes beyond their application in addressing
Global Challenges through open innovation. Their potential key benefit extends more broadly to
serving as a tool to enhance the valorization of intellectual assets for the benefit of
development, including but not limited to IP in the public domain. In this sense, the Project was
also relevant to the objectives of Strategic Goal III (facilitating the use of IP for development) of
the current Program and Budget.
43. Open Collaborative Projects offer opportunities for exchanging intellectual assets
worldwide on a voluntary basis in mutual benefit, regardless from a possible North-South divide.
This potentially benefits stakeholders from both the developing and the developed world.
44. Moreover, the model of Open Collaboration is applicable beyond what is commonly
understood as “innovation” and extends also to creative industries.
45. The sample of conference participants interviewed by the evaluator confirmed that their
expectations were met and that the studies presented were of use for them.
46. Many of them however also emphasized that the four initial activities of the Project were
only a first step into the right direction. Beyond the limited scope of the Project, more practical,
targeted support to fostering open innovation would be needed.
(C)
EFFECTIVENESS
47. Due to the limitations explained above, this section primarily compares planned against
expected results at the output level.
48. The delivery of the following main outputs reported by the Project were validated through
desk study and interviews:
(i)
Output 1: Taxonomy-Analytical Study
49. Based on a thorough review of existing relevant studies and literature on open innovation,
the Study17 mapped, clustered, analyzed and correlated different open collaborative initiatives
and the respective IP models they are based on.
50. The study, which was presented in the 8th session of the CDIP, also identified emerging
initial conditions and IP models for successful open collaborative initiatives.
51. Comments from Member States during and after the 8th session of the CDIP (see also
CDIP 8/9 page 26 - 27) were taken into account for a revised version.
(ii)
Output 2: Open-ended Meetings with Member States
52.
Two Open-ended Meetings with Member States were organized.
53. An informal side event of the 9th session of the CDIP was held on May 11, 2012 included
the following presentations:
(a) General Overview of the Taxonomy-Analytical Study for the Project on Open
Collaborative Projects and IP-Based Models (project);
17
Taxonomy-Analytical Study for the project on open collaborative projects and IP-based models (CDIP/8/INF/7),
October 31, 2011, commissioned by the Secretariat and prepared by Dr. Linus Dahlander, Assistant Professor,
Stanford University / European School of Management and Technology (ESMT), Berlin; Dr. David Gann, Professor
and Head of Innovation and Entrepreneurship, Imperial College Business School, London; and Dr. Gerard George,
Professor and Director, Rajiv Gandhi Centre, Imperial College Business School, London.
CDIP/15/3
Annex, page 13
(b) Roadmap for the Development Agenda Project on Open-Collaborative Projects and
IP-Based Models (Secretariat);
(c) Promoting Innovation and Access through Open Collaborative Models (Bolivia); the
Work of the WHO Consultative Expert Working Group on Research and Development and
their Recommendation for a Binding R&D Treaty (South Center, Geneva);
(d) Open Innovation Networks: Success Stories such as Ache in Brazil (MER,
University of Geneva);
(e) Discussion on the Essence, Logic and Approach of the Development Agenda
Project on Open-Collaborative Projects and IP-Based Models (Secretariat).
54. Moreover, a formal WIPO Meeting on June 18, 2012 was attended by about
20 participants with Delegates from 7 countries’ Permanent Missions to the United Nations
Office in Geneva as well as 3 NGOs. The meeting focused on obtaining feedback of Member
States on the Project and to agree on the way forward.
(iii)
In-depth evaluation study
55. The aim of commissioning an in-depth evaluation study was to identify pros and cons of
existing projects and identify the lessons learned for each open collaborative initiative.
56. The study18 illustrates with seven examples that open innovation is already very advanced
in the developing world. It identifies specific success factors for open innovation through a wellresearched taxonomy analysis of three specific projects. The report also provides specific
recommendations on how WIPO could broaden its service for organizations in developing
countries to incorporate open innovation.
(iv)
Expert meeting
57. In order to highlight the best practices in open collaborative projects for both public and
private organizations, a Global Conference was organized. According to the program, the
conference held on 22 and 23 January 2014 featured 17 speakers from developed and
developing countries as well as roundtable discussions on open innovation. 200 participants
attended.
58. The diversity of background and experiences of speakers, in particular the inclusion of
creative industries, which would normally not be associated with open collaborative projects,
broadened the perspectives of participants. On the other hand, some participants interviewed
would have wished a more in-depth discussion of fields directly relevant to them.
(v)
Interactive Platform
59. At the time of the evaluation, the platform was still in the process of being developed. The
evaluator received a presentation on the structure of the platform, which looked promising. It
was however premature to see how the final result might look like.
60. A study on “Global Knowledge Flows”19 for incorporation into the Interactive Platform was
prepared. Besides its comprehensiveness, a particularly remarkable feature of the report is an
18
In-Depth Evaluation Study for the Project on Open Collaborative Projects and IP-Based Models (CDIP 14/INF/14,
September 19, 2014), commissioned by the Secretariat, prepared by Prof. Ellen Enkel, Head of the Dr. Manfred
Bischoff Institute of Innovation Management of the Airbus Group, Chair of Innovation Management, Zeppelin
University, Friedrichshafen, Germany
19 Report on “Global Knowledge Flows” (CDIP 14/INF/13), commissioned by the Secretariat and prepared by
Community Systems Foundation, New York, USA.
CDIP/15/3
Annex, page 14
excellent visualization of knowledge flows through maps. It is planned to “animate” the maps on
the interactive platform and to update them regularly.
61. Keeping the interactive platform “alive” and up to date will require allocating specific
responsibilities and resources within the Secretariat.
(vi)
Initial outcomes observed
62. Apart from raising interest among the part of a relatively small audience that were not
already familiar with the topic, no particular positive or negatives outcomes were observed.
Before the interactive platform is operationalized, the reach of the Project remains limited.
Clearly, a follow-up will be needed in order to generate tangible effects.
(vii) Impact
63.
At this time, it was too early to assess results at the impact level.
(D)
EFFICIENCY
(i)
Financial implementation
64. Based on the last official financial report as per end of August, 2014 (CDIP 14/2 Annex III,
page 5), 27% of the total budget of 895,000 Swiss francs (including 161,000 Swiss francs for
personnel costs and 734,000 Swiss francs for non-personnel costs) was disbursed. Despite
repeated requests of the evaluator, a consolidated financial report that links expenditures to
budget lines for the whole duration of the Project was not available20. It is thus not possible to
analyze funds disbursed by category of costs and per output in order to assess financial
efficiency. Relating total costs to outputs delivered and a comparison with publically available
financial figures of other DA projects indicates that the Secretariat generally made economic
use of resources.
(ii)
Assessment of approach
65. For a first time activity in the field of Open Collaborative Projects, the Project’s approach,
discussed in more detail in section 2.A above proved to be appropriate in practice. Priorities at
this stage were rightly set on research, collection of good practices, validation of findings,
awareness raising and publicizing (not completed).
(iii)
Assessment of quality of outputs
66. An in-depth desk review complemented by expert interviews concluded that the three
studies produced under the Project are of good quality.
67. The overwhelmingly positive feed-back on the conferences and the expert meeting
evidenced by the Secretariat’s internal evaluation was validated through interviews of a sample
of conference participants.
68. Accordingly, the benefit of the conferences was to provide a platform for sharing of good
practices among a variety of stakeholders through presenting practical examples of different
models of open collaboration from a variety of industries. The de-politicized context of the
discussions and the fact that no attempt was made to advocate for a particular model were
mentioned as a particular benefit.
20
The evaluator received a report according budget lines and a report for the year 2014 per output, but not according
to budget lines.
CDIP/15/3
Annex, page 15
(iv)
Synergies with other activities conducted by the Secretariat
69. The Project complemented in particular different WIPO activities responding to different
DA recommendations under Cluster C (Technology Transfer, Information and Communication
Technologies and Access to Knowledge).
70. The use some studies conducted under DA_16_20_01 (CDIP/4/3)21 and DA_16_20_02
(CDIP/6/5)22 as an input to studies conducted under the Project resulted in some, although
rather limited, synergies.
71. Another example of seizing the opportunity for synergies among different DA-Projects is
the (planned) integration of the study results on “Global Knowledge Flows”23 (Output 5) into the
portal on Innovation and Technology Transfer Support Structure for National Institutions24.
72.
The evaluation found no overlaps with other activities of the Secretariat.
(E)
LIKELIHOOD OF SUSTAINABILITY OF RESULTS
73. It would be premature to attempt assessing the likelihood of sustainability of results.
Without any further activities, the initial results achieved are likely to be forfeited. The
interactive platform under establishment in particular will need regular updating and
maintenance, should it not loose its relevance within a fast evolving environment.
74. Incorporation of “resulting recommendations” into relevant WIPO Programs after approval
of the Member States was a project objective (“output” 6). Member States might have
deliberately chosen the wording “incorporation into relevant WIPO Programs”, since “Open
Collaborative Projects” as such are a tool and not a service that could be “mainstreamed”.
75. The wording of “output” 6 calls for a proposal of the Secretariat on how to promote Open
Collaboration and IP based models within WIPO’s existing Programs, possibly in the view of
preparing the Program and Budget for the Biennium 2016/2017. All stakeholders interviewed
highlighted the need for practical support (tools that provide guidance to companies, institutions,
inventors, etc. on how to enter into open collaboration) and to provide Member States with
advice on how to establish an enabling policy framework for open collaboration. Tools already
exist and there is no need to reinvent them. One option that might be worthwhile exploring is to
organize an “open competition” to develop the tool.
76. Discussions with stakeholders also highlighted the fact that many larger companies and
universities from developed countries have their own IP strategies, and are not familiar with the
way to enter into Open Collaborative Projects with actors in developing countries. Through
sharing the insight gained on models used in developing countries with them, WIPO could
provide a unique value added to foster new North-South partnerships.
77. It would exceed the scope of this evaluation to outline the way forward in detail. Looking
at existing WIPO Programs and consolidating different opinions of stakeholders interviewed,
Programs through which open collaboration might be promoted include the IP Academy
(Program 11), Small and Medium-Sized Enterprises (Program 30 including support to Member
States in drafting IP strategies), and also Program 14 under Strategic Goal IV (Services for
Access to Information and Knowledge), in particular through the Technology and Information
Support Centers (TISCs).
21
Project on intellectual property and the public domain
Project on patents and the public domain
23 Commissioned to an expert team from the Community Systems Foundation (CSF), cdip_14_inf_13
24 Established under a project for DA Recommendation 10 (Innovation and Technology Transfer Support Structure for
National Institutions) and the project on “Intellectual Property and Technology Transfer: Common Challenges –
Building Solutions” (DA_19_25_26_28_01, progress report in CDIP_14 Annex II).
22
CDIP/15/3
Annex, page 16
3.
CONCLUSIONS
78.
The findings and assessment above leads to the following conclusions:
Conclusion 1 on relevance: Strengthening the use of Open Collaborative Projects and IP
based models continues to be of high relevance
79. Open Collaborative Projects offer opportunities for exchanging intellectual assets
worldwide on a voluntary basis in mutual benefit, regardless of a possible North-South divide.
This is of high interest for both the developing and the developed world.
80. Through sharing the insight gained on models used, WIPO’s work provides a unique value
added to foster new North-South partnerships. The model of Open Collaboration is applicable
beyond what is commonly understood as “innovation” and extends also to creative industries.
Open Collaborative Projects are an effective tool to enhance the “valorization” of intellectual
assets, including but not limited to IP in the public domain.
81. In this sense, project objectives are highly relevant to facilitating the use of IP for
development, which is an international priority reflected in Strategic Goal III of the Program and
Budget for the Biennium 2014/2015. The conclusions of the Project are potentially relevant for
WIPO’s work in many fields, including promoting innovation, capacity building for IP service
providers, training, the cooperation with universities, and policy advice.
Conclusion 2 on project preparation and management: The Project addresses DA
Recommendation 36 in a creative and appropriate way. Activities were well organized.
The application of standard project planning and monitoring tools was however weak.
The project was significantly over budgeted and implemented at a slow pace.
82. The Secretariat translated the relatively open DA Recommendation 36 into a clearly
articulated delivery strategy and methodology. The type and planned sequencing of activities
was conducive to achieving objectives. Researching on the use of open collaborative models,
identifying good practices and subsequently offering a platform for discussing results and
sharing experiences among experts and other key stakeholders is an appropriate way to foster
the exchange of experience. Those activities that had been completed were well prepared and
organized.
83. The application of standard tools for result-based management in project planning and
monitoring was however weak. Progress reports list activities rather than comparing achieved
against planned results using specific, measurable, ambitious, relevant and time-bound
(“SMART”) indicators. Assumptions and risks were neither identified nor monitored. Due to a
lack of a realistic and detailed cost calculation, the Project was significantly over budgeted.
Inaccurate budgeting blocks resources, which would otherwise be available to generate benefits
for Member States. Furthermore, there is a risk that projects are “perpetuated” for the sole
reason that resources are still available. A budget and financial report presenting type of costs
for each objective is not available.
84. Project implementation advanced slowly. The originally planned 30 months should have
been sufficient to implement the limited number of activities. Even after the CDIP granted a
project extension, no measures were taken to accelerate implementation. Delays are costly and
cause a loss of momentum. If not explainable by the impact of unforeseeable external factors,
implementation delays erode the confidence of Member States into the Secretariat’s project
implementation capability.
CDIP/15/3
Annex, page 17
Conclusion 3: Outputs were of high quality, but not all of them were delivered. The
Secretariat generally made economic use of resources. Without a follow-up, initial
promising results are likely to be lost.
85. The Project did not reach all of its planned objectives. Through the delivery of four out of
six planned outputs, the Project contributed to the identification of successful Open
Collaborative Projects, in particular in developing countries, to deriving best practices from them
and to discussing the results within a limited audience.
86. The not yet completed Interactive Platform (Output 5) is pivotal for sharing good practices
with a wider audience and to make practical tools to support open innovation available.
87. Due to the delays in completing the platform, WIPO’s work in the field of Open
Collaboration risks to lose momentum. No steps have been taken towards an integration of
project results into WIPO’s existing Program activities.
88. The wording of “output” 6 indicates that Member States would expect a specific proposal
of the Secretariat. Without a follow-up, the Project’s reach remains limited and initial promising
results are likely to be lost.
4.
RECOMMENDATIONS
89.
From the conclusions above, the evaluation derives the following recommendations:
Recommendation 1 (from conclusion 3) to the WIPO Secretariat on finalizing the
interactive platform (planned Output 5 of the Project)
(a) Complete a beta version of the interactive platform
(b) Test-run the interactive platform and obtain feed-back from users
(c) Integrate the feed-back from users and
(d) Present a final version of the platform to the 17th session of the CDIP in November 2015
(e) Assign a clear responsibility and allocate resources to maintain and update the
interactive platform regularly.
Recommendation 2 (from conclusions 1 and 3) to the WIPO Secretariat on preparing a
proposal to the CDIP on how to facilitate Open Innovation through WIPO’s existing
Programs (planned Output 6 of the Project)
(a) Continuing identifying, collecting and sharing best practices in the field of Open
Collaborative Projects (studies)
(b) Offer practical capacity building (including tool kits) tailored to specific target users
(c) Offer capacity building to IP and/or innovation service providers in developing countries,
such as for example IP Offices, Technology Transfer Centers etc.
(d) Support specific open collaborative pilot projects in developing countries, and
(e) Advise Member States in creating an enabling environment for open collaboration in
their IP policies.
CDIP/15/3
Annex, page 18
Recommendation 3 (from conclusion 3) to the WIPO Secretariat on strengthening WIPO’s
presence in open collaboration conferences and fora:
90. A regular presence and visibility of WIPO in international conferences on Open Innovation
(including but not limited to events organized by other UN organization may help WIPO to
position itself as a “competence center” in the field of Open Collaborative Projects, to create
visibility and to benefit from additional experience of a wide array of conference participants.
Recommendation 4 (from conclusion 2) to the WIPO Secretariat on ensuring the
application of planning and monitoring tools in project cycle management:
91. The WIPO Secretariat should strengthen the quality control of new projects and progress
reports submitted to the CDIP in regards to proper application of WIPO’s existing project tools
for project cycle management are properly applied.
92. Consideration might be given to introduce the logical framework as a basis for project
cycle management.
93. Explore the possibility to introduce compulsory courses for future project managers on
project cycle management.
94.
Ensure regular coaching of project managers on a demand basis.
[Appendixes follow]
CDIP/15/3
Annex, page 19
LIST OF APPENDIXES
Appendix 1
Terms of reference
Appendix 2
List of persons interviewed
Appendix 3
List of documents25
25
The Project document CDIP/6/6 Rev. is available at:
http://www.wipo.int/meetings/en/doc_details.jsp?doc_id=149209
CDIP/15/3
APPENDIX I
APPENDIX I: TERMS OF REFERENCE
TERMS OF REFERENCE
Title of Assignment:
Project Evaluation: Project on Open
Collaborative Projects and IP Based Models
Name of unit/sector:
Development Agenda Coordination
Division (DACD)/ Development Sector
Place of Assignment:
Evaluator’s place of residence/duty
Expected places of travel (if applicable):
During your assignment, you will undertake
two missions to WIPO Headquarters,
Geneva, Switzerland (date to be determined)
Expected duration of assignment:
From December 15, 2014, to April 20, 2015
1.
Objective of the assignment
The present document represents the Terms of Reference (TOR) for the evaluation of the
Development Agenda Project on Open Collaborative Projects and IP Based Models, approved
during the sixth session of the Committee on Development and Intellectual Property (CDIP),
held in Geneva, in November 2010. The project document for this project is contained in
document CDIP/6/6 Rev. The project implementation started in January 2011 and was
completed in July 2014. The project consists of activities aimed at developing an Interactive
Platform for the exchange of experiences on open collaborative projects, such as the Human
Genome Project, in developed and developing countries, as well as intellectual property (IP)
models.
The project was implemented under the supervision of the Project Manager, Mr. Ali Jazairy,
Senior Counsellor, PCT International Cooperation Division.
This evaluation is intended to be a participative evaluation. It should provide for active
involvement in the evaluation process of those with a stake in the projects: project team,
partners, beneficiaries, and any other interested parties.
The main objective of this evaluation is two-fold:
1. Learning from experiences during project implementation: what worked well and
what did not work well for the benefit of continuing activities in this field. This includes
assessing the project design framework, project management, including monitoring and
reporting tools, as well as measuring and reporting on the results achieved to date and
assessing the likelihood of sustainability of results achieved; and
2.
providing evidence-based evaluative information to support the CDIP’s
CDIP/15/3
Appendix I, page 2
decision-making process.
In particular, the evaluation will assess the extent to which the project has been instrumental in:
(a) Establishing an interactive platform for the broadest possible exchange of
experiences on open collaborative projects and IP-based models; and
(b) Exchanging information, experiences and existing best practices and in enhancing
the understanding of potential uses of IP models/procedures in order to stimulate local
innovation.
To this end, the evaluation, in particular, will focus on assessing the following key evaluation
questions:
Project Design and Management:
(a) The appropriateness of the initial project document as a guide for project
implementation and assessment of results achieved;
(b) the project monitoring, self-evaluation and reporting tools and analysis of whether
they were useful and adequate to provide the project team and key stakeholders with
relevant information for decision-making purposes;
(c ) the extent to which other entities within the Secretariat have contributed and
enabled an effective and efficient project implementation;
(d) the extent to which the risks identified in the initial project document have
materialized or been mitigated; and
(e ) the project’s ability to respond to emerging trends, technologies and other external
forces.
Effectiveness:
(a) The usefulness of the project in establishing an Interactive Platform for the broadest
possible exchange of experiences on open collaborative projects and IP-based models.
(b) the effectiveness of the project in exchanging best practices for public ventures and
for private firms experiences; and
(c ) the effectiveness of the project in describing the IP models and procedures that
these open collaborative projects are based on and discuss the benefits and challenges
of each of them.
Sustainability
The likelihood for continued work on Open Collaborative Projects and IP-Based Models in
WIPO and its Member States.
Implementation of Development Agenda (DA) Recommendations
The extent to which the DA Recommendation 36 has been implemented through this project.
CDIP/15/3
Appendix I, page 3
In addition, the project time frame considered for this evaluation is 42 months (January 2011 –
July 2014). The focus shall not be on assessing individual activities but rather to evaluate the
project as a whole and its contribution in assessing the needs of Member States and identify the
resources or the means to address those needs, its evolution over time, its performance
including project design, project management, coordination, coherence, implementation and
results achieved.
In pursuance to the abovementioned objective, the evaluation methodology is aimed at
balancing the needs for learning and accountability. To this end, the evaluation should provide
for active involvement in the evaluation process of those with a stake in the project: project
team, senior managers, Member States and national intellectual property (IP) offices.
The external evaluation expert will be in charge of conducting the evaluation, in consultation
and collaboration with the project team and the Development Agenda Coordination Division
(DACD). The evaluation methodology will consist of the following:
(a) Desk review of relevant project related documentation including the project
framework (initial project document and study), progress reports, monitoring information,
mission reports and other relevant documents.
(b) interviews at the WIPO Secretariat (project team, other substantive entities
contributing to the project, etc.); and
(c)
2.
stakeholder interviews.
Deliverables/services
The evaluator will deliver:
(a) An inception report which contains a description of the evaluation methodology and
methodological approach; data collection tools (including eventual surveys of
beneficiaries and stakeholders); data analysis methods; key stakeholders to be
interviewed; additional evaluation questions; performance assessment criteria; and
evaluation work plan;
(b) draft evaluation report with actionable recommendations deriving from the findings
and conclusions;
(c) final evaluation report which includes an executive summary and structured as
follows:
(i)
description of the evaluation methodology used;
(ii) summary of key evidence-based findings centered on the key evaluation
questions;
(d)
(iii)
conclusions drawn based on the findings;
(iv)
recommendations emanating from the conclusions and lessons learned.
comprehensive executive summary of the final evaluation report.
This project evaluation is expected to start on December 15, 2014, and be finalized on April 20,
2015. The reporting language will be English.
CDIP/15/3
Appendix I, page 4
3.
Reporting
The Consultant will be under the supervision of the Director of the Development Agenda
Coordination Division (DACD). In addition, the evaluator shall:
(a) Work closely with the Development Agenda Coordination Division (DACD), the
Project Manager and the PCT International Cooperation Division. You shall also
coordinate with the relevant Program Managers in WIPO as required; and
(b) ensure the quality of data (validity, consistency and accuracy) throughout the
analytical reporting phases (inception report and final evaluation report).
4.
Profile
Mr. Daniel Keller has extensive experience in preparing, managing and evaluating projects, and
in conducting institutional assessments both in the public and private sectors. Mr. Keller also
has a previous experience with WIPO, he conducted evaluation reports on completed
Development Agenda Project, namely the Project on Enhancing South-South Cooperation on IP
and Development among Developing Countries and LDCs (document CDIP/7/6), and the
Project on Intellectual Property and Socio-Economic Development (document CDIP/5/7 Rev.)
5.
Duration of contract and payment
The contract will start on December 15, 2014 and will finish in April 20, 2015. During this
period, the following schedule should be followed:
The inception report should be submitted to WIPO by January 15, 2015. WIPO’s feedback shall
be communicated to you by January 20, 2015. The draft evaluation report shall be submitted to
WIPO by February 15, 2015. Factual corrections on the draft will be provided to you by
February 20, 2015. The final evaluation report shall be submitted by March 1, 2015. The final
version of the evaluation report containing a management response in an annex shall be
considered by the fifteenth session of the CDIP, to be held from April 20 to 24, 2015. You will
be required to present the evaluation report during that CDIP session.
[Appendix II follows]
CDIP/15/2
APPENDIX II
APPENDIX II: LIST OF PERSONS INTERVIEWED
No.
Name and function
1.
Mr. Ahmed Abdel Latif, International Centre for Trade and Sustainable
Development (ICTSD)
Mrs. Maya Katharina Bachner, Acting Director and Head, Program Management
2. and Performance Resource Planning, Program Management and Performance
Division, Administration and Management Sector
3. Mr. Irfan Baloch, Director, DACD, Development Sector
4.
Ms. Elen Enkel, Professor, Dr. Manfred Bischoff Institute for Innovation
Management, Airbus Group, Author of Evaluation Study
5.
Mr. Joseph Fometeu, Professor, University of Ngaoundéré, Cameroon (reference
person who did not participate in the Project)
6. Mr. George Ghandour, Senior Program Manager, DACD, Development Sector
7.
Mr. Gary Goldman, Community Systems Foundation, New York, NY, USA, coauthor, Report on “Global Knowledge Flows”
8.
Mr. Ali Jazairy, Senior Counsellor, PCT International Cooperation Division, Patents
and Technology Sector (Project Manager)
9. Ms. Sarah R. Lofti, writer-director and independent filmmaker
10.
Mr. Claus Matthes, Director, PCT Business Development Division, Patents and
Technology Sector
11.
Mr. Matthew Rainey, Director, SMEs and Entrepreneurship Support Division,
Department for Transition and Developed Countries (TCD)
12. Mr. John Sandage, Deputy Director General, Patents and Technology Sector
13.
Mr. Richard Wilder, Associate General Counsel, Bill and Melinda Gates
Foundation, Speaker of Expert Meeting
14.
Mr. Sacha Wunsch-Vincent, Senior Economic Officer, Economics Section,
Economics and Statistics Division
[Appendix III follows]
CDIP/15/2
APPENDIX III
APPENDIX III: LIST OF DOCUMENTS
Documents relating to monitoring & evaluation

Internal Audit and Oversight Division, Revised WIPO Evaluation Policy, May 2010

Internal Audit and Oversight Division, Evaluation and Inspection Section, Self-Evaluation
Guidelines, Version 1.1, April 2009

DAC Guidelines and Reference Series, Quality Standards for Development Evaluation,
OECD Development Assistance Committee (DAC), OECD 2010.

UNEG, Standards for Evaluation in the UN System, April 2005 (last updated on 18 June
2014)
WIPO programmatic documents

The 45 Adopted Recommendations under the WIPO Development Agenda by the General
Assembly of WIPO Member States, 2007

Medium Term Strategic Plan 2010-15: (Document A/48/3, September 16, 2010

WIPO Program and Budget for the Biennium 2010/2011, approved by the Assemblies of the
Member States of WIPO on October 1, 2009

WIPO Program and Budget for the Biennium 2012/2013, approved by the Assemblies of the
Member States of WIPO on September 29, 2011

WIPO Program and Budget for the Biennium 2014/2015, approved by the Assemblies of the
Member States of WIPO on December 12, 2013
Project documents and reports

Project Document (first version): Project proposal on “Open Collaborative Projects and IPBased Models”, CDIP/6/6, October 11, 2010

Project Document: Revised project proposal on “Open Collaborative Projects and IP-Based
Models”, CDIP/6/6 Rev, November 26, 2010

Progress report, CDIP 12/2 Annex VII (pages 59 – 68), including a revised timeline for
implementation, September 12, 2013

Latest progress report, CDIP 14/2 Annex III (pages 18 -26), August 28, 2014

Financial statements of the project (unofficial, detailing expenditures per output)

Participant feedback received on outputs 2 and 4 (questionnaires, not summarized) and
emails
Project Outputs:
Output 1: Taxonomy-Analytical Study

Taxonomy-Analytical Study for the project on open collaborative projects and IP-based
models (CDIP/8/INF/7), October 31, 2011, commissioned by the Secretariat and prepared
by Dr. Linus Dahlander, Assistant Professor, Stanford University / European School of
Management and Technology (ESMT), Berlin; Dr. David Gann, Professor and Head of
CDIP/15/3
Appendix III, page 2
Innovation and Entrepreneurship, Imperial College Business School, London; and Dr.
Gerard George, Professor and Director, Rajiv Gandhi Centre, Imperial College Business
School, London.

Revised Taxonomy-Analytical Study for the project on open collaborative projects and
IP-based models (CDIP/8/INF/7), April 23, 2013, idem
Output 2: Open-Ended Meeting with Member States

Program, Open-Ended Meeting with Member States in the Framework of the Development
Agenda Project on Open Collaborative Projects and IP-Based Models (Rec. 36) on June 18,
2012, including presentations on General Overview of the Taxonomy-Analytical Study for
the Project on Open Collaborative Projects and IP-Based Models (project); Roadmap for
the Development Agenda Project on Open-Collaborative Projects and IP-Based Models
(Secretariat); Promoting Innovation and Access through Open Collaborative Models
(Bolivia); the Work of the WHO Consultative Expert Working Group on Research and
Development and their Recommendation for a Binding R&D Treaty (South Center, Geneva);
Open Innovation Networks: Success Stories such as Ache in Brazil (MER, University of
Geneva); Discussion on the Essence, Logic and Approach of the Development Agenda
Project on Open-Collaborative Projects and IP-Based Models (Secretariat).
Output 3: In-Depth Evaluation Study

In-Depth Evaluation Study for the Project on Open Collaborative Projects and IP-Based
Models (CDIP 14/INF/14, September 19, 2014), commissioned by the Secretariat, prepared
by Prof. Ellen Enkel, Head of the Dr. Manfred Bischoff Institute of Innovation Management of
the Airbus Group, Chair of Innovation Management, Zeppelin University, Friedrichshafen,
Germany
Output 4: Expert Meeting

Meeting program, WIPO Conference on Open Innovation: Collaborative Projects and the
Future of Knowledge (meeting code: WIPO/INN/GE/14), January 22 – 23, 2014

News feature on WIPO Conference on Open Innovation: Collaborative Projects and the
Future of Knowledge http://www.wipo.int/pressroom/en/stories/collaborative_innovation.html

Article in IP Watch on WIPO Conference on Open Innovation: Collaborative Projects and the
Future of Knowledge, January 29, 2014 (retrieved on January 10, 2014 from:
http://www.ip-watch.org/2014/01/29/at-wipo-global-experts-share-experiences-on-opencollaboration
Output 5: Interactive Platform (platform content)

Report on “Global Knowledge Flows” (CDIP 14/INF/13), commissioned by the Secretariat
and prepared by Mr. Harry Goldman, Mr. Kris Oswalt, Ms. Adriana Valdez Young, Ms.
Becky Band Jain, Mr. Alexandre Toureh, Mr. John Toner, Mr. Gaurav Sharma, Ms. Irene
Inouye, Ms. Haya Shaat, Mr. Jitesh Dhoot and Mr. Vikesh Ojh, Community Systems
Foundation, New York, USA.
http://www.wipo.int/edocs/mdocs/mdocs/en/cdip_14/cdip_14_inf_13.pdf

Draft version of Interactive Platform (screen shots) of http://wwwocmstest.wipo.int/innovation [not publically accessible at the time of the evaluation]
[End of Appendix III and of document]