For Official Use
DSTI/EAS/STP/NESTI(2016)10
Organisation de Coopération et de Développement Économiques
Organisation for Economic Co-operation and Development
___________________________________________________________________________________________
English - Or. English
DIRECTORATE FOR SCIENCE, TECHNOLOGY AND INNOVATION
COMMITTEE FOR SCIENTIFIC AND TECHNOLOGICAL POLICY
DSTI/EAS/STP/NESTI(2016)10
For Official Use
Working Party of National Experts on Science and Technology Indicators
OECD BLUE SKY 2016 - TOWARDS THE NEXT GENERATION OF DATA AND INDICATORS
MAIN ISSUES RAISED AND POSSIBLE IMPLICATIONS FOR OECD
24-25 October 2016, Paris, OECD Headquarters
For further information, please contact: Economic Analysis and Statistics Division, STI, OECD.
Alessandra COLECCHIA, [email protected];
Fernando Galindo-Rueda, [email protected]
English - Or. English
Complete document available on OLIS in its original format
This document and any map included herein are without prejudice to the status of or sovereignty over any territory, to the delimitation of
international frontiers and boundaries and to the name of any territory, city or area.
DSTI/EAS/STP/NESTI(2016)10
OECD BLUE SKY 2016 - TOWARDS THE NEXT GENERATION OF DATA AND INDICATORS
MAIN ISSUES RAISED AND POSSIBLE IMPLICATIONS FOR OECD
1. Background and objectives
1.
Every 10 years, the OECD Blue Sky Forum engages the policy community, data users and
providers into an open dialogue to review and develop its long-term agenda on science, technology and
innovation (STI) data and indicators. This event is known as the “OECD Blue Sky Forum”, reflecting its
intention to provide an open and unconstrained discussion on evidence gaps in science and innovation and
on initiatives the international community can take to formulate and address data needs in this area.
2.
Blue Sky has previously been held in Paris (1996) and Ottawa (2006). On its last edition, Blue
Sky was marked by the first announcement of a Science for Science and Innovation Policy initiative, a
model that has become widely adopted in several countries. It also launched OECD work on innovation in
firms which exploited the potential of micro-data and informed the 2010 OECD Innovation Strategy with
the publication of Measuring Innovation: A New Perspective.
3.
The 2016 OECD Blue Sky Forum was incorporated in the 2015-16 Programme of Work and
Budget of the CSTP, under the responsibility of NESTI. Blue Sky follows the mandate set out by OECD
Science ministers, who last met in Daejeon, Korea, in October 2015. In their joint declaration, ministers
agreed to invite the OECD to use this unique Forum as a means to “continue improving statistics and
measurement systems to better capture the key features of science, technology and innovation”.
4.
Blue Sky 2016 took place on 19-21 September in the City of Ghent, Belgium, with support from
the Belgian Federal and the Flemish Regional Governments, and the City and the University of Ghent,
which kindly hosted the event in its Het Pand facilities. The main objectives for were set out as follows:
•
Discuss and review the main conceptual underpinnings of current frameworks for STI indicators
and data infrastructure initiatives, as well as their uses by the policy and the research
communities.
•
Explore the role of digital infrastructures in creating new opportunities for measurement and
analysis, as well as challenges to existing standards of collection and quality of STI indicators.
•
Provide new opportunities for collaboration and strengthen the dialogue between: policy makers,
data users and providers; national and global practices on indicators; efforts to build up and
maintain underlying data resources and efforts to develop indicators; official statisticians and
other practitioners; and STI data practitioners and practitioners in related statistical domains.
•
Lead to a forward-looking and policy relevant roadmap on STI measurement for OECD to
consider and implement with its membership, other international organisations and experts.
5.
In order to achieve these objectives, the Blue Sky forum programme was constructed in a
participatory fashion, with an open and well-publicised call for papers that welcomed original submissions
showcasing examples of data and indicator development with the potential for international adoption, as
2
DSTI/EAS/STP/NESTI(2016)10
well as examples of ground-breaking application of existing or new sources addressing questions and
providing evidence on the state of science and innovation systems and the role of STI policies
worldwide. The call encouraged not only academic contributions with policy relevance, but also concept
papers outlining possible strategies for STI data collection, measurement and quality improvements,
formulate user needs or specify potential initiatives by policy makers and administrators that can support
the infrastructure for the analysis of science and innovation phenomena.
6.
At nearly 400 participants from 48 different countries 1, the venue was at full capacity resulting in
not being possible to accommodate some last minute requests to participate. This was by far the most
intensely and diversely attended Blue Sky forum held to date, with very distinct communities coming
together for the first time under a same roof. Blue Sky brought together separate initiatives aimed at
pushing the boundaries of science, technology and innovation measurement through engagement with a
variety of research communities, working parties, task forces and expert groups that were formally or
informally represented at the Forum. The policy community was represented at a high level by Elke Sleurs
(State Secretary for Poverty Reduction, Equal Opportunities, People with Disabilities, Urban Policy and
Science Policy, Belgium), Kirsty Duncan (Science Minister, Canada); Manuel Heitor (Minister for
Science, Technology and Higher Education, Portugal), Carlos Moedas (the EU Commissioner for Research
and Innovation), and Mari Kiviniemi (OECD Deputy Secretary General).
7.
A scientific committee comprising world leading experts assisted the Blue Sky steering group 2
and the OECD secretariat in the process of screening nearly 200 proposals. 54 posters were on display and
74 papers were presented, principally in the parallel sessions (see Annex 2 for the main themes discussed
there). The plenary panels covered major issues of potential common interest to the broad set of
participants, engaging some of the top submissions as well as inviting leading experts. The keynotes
lectures provided by Luc Soete (Maastricht University) and Scott Stern (Massachusetts Institute of
Technology) and the plenary panels focused on the user dimensions of STI data and indicators (Science
and Innovation policy-making today: what big questions are begging for an answer?, Scope and limits of
indicator use by STI policy, Towards more inclusive science and innovation indicators and Science and
innovation policy-making in an era of Big Data), followed discussions on the challenges and opportunities
on the production of data and statistical evidence (New models and tools for measuring science and
innovation impacts, New data and frontier tools: the challenge for official statistics in science and
innovation, Looking forward: what data infrastructures and partnerships?).
2. Selected key messages arising from Blue Sky 2016 discussions
8.
While recognising the challenge of condensing in a short document all messages conveyed
during the nearly three days of discussions, panels, presentations and posters, this paper provides an
attempt to identify some recurrent themes that participants appeared to recognise as providing the main
threads of the debate.
1.
Approximately 40% of Blue Sky participants were based outside Europe, 12% in non OECD countries.
Nearly 40% of participants were women, although this percentage was smaller at nearly 30% for
presenters. The all-female opening panel was highlighted by several participants and also twitted on as a
positive development towards gender equality in science and innovation. The distribution of participants by
affiliation shows an even distribution of academics, government and international organisation officials and
staff from statistical agencies. Non-profit organisations and business community were also participating
although less well represented.
2.
The steering group comprised Svein Olav Nås (NESTI Chair), Ward Ziarko (Vice Chair of CSTP and
NESTI) and Pierre Verdoodt (NESTI delegate) both sponsors of the Forum and persons in charge of the
local organising team, and Luis Sanz Menéndez (Chair of CSTP at the time in which Blue Sky preparations
commenced).
3
DSTI/EAS/STP/NESTI(2016)10
Most of the issues and solutions identified at Blue Sky 2006 remain relevant today and going into the
future, but the landscape in which data on STI are produced and used has experienced a major
transformation that raises new challenges and opportunities.
9.
A major global financial crisis and a significant and protracted process of budget consolidation
across countries stand in between the 2006 and 2016 Blue Sky fora. This has contributed to shifting the
nature of policy questions from STI data towards better making the case for science, asking for the
generation of more comprehensive evidence on the impacts of investments in STI. Less timely, structural
indicators that prove useful insights in some contexts became less relevant in the presence of massive
shocks (e.g. financial, political). Furthermore, compared to Blue Sky 2006, there appear to be fewer
resources available within governments and agencies to maintain, let alone develop, new public data
infrastructures on STI, while there appears to be no shortage of interest in using available data.
10.
It was possible to note that some challenges, evidence gaps and opportunities had already been
identified in Blue Sky 2006, and while significant progress had been made on most of them, there was still
plenty to be achieved. Some areas identified at the time in 2006 are now in the mainstream - such as the
use and access to survey microdata and administrative data, and the measurement of key policy
instruments for innovation. Some have instead not fructified on a global basis as expected over the past
10 years, sometimes because it was not possible to reach a critical mass of countries interested in
consolidating specific instruments. In many respects, the approach to STI statistics has remained
largely national while the phenomena that are being analysed are increasingly global.
11.
In contrast, some areas have experienced such rapid developments that it is difficult to keep
track of existing initiatives. The transformational potential of digitalisation is being felt in all
dimensions of data production and use. “Data” was described as the source of one of the defining changes
experienced by our societies. The data revolution has reached the domain of science, technology and
innovation. This has facilitated the emergence of new STI evidence communities and new actors which
have brought considerable dynamism to the field. However, as it was already noted in 2006, a lot of work
is being carried out in silos which are still proving difficult to connect and leverage of their respective
capabilities.
There is a pressing need to place individuals and communities at the centre of policy design, and at the
centre of gathering and using evidence on science and innovation systems. Participatory processes in
the generation and use of STI data and indicators can be key to their sustainability and impact.
12.
At the Forum, Stephen Curry pointed out the importance of reminding ourselves about what
people really value. Several speakers stressed the importance of accounting for the "human" factor in
explaining STI phenomena of interest, such as inter-sectoral and international mobility (e.g. a scientist
decision to go back to her/his home country or decide not to pursue further a placement in industry) or
user-driven innovation (the motivation to develop a new solution to a particular community problem).
These factors play alongside economic and professional development incentives and frequently interact
with each other. The need for "human centred policy design", advocated by Minister Heitor, calls for the
systematic collection of data on placement outcomes, mobility and migration of highly trained individuals,
as pointed out by Paula Stephan. Also Sadao Nagaoka stressed the importance of directly surveying and
collecting information around individuals, especially when tracing scientific knowledge flows and their
impacts.
13.
The opening debate on measurement gaps, and in particular the remarks of Minister Heitor, have
also underscored the need to better characterise participatory processes of R&D and innovation policy
agenda setting to help engaging scientific institutions and actors with civil society, arguing there is a need
for collaboration with scientists, engineers and users to understand the knowledge production process and
4
DSTI/EAS/STP/NESTI(2016)10
its impact. Policy makers and researchers alluded to the importance of increasing citizen participation in
science and public support for science, but several also pointed to the need to frame the discussion in terms
of society’s engagement in innovation systems and preparedness for the changes brought about by
innovations, which also represents a potential object of measurement of relevance to policy.
The STI evidence community needs to address the still significant disconnect between users and
producers of STI data, statistics and analysis.
14.
The communication of complex messages and its translation for policy-making represents
significant challenges. It is understood that there is a demand for simple indicators to monitor and
benchmark STI systems that speak directly to the evidence needs of policy makers. The role of policy
makers was described as being essentially about managing uncertainty, requiring an "options thinking
approach" in which data plays a key role but so do narratives. Maryann Feldman underlined the power of
"story telling" in communicating research results to policy makers. Beyond metrics or narratives tailored
for policy makers, Ismael Rafols advocated the development of data toolkits that allow exploration of
choices in landscapes and allow citizens’ participation in decision making.
15.
Participating ministers welcomed the opportunity provided by Blue Sky to pause and reflect, and
some even recognised being struck by how much science and innovation can be driven by targets and
rankings - admitting the need to paying greater attention, instead, to the identification of insightful
indicators on science and innovation actors and their linkages, as well as to the infrastructure that enables
this to happen. Discussions showed the tension between evidence-based policy making and policybased or driven evidence with several allusions to politics in the so-called “post-truth” age. Participants
appeared to agree with Wolfgang Polt that indicator-driven policy is not tantamount to evidence-based
policy.
16.
Overall, most discussants warned about potential abuses of STI indicators that oversimplify
reality on the sole basis of what can be easily measured, and that obfuscate their interpretation. One
example noted related to the frequent interpretation of indicators as implying that higher values or rank
positions are necessarily better. The importance of providing some sound conceptual underpinning to what
indicators - especially composites - actually measure was raised. Jonathan Haskel pointed to the need to
conduct work on how to integrate and interpret indicators of innovation within established measures of
productivity and wellbeing, for example by drawing on methods for extending the measurement of
multifactor productivity with estimates of knowledge-based capital services. Mixing input and output
measures without a conceptual basis was advised against by many speakers, in particular Charles Edquist.
The solution proposed to the side effects of what Stephen Curry described as the “seductive power of
numbers” pointed in the direction of “educating the patient”, i.e. the users of STI data and indicators, and
to exchange best practices, e.g. those that indicate adequate benchmarking, uncertainty, heterogeneity,
etc..., supported by relevant training. The role of policy advisers, extensively represented at the Blue Sky
Forum, was considered to be very important in connecting data production and research into policy
making.
17.
Policy makers were also asked to consider the data dimensions within their policy planning and
evaluation. For example, policy makers were asked to impose the integration of data and evaluation
requirements into science and innovation support programmes, arguing that there are several instances
- such as when public funds are disbursed - in which public interest and accountability of policy actions
may override some privacy elements. As Adam Jaffe put it "we need to build a culture for evaluating
everything".
5
DSTI/EAS/STP/NESTI(2016)10
More than ever, users want to know where innovation comes from, and what it leads to.
18.
A recurrent theme throughout the Blue Sky 2016 Forum was the identification of the sources of
innovation and how different actors contribute to it over time. Innovation statistics have to take into
account the complementarities of efforts, the complex systems perspective, the flows of knowledge and the
time lags before actual impacts can be identified. Several speakers pointed out the difficulties in ensuring
that investments in R&D lead to economic and social outcomes of interest to national or local policy
makers in complex and open systems, calling for the missing elements to be identified and measured in a
more objective fashion. Blue Sky Forum participants were encouraged by Luc Soete to think beyond
current tools and data sources, and consider for example exploring the transformative potential of
BlockChain technology in science and innovation systems. Scott Stern provided examples of leveraging on
digitisation to deliver new metrics and underscored the need to develop "granular" capabilities to be able to
uncover the dynamics of innovation.
19.
The attribution of knowledge-based value creation and use was also present in discussions on
the contribution of science and innovation to productivity and societal welfare, and the need to populate
and parameterise models with evidence as opposed to untested assumptions. Theories and measurement of
value contribution exhibit some inherent limits - akin to what Giovanni Dosi described as finding the
marginal impact of an extra gram of butter to the taste of the cake. These are however at the heart of major
policy discussions around what part of the innovation system require public support, how to make growth
more inclusive, how to respond or prepare for technological change, and how to ensure the sustainability of
the nation state and its ability to fund its activities in today’s world without hampering the innovative
efforts that are required to bring up solutions to current challenges. In EU Commissioner Moedas' words
"If we are serious about the growth enhancing and job creating role of research and innovation, then we
need to be able to demonstrate and prove those".
Science and innovation statistics roadmaps need to take into account the entire value chain of STI data
generation and use for policy or other purposes.
20.
Blue Sky 2016 broadly contributed to de-emphasize the focus on indicators and extend our
outlook to the entire data value chain, taking into account interdependencies that span full data cycle and
take into account its reuse in different settings, which may include purposes and applications which were
not initially intended. Data can and do play different roles from feeding into agenda setting, policy design
and implementation and policy evaluation. The business case for data can be more easily articulated if all
those uses can be examined in a holistic way. In this spirit, Katy Börner and other speakers argued in
favour of going beyond the use of STI indicators as pointers to undertake a more systematic use of data for
modelling and predicting the evolution of entire STI systems. Data and indicators can be integrated into
established modelling tools that are already used to support decision making, for example in models of the
economy.
21.
A key issue highlighted by some speakers is the challenge of aligning STI measurement and
impact analysis practices to the core features of science and innovation processes - e.g. impacts arise
in the real world out of changes in organisational practices, complementary investments, are highly
diffused across space, actors and time. Breaking the dependence on what can be easily and readily
measured was pointed out by Koen Debackere as a major priority. . Likewise, policy impacts can take a
long time to materialise leaving the ex-post evaluation of final outcomes out of sync with policy decisions.
It was also noted that measurement frameworks have to evolve to account for changes in research and
innovation paradigms. For example, it was noted by Erkki Ormala that firms no longer manage innovations
but networks and platforms, raising the question of what are the appropriate tools to monitor innovation in
firms and account for connections.
6
DSTI/EAS/STP/NESTI(2016)10
22.
Blue Sky 2016 participants benefited from an open and transparent discussion of the potential
and the limitations of alternative metrics to identify the broader impacts of scientific activities. While
new models and tools, especially those drawing on social media and “big data” sources, offer great
promises and attract a great deal of attention, some participants warned against the hype and asked users to
reflect on what they actually measure beyond the hype. As Cassidy Sugimoto put it "altmetrics have not
been the panacea that we hoped for and they do not measure social impacts". Some also pointed to the
need to continue emphasising some old ideas, for example, instilling the importance of being serious about
evaluation and making a commitment to policy learning through experimentation.
23.
Qualitative information is increasingly becoming a source of quantitative evidence. Text
mining tools, e.g. natural language processing through inductive or deductive methods, underscore the
potential to alleviate some of the common challenges facing STI statistics, e.g. survey fatigue and unfitfor-purpose classification systems that are applied differently by human coders, and offer opportunities for
generating adaptable indicators. Effective application of these methods relies on fit-for-purpose, high
quality systems to collect qualitative information in a consistent fashion in the first place and avoid
potential manipulation. Managers of administrative databases become important data quality gatekeepers,
but incentives among those who provide the information still matter.
24.
The exchange of views between traditional users and producers of STI statistics and new actors
in the area of Research Information System (RIS) and repositories helped identify the importance of
understanding and managing the distinction and relationship between data “about” research and
innovation and data “from” and “for” research and innovation. The policy agenda on open science is
for example primarily focused on the latter but requires the standards, tools and infrastructures that can
also benefit the former, while the analysis of open science can help future policy decisions in this area.
Open science systems, in order to be sustainable, need appropriate attribution and accreditation tools that
reflect the actual use of the data as well as what enables the data to exist. The identification of tools and
standards that allow for this will have a major impact on the next generation of STI indicators.
For data infrastructures to fulfil their potential, it is necessary to build capabilities and encourage
coordination among different actors.
25.
As Scott Stern put it, the STI data and indicators community exists to create data and metrics to
gain shared understanding, evaluate policy alternatives and identify gaps. It was acknowledged that
the range of questions being asked requires a granular and linkable capability to capture the dynamics of
science and innovation, which can adapt to newly emerging sources of data which at this point we do not
know where they will come from. Blue Sky 2016 represented a major call to leverage on and influence
the digitalisation process to upgrade existing and deliver new infrastructures and measurement
systems to deliver what is expected of them.
26.
Participation incentives have to be an integral part of data infrastructure build up. This applies
for example in the context of survey participation, thinking about the benefits to respondents, and from the
perspective of data sharing into repositories, adoption of standards, etc... Identifiers can only work if the
subjects find it convenient to use them systematically. Laurel Haak argued that building infrastructures that
transform evidence capabilities require solutions that are only 20% technological. The remaining 80%
entails a process of social change that requires constant community engagement and the cumulative
development of trust. This message appeared to resonate across highly dissimilar groups operating in
different settings.
27.
Driving progress requires a careful process to identify which are the major obstacles to
developing infrastructures. This often starts within public administrations, where data are fragmented and
synergies are left underexploited. Lack of policy awareness about the blocking factors and their
7
DSTI/EAS/STP/NESTI(2016)10
implications can result in insufficient steps being made to assess the legal framework for data exchange
and re-use and to identify sustainable “business models” for data. The Panel on Data Infrastructures
identified as major issues: the sharing of data so that it is reusable; the need to directly involve those
individuals about whom data are being collected; have open standards for persistent identifiers in datasets individuals, institutions, projects - and especially ensure policy consistency and continuity over time. There
was an open call to communities to come together to develop the common infrastructure. At the same time,
it was noted that it was neither feasible nor desirable to aim to develop a methodology to account for
everything, suggesting instead prioritising the search for a common core, as content agnostic as possible,
through which gaps can be identified and on top of which topics and new applications can be added over
time as feasible and as required.
National statistical organisations (NSOs) are experiencing major disruptions but have unique assets and
capabilities they can leverage to deliver contributions to the STI data infrastructure that no other
organisations can match at present.
28.
Many researchers viewed the role of official statistical operations as concerned with
constructing the basic data infrastructure and making it linkable and usable. This represents a change with
respect to the traditional model entirely focused on the release of aggregate statistics. Many NSOs have
taken on seriously this additional use of their data and dedicated resources to create safe spaces where data
can be linked and analysed in a confidential way by researchers. Many NSOs have also identified ways to
use the full potential of microdata to serve their core business needs, but there are still too many STI
relevant databases that languish without being used. In the case of experimental STI databases, this has no
doubt contributed to a loss of interest and a discontinuation of otherwise promising developments in need
of further national and global consolidation. Researchers on the other hand need to understand better how
NSOs operate and be able to present their research proposals in ways that also deliver operational benefits
to NSOs who may otherwise struggle to support access infrastructures. Some participants at Blue Sky 2016
have demonstrated how research-NSO partnerships can be built over time.
29.
Awareness is rising with NSOs about the use new data sources and tools to deliver their core
objectives, for example web-scraping is being used to identify “rare” populations - several other examples
were shared at Blue Sky. The approach seems to be very much one of learning by doing to address the
technical and cultural barriers. Process and product innovation within NSOs are constrained by day-to-day
operational requirements that may be end up absorbing all resources and leaving little room for adapting
business models to changing demands and new opportunities. NSOs are also too often bound within the
national scope of their activities when it comes to addressing inherently global phenomena. In order to
pursuit their mission, the exchange of information and coordination of activities is bound to be essential.
30.
Notwithstanding these difficulties, there are no other entities with the mandate and formal
responsibilities that can substitute for some of the data infrastructure potential of NSOs. NSOs are
uniquely placed to assess the reliability of new data sources and methods, and to conduct representative
statistical surveys. They can provide the clearing houses asked for by researchers that combine information
sources in ways that other organisations cannot. It is therefore relevant for research funding organisations
supporting work on science and innovation to take into account the role of NSOs.
3. Potential implications for OECD work on STI data and indicators
31.
Blue Sky participants were encouraged from the start, and at all levels, to use this opportunity to
share what they thought the OECD could contribute to the STI evidence agenda. Kaye Husbands Fealing
highlighted that there cannot be development of Blue Sky indicators without a framework of where to put
them, especially a data quality framework. Jonathan Haskel stressed the importance of having a framework
8
DSTI/EAS/STP/NESTI(2016)10
to be able to have a conversation, pointing to the intangible assets framework as a way to frame the
discussion on broader innovation with finance ministers and all those who have the budget to invest.
32.
While the international community continues to work on some of the questions that were already
raised at the time of Blue Sky 2006, a lot of progress has been made in measurement thanks to the
exploitation of micro-data and micro-data linking of inputs and outputs, recalled Svein Olaf Nås, and
thanks to the measurement of investment in innovation assets within the National Accounts framework,
reminded Jonathan Haskel.
33.
To achieve a real impact on our society, said EU Commissioner Moedas, over the next ten years
we should strive to maximise the value of the massive quantity of data available to us. New data – big data,
web data and open data – data combinations and interactive mapping and reporting tools were discussed at
the Forum, with participants highlighting both the exciting opportunities and the challenges they entail the “uncomfortable data” as Jeffrey Alexander referred to. Participants seemed to agree that these new
tools and data sources need to be looked at as “toolkits” rather than as providing “silver bullets” answers
for policy makers. Looking ahead, the measurement community will need to make use all types of data
sources and methods to meet objectives. Hence the community will need to embark in partnerships and all,
including the OECD, will need to leverage on "the blended community of practice" as Kaye Husband
Fealing put it.
34.
The work on STI-relevant data and indicators at OECD spans several committees, working
parties and directorates which bring in unique insights and competences. What follows is an attempt to
identify some of those messages, which started with Minister Manuel Heitor’s introductory keynote
remarks and supporting paper. These offered a series of reflections concerning the historical and future role
of the OECD, pointing to its work on defining standards, compiling statistical information, building a
global infrastructure and instructing and helping data users worldwide.
OECD as interlocutor across communities sharing an interest in data on STI
• OECD should continue to play an interlocutor role between the various communities with an interest
in evidence and intelligence on science, technology and innovation, including not only academia and
policy, but also operational delivery agencies ranging from NSOs, funding agencies, as well
universities, public research organisations, private data providers and non-profit organisations working
in this area.
• A decennial forum like Blue Sky provides a significant degree of perspective and a clear milestone for
assessing improvement, but more regular interactions may be needed in areas where possibilities
evolve at a “neck-breaking” speed. Targeted symposia, working papers, briefs, under the Blue Sky
“brand” may contribute to developing participatory data futures and gaps analysis and data and
evidence development roadmaps for the international community producing and using evidence on
science technology and innovation.
OECD promoting action to foster international policy co-ordination for better and better use of STI
evidence
• OECD could contribute to national efforts to develop an evidence culture among the international
science and innovation policy making community, identifying and disseminating best practices in
evidence use for policy making, recognising the value of data and statistics at multiple levels of
decision making, and facilitating the dissemination of best practices on how to leverage on the
digitalisation of STI to foster evidence-based policy making and implementation.
9
DSTI/EAS/STP/NESTI(2016)10
• In this vein, OECD could recommend national governments to empower national statistical offices to
secure access to and use of relevant commercial and administrative data on science, technology and
innovation for statistical purposes in areas of significant relevance for the public interest - without
undermining the legitimate private interests or the incentives to invest in data infrastructures to
generate new value propositions. The OECD Council has recently adopted a Recommendation on
Good Statistical Practice, which refers, among other things, to ensuring NSO access to administrative
data for official statistics and points as an example of good practice the existence of provisions to
allow external user access to micro-data for statistical research purposes under strict protocols and
only after anonymisation of the data. The implementation of the recommendation for the area of
science, technology and innovation falls clearly within the aims of Blue Sky.
• In light of the apparent benefits of making administrative data on science and research funding more
openly available not only to benefit statistical evidence but also to governance of science and
innovation systems, OECD could work to develop a recommendation for OECD governments to
make data on government public funding of STI activities publicly accessible.
• OECD could work with national governments to identify or contribute to the development of
relevant, internationally multi-operable and scalable administrative data standards for coordinated adoption. Those standards could serve a diversity of needs, among which supporting
evidence for policy making and evaluation. To do so, OECD should develop the competences required
to engage its committees in relevant discussions about standards and practices for administrative data
on science and innovation in areas where coordinated international policy action can be of widespread
societal benefit, engaging with external stakeholders and existing initiatives as appropriate.
• OECD may consider providing hands-on guidance, e.g. using videos and possibly short MOOCs, on
the use of its STI data and indicators, aimed at a potentially broad audience.
OECD requested to extend its work across different thematic areas, strengthening and contributing
to developing the new generation data infrastructures where it is uniquely placed to do so
• Articulate the call for a higher attention to the role of individuals into suitable measurement
initiatives. Key gaps identified related to the characterisation of professional practice-based research,
the placement of graduates in firms and their mobility as part of measuring the contribution of higher
education to society, and the role of incentives and skills. Any efforts should take into account the
lessons learned from the experiences implemented in this area since Blue Sky 2006.
•
Extend its overall framework for conceptualising and measuring innovation beyond business and
to consider what practices may be recommended or would need to be tested for collecting data on the
role of individuals, public sector organisations, and non-profits.
•
Given the increasing importance of microdata-based analysis for international comparative policy
analysis, OECD could be asked to consider options for building up international secure
infrastructures and securing institutional agreements that facilitate the linking and analysis of
microdata sources in order to support demand to extend and deepen its ongoing programme of
distributed analysis projects.
•
OECD could work to develop, in collaboration with national authorities, an analytical infrastructure at
the project and researcher level aimed at facilitating the mapping of global public efforts to support
research and innovation towards a range of possible societal objectives, from the very aggregate
to the very specific, so that global funding gaps can be readily identified.
•
OECD could be asked to evaluate different models for securing prima facie information directly
from key stakeholders in the domain of scientific research and innovation across the globe in
order to identify, on a timelier basis, key emerging challenges and possible policy responses.
10
DSTI/EAS/STP/NESTI(2016)10
•
OECD should continue its efforts to develop statistical frameworks intended to reflect the broad and
connected nature of science, technology and innovation, the process of and implications of
digitalisation, as well as the plurality of actors involved, based on available experiences and tools, and
in partnership with the relevant organisations.
•
As an economic organisation, OECD could engage more actively in the integration of science,
technology and innovation in economic statistics and in developing database models to account for
the contribution of knowledge to economic performance within and across countries.
•
One overriding consideration for OECD, compared to the messages from the 2006 Blue Sky, is to
ensure that STI statistics do not miss out on the phenomena of internationalisation because as a result
of statistics remaining strictly national in their approach.
4. Concluding remarks and questions to CSTP delegates
35.
It is challenging to condense in a few pages the main points and potential implications arising
from nearly three days of intense discussions in plenary and parallel groups. The concluding messages
appeared to point out to a need to move beyond discussions in which two opposing approaches are
compared and one ends up prevailing (e.g. traditional vs new sources, economic vs social measures,
narratives vs numbers, business vs academia). Several of the suggestions appeared to echo the need to use
methods applied in the worlds of research and innovation, to exploit production and usage
complementarities and get practitioners to work with users to introduce the practices and products that
have the potential to transform our understanding of science, technology and innovation systems and
policy makers ability to act on evidence.
36.
The Blue Sky Forum needs to be an overarching and continued conversation - a theme picked up
by many speakers - comprising several inter-related conversations involving several communities,
languages and contextual factors. Conversations are an ongoing process, which do not stop as the meeting
ends. The community participating in Blue Sky 2016 appeared to appreciate the involvement of the OECD
in facilitating this discussion.
37.
Beyond the wealth of available outputs 3, it is important to track as much as possible the medium
to long term impacts of the Blue Sky Forum over time, including the less tangible impact of informal
discussions in corridors and in Het Pand’s magnificent courtyard and library, and over meals and coffee
breaks, often among people who had never met before and may begin new collaborations as a result of this
experience. These convivial elements represent an important dimension of Blue Sky that participants
appeared to appreciate and provide a basis for implementing a Blue Sky agenda. 4
Next steps
38.
By the time this paper has been discussed at CSTP, a brief discussion would have already taken
place at the Global Science Forum and similar discussions may take place at other CSTP working parties.
Although Blue Sky is an activity within the CSTP programme of work, it also has implications and
potential benefits to other OECD committees. External groups are equally interested in the outcomes of
Blue Sky.
3.
All the presentation and poster materials are available online as well as the streaming videos for all
sessions that took place in the main plenary room. For more details see the Annex section.
4.
Some early evidence and feedback is already available from the twitter feed associated to the Forum
#BlueSky3.
11
DSTI/EAS/STP/NESTI(2016)10
39.
The draft CSTP Programme of Work and Budget for 2017-18 [DSTI/STP(2016)1/REV3]
features a number of activities that relate directly to the main messages arising from the Blue Sky Forum
and link to the points discussed at the joint CSTP-NESTI workshop held in March 2016. These relate in
particular, but not exclusively, to:
•
The cross-cutting CSTP project on bringing STI policy and governance into the digital era (1.6),
as a vehicle for exploring synergies between STI data and decision making.
•
The NESTI project on measuring the digitalisation of science through an international survey of
scientific authors (ISSA) (1.5), as a vehicle for finding out about digitalisation as well as a first
step towards building a platform for securing evidence from key stakeholders in the world of
science and innovation.
•
The joint CSTP/CIIE distributed work on R&D microdata to analyse the incidence and impacts
of public support policies (4.1)
•
The NESTI proposal for a proof of concept for an international micro database on public funding
of R&D projects (Fundstat) (4.5), drawing on the increasing openness and availability of such
administrative microdata
•
The development of STI Statistical guidance arising from Blue Sky 2016 recommendations (5.1),
particularly those relating to extending the existing frameworks to areas for which a strong
demand exists and it proves feasible to make recommendations across countries
•
The Science, Technology and Industry Scoreboard (5.3), as a vehicle for experimentation on STI
indicator production and communication.
40.
These projects provide effective mechanisms to proceed with the implementation of some of the
major messages arising from the Blue Sky Forum, which also allowed the secretariat to identify potential
external partners and advisors for these projects. Blue Sky does however seek to inform a longer term view
for STI statistics beyond the 2 year cycle of the forthcoming PWB. In the past decade, NESTI has used
roadmap documents to help articulate this long term view, a practice that has been increasingly adopted by
other CSTP working parties. It is expected that a similar approach to planning internal work will be used in
the near future and presented to the CSTP. Likewise, CSTP may wish to engage, directly or through
NESTI, in the development of thematic roadmaps with external communities and experts in areas where
such an approach is considered relevant and necessary in order to drive progress towards better STI
policies.
41.
In light of this summary note, CSTP delegates are invited to discuss the following points:
•
For those delegates who participated in Blue Sky, does this summary/overview document capture
the main discussions and messages arising from the Forum?
•
Do you consider the initial objectives of Blue Sky 2016 to have been met and what would you
suggest to monitor and enhance its medium to longer-term impact?
•
How do these messages align with the 2017/18 draft programme of work and budget?
•
How do delegates think that CSTP can use the Blue Sky process/brand to inform its long term
approach to building its statistical infrastructure?
•
What do you think is the most appropriate mechanism for publicly disseminating the key
messages and potential implications of Blue Sky?
12
DSTI/EAS/STP/NESTI(2016)10
ANNEX
A.1. Access to presentations and papers
The presentations, papers and posters displayed at Blue Sky are available online, on the Forum’s webpage. The
same applies to videos for the sessions that took place in the plenary room.
www.oecd.org/sti/blue-sky.htm
For convenience, presentations and papers can be also downloaded in bulk from:
Presentations day 1 https://www.dropbox.com/s/ut4ldq0kg0iny59/Day%201%20Mon%20PPTs.zip?dl=0
Presentations day 2 https://www.dropbox.com/s/0a3wekax9iii84e/Day%202%20Tues%20PPTs.zip?dl=0
Presentations day 3 https://www.dropbox.com/s/2q8fln3nea3yxq8/Day%203%20Wed%20PPTs.zip?dl=0
Papers: https://www.dropbox.com/s/jx5il8d39jo3wil/papers.zip?dl=0
13
DSTI/EAS/STP/NESTI(2016)10
A.2. A flavour of what was discussed in the 15 Blue Sky 2016 parallel sessions - selected 3 messages per session
Monday 19 September 2016
Data analytics for
science and
innovation
 Text mining tools promise to alleviate some of the common challenges facing STI statistics, e.g.
survey fatigue and unfit-for-purpose classification systems that are applied differently by human
coders (e.g. patent assessors using USPC).
 Theory-driven text mining offers new opportunities for generating STI indicators, e.g. through near
real-time monitoring and online media monitoring for sentiment analysis.
 Text mining depends on vocabularies, ontologies and other linguistic techniques. These can be
defined manually or automatically, and deductively (e.g. through topic modelling) or inductively (e.g.
through machine learning algorithms) – or in a combination of these approaches.
Technology
diffusion and
breakthroughs
 Appropriate reference frames / reference data sets / benchmarks are important requisites for the
assessment of technology diffusion.
 A long-term (funding/analytical/strategic) focus is beneficial in the assessment of technology diffusion,
in order to allow for the recognition of long-term dynamics and changes within the field.
 In advanced assessments of technology diffusion it is of great value to allow for an agile / dynamic
approach to data collection, as opposed to dependence upon a static data repository.
Developing novel
indicators from
scientometrics
 Traditional bibliometric indicators should be reviewed to add meaning and international comparability.
Over simplification – e.g. rankings, impact factors - can have negative implications.
 A quality dimension of process of validation –indicators of peer review - should be integrated. Adding
more dimensions could capture real author contributions as well as novelty.
 We should provide a more informed role to the users of bibliometric information
 Overall…NO! We are capturing something but improvement needed.
Capturing
innovation in firms:  Survey design, question design, content, implementation, matter for data quality and international
comparability.
do we get it right?
 Also respondent characteristics have significant impacts (e.g. their expertise in innovation at the
individual and firm level, micro firm, whether or not they buy in their major innovations,
translation/cultural aspects, etc.)
Leveraging the
potential of
administrative
data for science
and innovation
policy
 Metadata for research projects is inherently complicated; data access does not solve the problem;
need to consistently identify and measure R&D projects
 Tremendous potential in using machine-learning techniques to organize the large, unstructured data
and make it amenable for analysis
 Tremendous interest in networks and linkages and this raises difficult problems in disambiguation;
potential interesting work going forward on this
Tuesday 20 September 2016
Innovation and IP:
what data gaps
limit policy
discussion?
 IPRs beyond patents: need for holistic view (for instance, exploiting data on other IP - TMs, utility
models…). More information needed about trade secrets in particular.
 Better understanding of the use of IP by end users in products. Here we need better data, for instance
product-patent pairs. Licensing data would be particularly useful. So far we have been limited to just a
few sectors, like pharma.
 Better understanding of the mechanism of knowledge flows. Again, better data is needed, for instance
the diffusion from the scientific literature to practitioners (via the "enlightenment literature").
Researchers on
 Bibliometric data can provide a wealth of information on mobility. Data can provide levels of
14
DSTI/EAS/STP/NESTI(2016)10
the move
aggregation from the country to the region, institution and individual.
 Combining different sources of data can provide larger opportunities on a global scale. However,
linking challenges need to be resolved.
 Technology now provides new tools to scrape/mine Internet (e.g. CVs) such as Natural Processing
Language (NLP).
Interaction and
impacts of STI
policies
 Program evaluation - significant progress in both techniques and availability of linked datasets since
Blue Sky 2
 More to be done to assess the efficiency of programs and the joint impact of policies (but a unique
identifier for each firms using government support programs and complete information of each
support enjoyed by the firm are needed)
 STI System evaluation – complex, no appropriate model currently available. Operational definition of
STI system and internationally comparable proxies of policy levers are needed
Capturing hidden
innovators
 Go beyond definition of formal private sector/market.
 Need to extend the definition of innovation to cover households and public sector but also informal
business sector especially as the geography of innovation is changing. Social innovation more
problematic at this stage: definition still confusing.
 Need to investigate more the methodologies to capture innovation beyond formal private sector. Need
to define survey methodology that needs to be different from private sector one as the characteristics
are quite different.
 Even public sector is a controversial definition: is it only public administration? Does it include
universities? Hospitals? Need to do more research to see if all public sectors innovate the same way
or if there are substantial differences.
STI actors: the
potential of direct
surveys
 Bibliographic information not sufficient to explain research and innovation processes. Surveys are
useful and necessary to understand motivations driving research and research orientations.
 Surveys are necessary and useful to measure perceptions and opinions of Actors regarding the
development of the STI system, how institutional setting affects their behaviour, or the impact of
institutional reforms
 OECD should focus on global issues but still work with local researchers to increase the quality of the
data
 OECD should consider implementing direct surveys to address policy gaps and when data is not
sufficient to answer key policy questions
Wednesday 21 September 2016
Beyond indicators:
the innovation and
productivity nexus
 Micro-level: production functions are a useful tool and provide a conceptual framework for estimating
rates of return on investments. They are relevant for public policy to estimate the existence of
complementarities (or substitution effects)
 Macro-level: governments (and society) need to know what the rates of return are from different
public investments and measure spillovers from all intangible investments (education, training and
R&D), including by the public sector
 Improving productivity-innovation nexus needs better macro-micro nexus:
o Need for more micro-level measures to better understand aggregate dynamics and
determinants
o Encourage linking across different datasets (macro and micro-level datasets including firm,
bilibiometrics, patent data, etc.)
Towards
standards for a
common research
 Big potential in linking data on researchers (inputs, outputs of research, affiliations, geographical
information etc.) for a better understanding of their behaviour and for a better informed policy making
 Advances are being made towards data integration but many concepts remain black boxes. More
15
DSTI/EAS/STP/NESTI(2016)10
infrastructure
dialogue is needed between different communities to promote mutual understanding
 Models and experimentation to monitor open science are emerging. What are the metrics for open
science, being aware of the fact that open science is more than open access and open data? What
role for the OECD?
Trust, culture and
citizen's
engagement in
science and
innovation
 Although science is global, ‘science culture’ remains local; innovation is a collective process and
depends on social, spatial and historical contexts
 Develop metrics to account for culture in public understanding and attitudes to science and
innovation. Not country rankings, but cluster analysis across a set of variables
 Policy making could be helped by considering different approaches to segmenting populations in
surveys. Disengaged people have different, but valid, attitudes
 Scientists often don’t communicate what the public wants to know
 Could OECD become curator of existing subjective databases around the world? Develop a “Frascati
manual” on public attitudes to science and innovation – a “Ghent Manual” ?
Developing novel
approaches to
measure human
capital and
innovation
 R&D sample survey in Germany shows that gender, education and nationality diversity can make a
difference in research teams and is positively related to innovative capacity. More historical data are
needed to determine causality (R&D sample survey)
 Mobility across research fields leads to less valuable inventions (loss of specialisation) but more novel
inventions (cross-fertilisation of ideas). Collaboration and access to scientific publications can help
balance the shortcomings of mobility.
 Oslo Manual provides clear guidelines on how to collect data but overlooks issues related to human
capital, impact on outcomes and regional innovation. Linking data from different sources could give
new insights without running new surveys.
Surveying
innovation in
different contexts
 More comprehensive and different indicators of innovation are needed to capture innovation practices
in non-traditional sectors and in developing economies. => These need to better capture incremental
and non-technological innovations, the sourcing of external knowledge and sectoral specificities
 There is a bias towards manufacturing in much of the analysis of innovation. Information on
innovation in rural areas, in mining, utilities and agriculture needs to be collected more
comprehensively.
 Surveys have to aim for more objective comparable information on innovation to capture those
innovating incrementally. The framing of surveys matters for responses.
16