The improvement of Ex-Post Impact Assessment as a tool for Policy Making in the
field of RTDI
Peter Teirlinck – Hogeschool-Universiteit Brussel & Universiteit Antwerpen
Acknowledgements
This paper presents a summary of the literature review undertaken for FP7 OMC-net
project 234501 ‘Optimising the Policy Mix by the Development of a Common
Methodology for the Assessment of (Socio-) Economic Impacts of RTDI Public
Funding’. Launched in March 2009, this project brought together a network of 15
partners - mainly public administrations and agencies - from around Europe (a mix of
EU member states and Associated States) with the goal of exchanging information and
mutual learning on national activities with respect to the assessment of the (socio-)
economic impacts of public funding (policies and instruments) of Research,
Technological Development and Innovation (RTDI). The work presented here takes on
board the insights provided in a joint effort with Henri Delanghe (European
Commission, DG Research).
1. Introduction
Europe is facing major societal challenges related to global competition, the
transformation to a knowledge and service economy, demographic sustainability, social
justice, climate change, secure, sustainable and competitive energy, migratory pressure,
and security and safety (European Commission, 2007). In response to these challenges
the European Commission in close collaboration with the Member States presented - as
a follow-up of the Lisbon strategy - ‘Europe 2020 - A strategy for smart, sustainable
and inclusive growth’ (European Commission, 2010a). Instrumental to achieve such
growth is the Flagship Initiative ‘Innovation Union’ (European Commission, 2010b)
which states that close monitoring of the impact of various policy measures (mainly
related to the creation of adequate framework conditions for innovation) is needed.
Public European and, even on a larger scale, national (and regional) Research,
Technology Development and Innovation (RTDI) funding plays an important role in
making the EU 2020 ambitions real. The key role assigned to public funding relates to
the existence of market failures, which prevent the private sector from investing in
research at the socially desired optimum level (Arrow, 1962).
Prior to public intervention governments must carefully design the way in which
they will intervene (SEC, 2007), and RTDI support should be provided in the most
appropriate areas (those with large spillover effects and where the private sector would
not get involved in on its own) and through the most effective instruments. Increasing
budget constraints for public funding in Europe (enforced by the economic and financial
crisis starting in 2008-2009) enhances this selection process. Moreover, from a policy
mix perspective (Nauwelaers, 2009) there is a growing demand to justify each euro
spent for RTDI in comparison with other policy intervention areas (like e.g. health,
environment, ...), as well as to consider the interactions and synergies between policy
measures in different policy areas.
In contrast with several other policy areas, assessing the socio-economic impacts
of public funding for RTDI is not a common practice in Europe (Delanghe and
2
Teirlinck, 2009). In order to convince policy makers not to cut down research and
innovation budgets, there is a necessity to pay more attention to the long term socioeconomic outcomes of research. This paper provides insights into a framework to do so
by addressing methodological aspects and a focus on the missing link between impact
assessment and policy design (Delanghe and Teirlinck, 2009).
Section 2 defines ‘ex-post impact assessment’, presents the ‘logic intervention
model’ as a framework for analyzing the impacts of public funding for RTDI, and
highlights the fact that the policy cycle is not closed. Section 3 presents empirical
evidence on shortcomings in existing impact assessment/evaluation exercises. The focus
is on impact assessment studies ordered by public administrations. Section 4 concludes
and provides recommendations for improving the usefulness of impact assessment
exercises as a tool for policy making.
2. Logic intervention model and ex-post impact assessment in the policy cycle
Assessing the socio-economic impacts of public policy is becoming increasingly
important as the changing role and position of government has resulted in a growing
demand for evidence-based policies (OECD, 2007: 4). Doing so allows policy makers
to address fundamental questions: which intervention and support instruments work,
which do not, and how can they be improved? In this section the concept of impact is
defined and its interrelations with policy design are brought together in so-called logic
intervention models. These models play an important role in ‘evidence-based policymaking’.
3
2.1 The logic intervention model and ex-post impact assessment
Regardless of its nature (policy, programme, measure, project), a public intervention
can be analyzed as a set of financial, organizational and human resources mobilized to
achieve, in a given period of time, an objective or set of objectives, with the aim of
solving or overcoming a problem or difficulty affecting targeted groups.
Logic intervention models (Figure 1) can be defined as ‘logically consistent
descriptions of the design of a programme as well as of the expected impact’ (Lengrand
et al., 2006: 76). They help identify and set out the relationship between the socioeconomic needs to be addressed by the intervention and its objectives (initial statements
of the outcomes intended to be achieved by an intervention), inputs, outputs, and
outcomes. Outcomes include results (immediate changes that arise for direct addressees
at the end of their participation in a public intervention) and impacts (longer term effects
of the intervention).
In contrast with results (which are directly influenced by the policy
intervention), impacts can be defined as what a policy measure can influence only
indirectly. They refer to longer term socio-economic consequences which can be
observed only after a certain period after the completion of an intervention (hence expost impact assessment), and which may affect either direct addressees of the
intervention or indirect addressees falling outside the boundary of the intervention, who
may be winners or losers. Important to note is the broad scope (socio-economic, direct
and indirect, positive and negative) and time dimension (longer term, observed after a
certain period after the completion of an intervention) of impacts.
4
Figure 1: The Logic Intervention Model
Target
populations
Global impacts
Intermediate
impacts
Needs,
Problems
and issues
Results
Outcomes
Design
and
implementation
Objectives
Inputs
Outputs
Efficiency
Relevance
Effectiveness
Evaluation
Utility and sustainability
Source: Adapted from European Commission (2004: 72). See also European Court of
Auditors (sd: 17).
5
The different building blocks of the logic model can be linked through different
interrelations capturing ‘relevance’, ‘effectiveness’, ‘efficiency’, and ‘utility and
sustainability’ (European Commission, 2004: 39). ‘Relevance’ relates to the extent to
which an intervention’s objectives are pertinent to the needs, problems and issues to be
addressed. Closely related to relevance are the extent to which the intervention logic
does not contradict other interventions with similar objectives (‘coherence’) and the
extent to which resources are available in due time, in appropriate quantity and quality,
and at the best price (‘economy’). ‘Effectiveness’ refers to the extent to which the
objectives set are achieved in the intended results and impacts. ‘Efficiency’ refers to the
extent to which the desired outcomes are achieved at a reasonable cost. The principle of
efficiency is concerned with the best relationship between resources employed and
outcomes. A last interrelation links needs with results and impacts. ‘Utility’ refers to the
extent to which outcomes correspond with the needs, problems and issues to be
addressed. ‘Sustainability’ means the extent to which positive outcomes are likely to
last after an intervention has stopped.
‘Ex-post impact assessment’ focuses in particular on the broad, longer term
impacts of public interventions. As such it looks mainly at effectiveness, utility and
sustainability. It should be seen in relation to the broader activity of evaluation which
can be defined as ‘a matter of seeing how well a policy or programme has achieved the
objectives set for it’ (Lengrand et al., 2006: 31).
The focus on the broad, longer term impacts of public interventions raises the
important challenge to look beyond the single policy, programme, measure or project to
the broader policy mix. The policy mix refers to the combination of policy instruments,
which interact to influence the quantity and quality of R&D investments in public and
6
private sectors. Policy instruments are all programmes, organizations, rules and
regulations with an active involvement of the public sector, which intentionally or
unintentionally affect R&D investments. Interactions point to the fact that the influence
of one policy instrument is modified by the co-existence of other policy instruments in
the policy mix (OECD, 2010).
2.2 Ex-post impact assessment and closing the policy cycle
The policy cycle is generally conceived to consist of four basic stages. In stage one, the
‘agenda-setting, problem identification’ stage, policy problems are defined and policy
issues are raised, introduced to the political stage by different governmental institutions,
individuals, interest groups, or specific events. In stage two, the ‘policy formulation’
stage, analysis and politics determine how the agenda item is translated into legislation.
This encompasses several aspects including the development and consideration of
alternative policy options, the selection of a preferred option, and its adoption. Next, in
stage three, the ‘implementation’ stage, the adopted policy is implemented
(administered and enforced) by the bureaucracy, or by a government agency. The
bureaucracy or agency translates the policy into a concrete set of actions, and makes
judgments as to the intent, goals, timetables, program design, and reporting methods.
Finally, in the ‘evaluation’ stage, the implementation of policy is evaluated in order to
find out what is working and what is not. The impacts of the policy are assessed. If
goals exist, the effectiveness of the policy and its components can be determined and
side-effects have to be discovered and evaluated.
7
Figure 2 presents the place of ex-post impact assessment in the policy cycle
taking into account that (i) the policy formulation process is supported by ex-ante
evaluation/impact assessment; (ii) the policy implementation process is supported by
monitoring and interim evaluation; and (iii) that the evaluation process consists of expost evaluation and ex-post impact assessment.
Figure 2: Ex-post impact assessment in the policy cycle
Agendasetting
Policyformulation
Implementation
Evaluation
Ex-ante
evaluation
Monitoring
Interim
evaluation
Ex-post
evaluation
Ex-post
Impact
assessment
Source: Delanghe and Teirlinck (2009).
8
The different activities underpinning evidence-based policy-making assume the
closure of the policy cycle through full feedback effects and learning from monitoring,
interim evaluation, ex-post evaluation and ex-post impact assessment. The closure of
the policy cycle depends first and foremost on the usefulness for policy making of the
evaluation product. This is determined by a variety of factors including (Harty and
Newcomer, 2004 and Bayhan et al., 2011): the degree of clarity of the initial
intervention logic (and the feasibility of the assessment); the degree of stakeholder
involvement and consensus on evaluation criteria and expectations; the timing of the
assessment; the extent to which failures are assessed as well as successes; the degree to
which reasoned, pre-tested methodologies and indicators are used to generate
information which is easily understood and interpreted; the extent to which the
information produced is relevant for broader programme and policy design rather than
highly context-specific; the balance between data collection and analysis; and the extent
of dissemination of information produced.
In the next section the obstacles hampering feedback effects from ex-post impact
assessment in policy design will be looked after in an empirical way.
3. Use and usefuleness of ex-post impact assessment exercises
This section looks in an empirical way at the use (section 3.1) and usefulness (section
3.2) of impact assessment practices in Europe. With regard to the use of impact
assessment, it is investigated to what extent impact assessment is implemented at
9
country level in Europe. With regard to usefulness, attention is paid to factors that
hamper the usefulness of ex-post impact assessment for policy making. As starting point
of the analysis we take 21 ex-post impact assessment exercises ordered by government
institutions.
The focus of the analysis is on ex-post impact assessment/evaluation exercises in
three policy clusters: public funding of private RTDI, public funding of industry-science
relations, and public funding of public research organizations. The clusters are related to
challenges for National Innovation Systems in most European countries related to: (i)
low R&D investments and the need to increase the overall R&D spending (especially in
the private sector); (ii) a need for improvement of the interactions between industry and
science actors; and (iii) a better governance of research and innovation in public
research organizations.
3.1 Use of ex-post impact assessment at country level in Europe
In this section a global analysis of the R&D and innovation policies of 27 EU countries
will be offered. The used information is based on the database of the European
Inventory of Research and Innovation Policy Measures created - on behalf of the
European Commission - by the ERAWATCH organization, and combined with a
questionnaire among the eleven countries participating in the OMC-net project.
Based on an analysis of the qualitative data of the ERAWATCH-website, the
evaluation of R&D and innovation policies turns out to have developed strongly in
European Union Member States. However, it seems to have become a standard practice
in some countries only, and the countries can be classified in four groups based on the
10
existing evaluation culture (Eparvier, 2009). A first group brings together leading
countries in evaluation. At present, these countries (Austria, Denmark, Finland,
Germany, Ireland, the Netherlands, Sweden and the United Kingdom) have a wellestablished culture of evaluation and carry out systematic evaluations of programs and
institutions. A second group is made of countries that have strongly reinforced
evaluation practices, structures and culture (Belgium, Czech Republic, Estonia,
France, Portugal and Slovenia). These countries recently engaged in a process of
systematizing evaluation of programs and of institutions. A third group gathers
countries that have recently established or that are about to set up evaluation
practices, structures and culture (Bulgaria, Italy, Luxembourg, Malta, Romania and
Slovakia). A last group is composed of countries that do carry out evaluations but
not on a systematic basis (Cyprus, Greece, Hungary, Latvia, Lithuania, Poland and
Spain).
Based on an in-depth analysis of 814 instruments (for which detailed
information was available) used in the 27 EU Member States (ERAWATCH Research
Inventory Report: overview across EU countries - http://cordis.europa.eu/erawatch) the
classification of the use of evaluation in Europe could be made as presented in Table 1.
11
Table 1: The ex-post evaluation/assessment culture: a qualitative versus
quantitative assessment
Qualitative assessment of the evaluation culture included in
the ERAWATCH country profiles published in the web site
Percentage of
instruments
culture
evaluated ex
post
Systematic or
Low evaluation
Frequent but not
well established
systematic evaluation
evaluation
(Turkey*)
culture
Czech Republic,
Less than 10%
Lithuania, Portugal,
(Cyprus*)
Hungary, Poland,
Belgium
Malta, Greece
Between
10 and 30%
Luxembourg Spain
Italy
Romania, France,
Netherlands,
Ireland, Sweden,
Austria, United
Slovenia
Kingdom
Between
Slovak Republic,
30 and 40%
Latvia, Estonia
Over 45%
Bulgaria
Finland, Denmark,
Germany
Source: Delanghe et al., 2011. *No qualitative assessment was obtained for Cyprus and
no statistical data were available for Turkey.
12
A more detailed analysis of funding measures in the three target policy clusters
revealed that each cluster accounts for about 10% of the total RTDI government
budgets. Important with respect to policy mix and causal linkages between policy
measures and outcomes is that more than half of the policy instruments target several
policy clusters at the same time. In terms of evaluation, more than half of the
instruments have not been evaluated so far and there are marked differences in
propensity to evaluate according to the policy area and according to the country.
Specifically with regard to ex-post evaluation, the propensity to do an assessment
differs according to the type of policy instrument and according to the country of origin.
Instruments aimed at the direct or indirect promotion of private R&D investments are
evaluated in one out of four cases when it concerns indirect support for R&D; for direct
support this amounted to one out of six. About half of the instruments related to public
research organizations and universities were evaluated ex-post. Finally, close to onefourth of the instruments focused on public-private R&D cooperation and of the
measures considered in support of infrastructure for public-private linkages were
evaluated ex-post. In the case of the instruments to foster knowledge transfer, this
percentage was almost 14% (for more detailed information, see Delanghe et al., 2011).
An additional survey among the OMC-net project partners identified that the
planning of impact assessment in the policy cycle remains predominantly ad hoc.
Nevertheless, there is an important trend towards more interest in the use of impact
assessment in RTDI policy making (Table 2).
13
Table 2: Planning of impact assessment in the policy cycle for policy measures
funding RTDI
Extent of planning of impact assessment in the policy
cycle
for policy measures funding RTDI
Occasional
Not/seldom
– ad hoc
Systematic
Austria (Legal
Bulgaria
Recent and
Estonia
planned
changes in the
Increased
Malta
use of impact
use
Spain
assessment in
Turkey
RTDI policy
Iceland
making
Austria (for
Basis for
funding not
universities)*
oriented at
Czech Republic
universities)
(for institutional
Belgium
support for research
France
- focused at
Sweden (Not
scientific impact!)
VINNOVA -
Sweden
Focused at
(VINNOVA –
scientific
focused at impact in
impact)
terms of sustainable
development)
No change
Note: in each of the countries having experience in impact assessment the results of
these exercises are used mainly for the design of new policy measures or for the
improvement of existing ones. *The systemic planning of impact assessment of
Austrian universities is indicated in Austrian law, but de facto not reality until now.
14
3.2 Usefulness of ex-post impact assessment exercises
In the analysis of the impact assessment exercises attention is paid to ‘methods, data
collection and indicators’ and to ‘user involvement and recommendations’. The focus of
the analysis is on the 21 ex-post impact assessment exercises (presented in Table 3) that
have been selected from 7 countries and that have been considered important exercises
at country level since the year 2000. A common characteristic of these exercises is that
they are ordered by government institutions.
15
Table 3: Overview of impact assessment exercises in the field of public funding for
RTDI
Country Impact assessment study
Intellectual Property Rights at Austrian Universities: Evaluation of the
Austria
uni:invent programme
Evaluation of the Austrian Industrial Research Promotion Fund (FFF) -
Austria
Impact Analysis
Impact of the Federal Scientific Institutions on the economic, scientific, social
Belgium
and cultural development in Belgium
VRWB research: impact analysis of fiscal stimuli on R&D expenditures in
Belgium
Flanders
Belgium
The impact of public R&D funding in Flanders
A look into the Black Box: What difference do IWT R&D grants make for
Belgium
their clients
Making the difference: The evaluation of 'Behavioural Additionality" of R&D
Belgium
subsidies.
Bulgaria
Peer-review of the national R&D system under the OMC forth cycle
Bulgaria
Evaluation of the National Science Foundation (NSF)
PREDIT 3: National research and innovation programme in terrestrial
France
transports
16
Evaluation of PREBAT (Research and exploitation of energy in buildings France
2005-2009)
Impact study SRC 2006 / impact of French research and technology
France
organizations (RTO's)
France
Report on the valorization of research
Evaluation of the first phase (2006-2008) of the policy on competitiveness
France
clusters (pôles de compétitivité)
France
Competititveness clusters (pôles de compétitivité), what can we expect?
Which articulation between PRES, RTRA and competitiveness clusters (pôles
France
de compétitivité)
Information report on the economic incidences of an augmentation of research
France
expenditures in Europe
Impact of the research tax credit on the funding of private R&D: an
France
econometric evaluation
Spain
Impact Assessment of the FP6 in the Spanish R&D public system
Spain
Impact of R&D on the Industry
Turkey
An Assessment of the Industrial Technology Project
To collect and harmonize the information for the 21 studies, a survey has been
set up in order to get the maximum synergy possible with other works already done in
the area, mainly the Inno-Policy Trendchart repository of European policy measures and
17
Inno-Appraisal repository of European evaluation studies (http://www.proinnoeurope.eu/index.cfm?fuseaction=page.display&topicID=262&parent ID=52).
Four types of impacts (scientific, technological, economic, and societal) were
considered. Economic impacts were most prominent (86% of the cases), followed by
scientific impacts (48%). In less than one third of the cases broader societal impacts
were investigated. The latter can be closely related to the fact that only 2 out of the 21
studies had a broader societal impact as initial objective. Technological impacts were
the focus in less than one fourth of the studies.
3.2.1 Methods and indicators
Insights into the use (good practices) and possibilities for harmonization in terms of
methodologies and indicators for impact assessment are crucial elements to promote
impact assessment at the level of policy makers and to stimulate them to make more use
of impact assessment in the policy making process (OECD, 2009).
Methods
The harmonized analysis of the 21 case studies reveals that 40% of the exercises are not
based on previously designed methodologies. Moreover, in close to eighty percent of
the cases these methodologies have not been used for other impact assessment exercises
of the same policy measure (this case sensitivity can be related either to the fact that the
methodology was not considered appropriate or to the fact that no other impact
assessment exercises have been performed).
18
The focus of the studies is mainly on the consequences or results of the measure
in terms of ‘outcomes’, and to verify if the goals of the measure were achieved.
Moreover, considerable attention is paid to input, output, and behavioural additionality.
Programme or project implementation efficiency as well as policy/strategy development
are covered in less than one third of the exercises. This seems to indicate that impact
assessment exercises not directly aim at policy (re-)design or fail to do so.
19
Table 4: Topics covered in impact assessment exercises of public funding for RTDI
Programme Implementation
Outcomes
86% Efficiency
24%
Goal Attainment/Effectiveness
62% Quality of outputs
24%
Output additionality
38% Internal consistency
14%
Behavioural additionality
33% Coherence/Complementarity
14%
Input additionality
33% External consistency
5%
Project implementation
Policy/Strategy Development
29% Efficiency
5%
24% Gender issues
5%
Value for Money/Return on
Investment/Cost-Benefit Analysis
Source: Questionnaire on impact assessment for public funding of RTDI, analysis of the
21 case studies presented in Table 3. Note: responses may be biased due to difficulties
for institutions to criticize exercises they or other ministries/agencies have ordered.
20
The appropriateness of the design of the study, the chosen methodology related to the
objectives of the impact assessment and the nature of the policy measure - on average can be considered fairly appropriate (measured on a five-point Likert Scale). Lack of
experience and data and too much hypotheses underlying the relationship between
RTDI funding and impacts are the hampering factors for the appropriateness of the
design of impact assessment exercises. Complexity to measure net impacts is a main
barrier for the usefulness for mutual learning of these exercises.
Indicators
Preferred ways of data collection include participant surveys, existing databases and
interviews. Descriptive, case-study analysis and econometric analysis are the main data
analysis methods for impact assessment exercises. About three-fourths of the studies
combine qualitative and quantitative approaches for data analysis.
Barely in one third of the cases impact indicators are considered at the set-up of the
policy measure (or in other words included in the logic intervention model). Most
exercises make use of existing indicators. Only in one case completely new indicators in
the field of impact assessment were created.
21
Table 5: Appropriateness of indicators for impact measurement
Impact
Appropriateness of indicators for the
Efficiency (in
measurement of impacts
terms of
Not
Sometimes
appropriate
appropriate
Appropriate
information
provided/
resources
devoted)*
Scientific
11%
33%
56%
2.57
Technological
0%
50%
50%
2.67
Economic
0%
27%
73%
2.86
83%
0%
1.00
Societal (including 17%
environmental)
Source: Questionnaire on impact assessment for public funding of RTDI, analysis of the
21 case studies presented in Table 3. * Average efficiency: average based on a 5-point
Likert scale (not efficient = 0; moderately efficient = 1; average efficient =2; fairly
efficient = 3; very efficient = 4).
22
The high reliance on existing indicators could explain the limited perceived
appropriateness of the indicators, especially with regard to societal, technological and
scientific impacts. Important in relation to the broad EU 2020 challenges is the finding
that in none of the exercises indicators were found fully appropriate to measure societal
impacts. This could be related to the context specificity and to the need to consider a
broader policy mix. For example: the impact on society due to both research and norms
& legislation when considering the decrease of mortal accidents, or the decrease of the
emission of greenhouse gases.
In a considerable number of cases indicators to measure scientific and societal
impacts were found completely inappropriate. Not surprisingly, these findings resulted
in a lower perceived efficiency of the indicators to measure societal impacts. Important
shortcomings in terms of indicators are data availability (including the high number of
potentially useful indicators and the possibility to make comparisons) and lack of
appropriateness to adapt to specificities in terms of the actors benefiting from the public
funding.
3.2.3 User involvement, recommendations and consequences
Based on the analysis of the 21 case studies, several hampering factors for the
usefulness of impact assessment as a tool for policy making have been identified. A
main finding is that too little attention is paid to policy/strategy development and to
project implementation efficiency in impact assessment. This means that the policy
cycle is not closed. This can be related to the fact that evening less than half of the
impact assessment studies ordered by government administrations/agencies, politicians
23
and the program management are involved. This confirms the existence of a gap
between policy formulation and implementation.
Moreover, relatively low involvement can be witnessed of those directly or
potentially supported by the policy measure or programme. Therefore, it comes as no
surprise that impact assessment exercises mainly provide recommendations for redesign, expansion or management of the programme/measure under consideration.
Related to the policy mix, broader policy design is not found to be an objective in more
than half of the impact assessment exercises and the influence and relation with other
policy measures (e.g. in terms of re-design or merger) are limited.
24
Table 6: Composition of the user group of the impact assessment exercise
Policy Makers (Government
Potential Users of the Policy
officials)
95%
Measure
33%
Programme Management
48%
Auditors/Financial Authorities
14%
Policy Analysts
48%
General Public
14%
Those directly supported by the
External/ (co)sponsor of the
measure
43%
Policy Makers (politicians)
43%
measure
10%
Source: Questionnaire on impact assessment for public funding of RTDI
25
Table 7: Recommendations and consequences of impact assessment exercises
% of
% of
Recommendations?
cases
case
Consequences?
s
Termination/merger of the
No
14
measure
0
52
Re-design of the measure
52
management
48
Re-design of another measure
14
Yes, targeted at broader policy design
43
Expansion of the measure
29
No consequences
38
Yes, targeted at the design of the
measure
Yes, targeted at the programme
Source: Questionnaire on impact assessment for public funding of RTDI
26
4. Reflections on how to improve impact assessment as a tool for policy making
Over the last decade increasing resources have been allocated to RTDI policies and expost impact assessment has been stepped up with some delay. Progress has been in
methods for impact assessment and more data are now available. Thus, there is both
more demand for the assessment of policies that absorb increasing resources and more
supply in terms of sophisticated methods, data, and experience. However, there are
marked differences in the practice of impact assessment according to the policy area
under consideration and also between countries. On average, in Europe, impact
assessment of public funding of RTDI still rather occurs in an ad hoc manner and little
attention is paid to the broad societal impacts of public funding for RTDI.
Assessing the economic, scientific, technologic and societal impacts of RTDI
policies is not a straightforward task. Methodologically there are several challenges, as
there are challenges concerning the utilisation of impact assessment findings and results
in the design of policy measures.
Four major methodological lessons can be derived from the impact assessment
exercises. First, a need for awareness of the difficulty to measure (quantify and qualify)
indirect economic and technological effects (not least by the receivers of public funding
for RTDI). Second, the importance to consider different policy measures at the same
time (policy mix). Also, there is a need for recognition of problems in terms of
attribution (both to specific measures and to specific territories) and timing. Third,
awareness is necessary of the lack of data to analyze and to benchmark the impacts
resulting in a lack of evidence-based conclusions. A final lesson is the need of
experienced people in evaluation/impact assessment itself, especially to bridge the gap
between the science community and policy stakeholders.
27
Also with regard to the full utilization of impact assessment and its results for
the optimization of RTDI policies, several hampering factors can be identified. A main
barrier consists of the lack of (early) involvement of government
administrations/agencies, politicians and the programme management
agencies/organizations in impact assessment studies. This results in a disrupted
feedback loop between policy formulation, implementation and policy optimization. A
second important hampering factor relates to the relative isolation in which impact
assessment is carried out. This means that influence on, and relation with, other policy
measures (other instruments in the policy mix) is often (too) limited.
A final main lesson of the mutual learning enhanced during the production of
this review is the recognition that different terminologies are used with regard to
evaluation and impact assessment. The logic intervention model has been presented as a
common methodological approach to impact assessment. It offers the advantage to
better link policy objectives with long term impacts and to consider methods and
indicators appropriate for doing so from the moment the policy measure is introduced
(i.e. in an ex-ante manner).
References
Arrow, K.J. (1962): Economic Welfare and the Allocation of Resources for Inventions,
in Nelson R.R. (ed.), The Rate and Direction of Inventive Activity: Economic and
Social Factors. Princeton University Press, Princeton.
Bayhan, D., Bukulmez, E., Ozdemir, A.H., Gok, A. and Edler, J. (2011): Governance
and Usefulness of Impact Assessments. Chapter 3 in Teirlinck, P. (ed.) Lessons for ex-
28
post impact assessment of public funding for RTDI in the policy cycle - Findings from
FP7 OMC-net project 234501: Optimising the Policy Mix by the Development of a
Common Methodology for the Assessment of (Socio-) Economic Impacts of RTDI
Public Funding (CIA4OPM), European Commission, Brussels.
Delanghe, H. and Teirlinck, P. (2009): Optimising the Policy Mix by the Development
of a Common Methodology for the Assessment of (Socio-)Economic Impacts of RTDI
Public Funding (CIA4OPM), DSTI/STP/TIP(2009)18, pp. 67, OECD, Paris.
Delanghe, H., Teirlinck, P., Heijs, J. and Sachwald, F. (2011): Practice of Impact
Assessment in the Policy Cycle, Chapter 2 in Teirlinck, P. (ed.) Lessons for ex-post
impact assessment of public funding for RTDI in the policy cycle - Findings from FP7
OMC-net project 234501: 'Optimising the Policy Mix by the Development of a
Common Methodology for the Assessment of (Socio-) Economic Impacts of RTDI
Public Funding (CIA4OPM), European Commission, Brussels.
Eparvier, P. (2009): ERAWATCH Research Inventory Report for: Overview Across EU
Countries. European Commission.
European Commission (2004): Evaluating EU Activities – A Practical Guide for the
Commission Services, Office for Official Publications of the European Communities,
Luxembourg.
European Commission (2007): Europe in the global research landscape. European
Commission, DG Research, Brussels.
European Commission (2010a): EUROPE 2020 - A strategy for smart, sustainable and
inclusive growth, 3.3.2010, COM (2010), European Commission, Brussels.
29
European Commission (2010b): Europe 2020 Flagship Initiative Innovation Union –
COM (2010) 546, European Commission, Brussels.
European Court of Auditors (sd), Performance Audit Manual, Court of Auditors,
Luxembourg.
Hatry, H.P. and K.E. Newcomer (2004), ‘Pitfalls of Evaluation’, in: J.S. Wholey, H.P.
Hatry and K.E. Newcomer, Handbook of Practical Program Evaluation.
Lengrand, L. & Associés, PREST, ANRT, Reidev Ltd. (2006), SMART
INNOVATION: A Practical Guide to Evaluating Innovation Programmes, ECSC-ECEAEC, Brussels-Luxembourg.
Nauwelaers, C. (2009), Policy Mixes for R&D in Europe. UNU-MERIT, April 2009. A
study commissioned by the European Commission - Directorate-General for Research.
OECD (2007), Assessing the Socio-Economic Impacts of Public Investment in R&D,
OECD, Paris.
OECD (2009), Enhancing Public Research Performance through Evaluation, Impact
Assessment and Priority Setting, OECD, Paris.
OECD (2010), STI Outlook, OECD, Paris.
SEC (2007), SEC(2007)213 of 21/02/2007 Responding to strategic needs: Reinforcing
the use of evaluation. http://ec.europa.eu/budget/library/documents/evaluation/
communication/eval_comm_sec_2007_213_en.pdf.
30