Tailor-made evaluation concepts
for innovation policy learning
Research and the Knowledge Based Society – Measuring the Link
24th May 2004, NUI Galway, Ireland
Stefan Kuhlmann (ISI; UU), Jakob Edler (ISI)
IS
I
page 1
page 1
F
r
a
u
n
h
o
fe
rIn
s
titu
te
IS
I
F
r
a
u
n
h
o
fe
rIn
s
titu
te
S
y
s
te
m
s
a
n
d
In
n
o
v
a
tio
n
R
e
s
e
a
rc
h
S
y
s
te
m
s
a
n
d
In
n
o
v
a
tio
n
R
e
s
e
a
r
c
h
Copernicus Institute
for Sustainable
Stefan
Kuhlmann / Jakob Edler:
Development and Innovation
Tailor-made evaluation concepts for innovation policy learning
Overview
 Scope of innovation policy evaluation
 Four poles of evaluation missions and approaches
 Two opposed examples
 Summative, quantitative poles
example: Relationship between R&D collaboration, subsidies and patenting
 Formative, qualitative poles
example: Assessment of policy instruments supporting "competence
centres"
 Conclusions
page 2
IS
I
F
r
a
u
n
h
o
fe
rIn
s
titu
te
S
y
s
te
m
s
a
n
d
In
n
o
v
a
tio
n
R
e
s
e
a
r
c
h
Stefan Kuhlmann / Jakob Edler:
Tailor-made evaluation concepts for innovation policy learning
Typical R&D evaluation issues and questions
(Source: Arnold/Guy 1997, 72)
Appropriateness: Was it the right thing to do?
Economy: Has it worked out cheaper than we expected?
Effectiveness: Has it lived up to the expectations?
Efficiency: What’s the return on investment (ROI)?
Efficacy: How does the ROI compare with expectations?
Process efficiency: Is it working well?
Quality: How good are the outputs?
Impact: What has happened as a result of it?
Additionality: What has happened over and above what would have happened anyway?
Displacement: What hasn’t happened which would have happened in its absence?
Process Improvement: How can we do it better?
Strategy: What should we do next?
page 3
IS
I
F
r
a
u
n
h
o
fe
rIn
s
titu
te
S
y
s
te
m
s
a
n
d
In
n
o
v
a
tio
n
R
e
s
e
a
r
c
h
Stefan Kuhlmann / Jakob Edler:
Tailor-made evaluation concepts for innovation policy learning
Impact dimensions of public research and innovation spending
Direct impacts
Indirect impacts
Main domains of
impact of public
spending
Short-term
Long-term
Short-term
Long-term
Science
(“Wissenschaft”)
Typical impacts
scientific
findings
knowledge
improved
teaching
industrial
spill-overs
improved
technology
improved
technical
know-how
increased
productivity
improved
competitiveness
improved
understanding
problemsolving
increased
problem awareness
increased
general satisfaction
Economy and
society
Typical impacts
Policy
Typical impacts
page 4
IS
I
F
r
a
u
n
h
o
fe
rIn
s
titu
te
S
y
s
te
m
s
a
n
d
In
n
o
v
a
tio
n
R
e
s
e
a
r
c
h
Stefan Kuhlmann / Jakob Edler:
Tailor-made evaluation concepts for innovation policy learning
Summative and formative evaluation
 Summative Evaluation
 systematic, indicator based
 mainly ex post - or interim - measurement and assessment of the performance of
programmes (including projects)
 to assess the programme design, implementation management and the leverage of
funding and to learn for future approaches
 Formative Evaluation
 systematic consulting, moderating, assessing activities
 seeking to assist policy makers, programme managers and programme participants
 throughout the whole life cycle of funding programmes
 to make all actors involved learn and (re-)adjust
 and thus contribute to the overall success (and/or improvement and/or termination) of
programmes and funded structures and to learn for future approaches.
page 5
IS
I
F
r
a
u
n
h
o
fe
rIn
s
titu
te
S
y
s
te
m
s
a
n
d
In
n
o
v
a
tio
n
R
e
s
e
a
r
c
h
Stefan Kuhlmann / Jakob Edler:
Tailor-made evaluation concepts for innovation policy learning
Evaluation methods, quantitative and qualitative
Source: Polt, W. et al., RTD Evaluation
Toolbox,
http://epub.jrc.es/evaluationtoolbox/start.swf
 Quantitative: Statistical data analysis
 Innovation Surveys: basic data describe the innovation process, using descriptive statistics
 Benchmarking: comparisons based on a relevant set of indicators across entities
 Quantitative: Modelling methodologies
 Macroeconomic modelling and simulation: broader socioeconomic impact of policy
interventions
 Microeconometric modelling: effects of policy intervention at the level of individuals or firms
 Productivity analysis: impact of R&D on productivity growth at different levels data aggregation
 Comparison group approach: effect on participants using statistical sophisticated techniques
 Qualitative and semi-quantitative methodologies
 Interviews and case studies: direct observation of naturally occurring events to investigate
behaviours in their indigenous social setting
 Cost-benefit analysis: economic efficiency by appraising economic and social effects
 Expert panels/peer review: scientific output relying on the perception of peer scientists
 Network analysis: structure of cooperation relationships and consequences for individuals and
their social connections into networks
page 6
 Foresight/ technology assessment: identification of potential mismatches in the strategic
efficiency of projects and programmes
IS
I
F
r
a
u
n
h
o
fe
rIn
s
titu
te
S
y
s
te
m
s
a
n
d
In
n
o
v
a
tio
n
R
e
s
e
a
r
c
h
Stefan Kuhlmann / Jakob Edler:
Tailor-made evaluation concepts for innovation policy learning
Evaluation Matrix: Matching policy instruments and methods
Source: Polt, W. et al., RTD Evaluation Toolbox,
http://epub.jrc.es/evaluationtoolbox/start.swf
page 7
IS
I
F
r
a
u
n
h
o
fe
rIn
s
titu
te
S
y
s
te
m
s
a
n
d
In
n
o
v
a
tio
n
R
e
s
e
a
r
c
h
Stefan Kuhlmann / Jakob Edler:
Tailor-made evaluation concepts for innovation policy learning
Four poles of evaluation missions and approaches
qualitative
 Analysis of policy context and governance
 Need for …
 awareness of diversity of
actors' perspectives
 methodology mix
summative
formative
 Measurement of policy assumptions,
outputs and effects
 Need for …
 robust operationalisation
 (sophisticated) methodologies
 reliable and encompassing data quantitative
page 8
IS
I
F
r
a
u
n
h
o
fe
rIn
s
titu
te
S
y
s
te
m
s
a
n
d
In
n
o
v
a
tio
n
R
e
s
e
a
r
c
h
Stefan Kuhlmann / Jakob Edler:
Tailor-made evaluation concepts for innovation policy learning
Summative, quantitative poles - S/T indicators
and stages of innovation
Knowledge stock
Fundamental
Applied
Experimental
research
research
developement
Standardisation
Output Indicators:
Patent
Scientific
publication
(Technometric)
application
Idea, theory,
discovery
Resource indicators
Literature
Patent
citation
citation
Technical design
R&D results indicators
R&D personnel
Product design,
innovation
Internal R&D
expenditures
Expenditures für
knowledge
transfer, fees,
licences,
standards
documents
Measurable functions
Measurable feed-back
Intangible functions
F
r
a
u
n
h
o
fe
rIn
s
titu
te
S
y
s
te
m
s
a
n
d
In
n
o
v
a
tio
n
R
e
s
e
a
r
c
h
innovation counts
R&D-intensive
goods:
employment,
production
growth, factor
productivity
Various
foreign trade
indicators
market shares
Imitation,
improvement,
diffusion,
exploitation,
disposal
IS
I
characteristics,
page 9
Stefan Kuhlmann / Jakob Edler:
Tailor-made evaluation concepts for innovation policy learning
Summative, quantitative poles - example:
Relationship between R&D collaboration, subsidies and patenting
 Recent evaluative study of D. Czarnitzki (ZEW), B. Ebersberger (VTT GTS) and Andreas
Fier (ZEW):
The Relationship between R&D Collaboration, Subsidies and Patenting Activity:
Empirical Evidence from Finland and Germany (Preliminary version to be presented at
the IIOC 2004, Chicago, IL)
 Focus of this evaluative study:
 Summative question: Investigation whether public R&D subsidies in Finland and in
Germany have a positive impact on the innovation output (effects of public
incentives and R&D collaboration on innovative output of companies measured by
their patenting activity).
 Quantitative approach: Treatment effects analysis to assess whether policy and/or
collaboration yield a positive benefit in terms of patent activity, with a sample of
German an Finnish firms. Study applies an econometric matching taking a possible
selection bias into account.
page 10
IS
I
F
r
a
u
n
h
o
fe
rIn
s
titu
te
S
y
s
te
m
s
a
n
d
In
n
o
v
a
tio
n
R
e
s
e
a
r
c
h
Stefan Kuhlmann / Jakob Edler:
Tailor-made evaluation concepts for innovation policy learning
Analysis of public funding, collaboration & patent outcome
Descriptive statistics (based on CIS data)
Source: Czarnitzki, Ebersberger and Fier, 2004
Germany, N=1,464
Finland, N=1,520
Definition
Variable
Mean Std.d
Min.
Max. Mean Std.d.
Min.
Max.
Patent application (dummy)
PATENT
0.438 0.496
0
1 0.259 0.437
0
1
Employees in 1,000
EMP
0.312 0.401 0.011
2.5 0.182 0.214
Share of R&D employees
RDEMP
0.087 0.187
0
1 0.076 0.117
0
1
Patent stock (dummy)
LAGPAT
0.439 0.496
0
1 0.275 0.433
0
1
Export amount devided by turnov EXQU
0.242 0.239
0
1 0.342 0.314
0
1
Public funding (dummy)
FUND
0.208 0.406
0
1 0.483 0.500
0
1
Co-operation (dummy)
CO
0.287 0.452
0
1 0.643 0.479
0
1
Public fund. x Co-op. (dummy)
BOTH
0.110 0.313
0
1 0.386 0.487
0
1
Year 2000 (dummy)
YEAR
0.333 0.401
0
1 0.602 0.490
0
1
0.01 2.025
Note: The variables in the analysis also include 5 industry dummies (INDUSTRY) not reported here.
page 11
IS
I
F
r
a
u
n
h
o
fe
rIn
s
titu
te
S
y
s
te
m
s
a
n
d
In
n
o
v
a
tio
n
R
e
s
e
a
r
c
h
Stefan Kuhlmann / Jakob Edler:
Tailor-made evaluation concepts for innovation policy learning
Summative, quantitative poles – results of example
 Results for Germany:
 Public funding and collaboration (and both) lead to improved innovative
performance
 This hypothesis is not supported for firms that receive R&D subsidies for individual
research
 Results for Finland:
 Firms actually collaborating and receiving funding, would exhibit less patenting
activity if the goverment had not subsidized those firms
 In this case, firms might not be able to raise enough capital to maintain their high
innovation efforts
Source: Czarnitzki, Ebersberger and Fier, 2004
page 12
IS
I
F
r
a
u
n
h
o
fe
rIn
s
titu
te
S
y
s
te
m
s
a
n
d
In
n
o
v
a
tio
n
R
e
s
e
a
r
c
h
Stefan Kuhlmann / Jakob Edler:
Tailor-made evaluation concepts for innovation policy learning
Summative, quantitative poles – conclusions from example
 Quantitative summative evaluation provides information about relevant
measurable outputs and effects; information can be highly likely and quite
sophisticated
 Quantitative summative evaluation has only limited potential
 to explain causality of measured effects
 to explore other (indirect) effects, like 'behavioural additionality', learning
 A formative analysis/evaluation of economic and policy context would help to
understand differences and promising starting points for improved policies.
page 13
IS
I
F
r
a
u
n
h
o
fe
rIn
s
titu
te
S
y
s
te
m
s
a
n
d
In
n
o
v
a
tio
n
R
e
s
e
a
r
c
h
Stefan Kuhlmann / Jakob Edler:
Tailor-made evaluation concepts for innovation policy learning
Formative, qualitative
poles:
Innovation stakeholder
arena as context
National
research
centers
Multinational
companie
s
 Competition for funds
Evaluation ...
asResearch
formative
councils
learning
medium
Universities
SME
associations
 Differing interests,
perspectives and values
Contract
research
institutes
Industrial
associations
Consumer
groups
National
research
ministry
 No dominant player?
 Contested policies
 Need for alignment,
otherwise: exit
National
parliament
Regional
governments
Environment
groups
Other
national
ministries
EU
Commission
page 14
IS
I
F
r
a
u
n
h
o
fe
rIn
s
titu
te
S
y
s
te
m
s
a
n
d
In
n
o
v
a
tio
n
R
e
s
e
a
r
c
h
Stefan Kuhlmann / Jakob Edler:
Tailor-made evaluation concepts for innovation policy learning
Formative, qualitative poles - example:
Assessment of policy instruments supporting "competence centres"
 Recent evaluative study of Jakob Edler, S. Bührer, V. Lo, C. Rainfurth (Fraunhofer ISI)
and S. Sheikh (KMU Forschung Austria), Future of competence centre programmes (K
plus and K ind/net) and future of competence centres, Karlsruhe/Vienna 2003 (Study on
behalf of two Austrian Federal Ministries)
 Focus of this evaluative study:
 Formative question: Strategic advice with respect to the future development of two
competence centre support programmes (K plus and K ind/net): Differences of the
appropriateness of the two progs? Fit of the two progs' targets and implementation?
 (Prevailingly) qualitative approach: Evaluation as 'critical friend' of policymakers and
stakeholders, questioning policymakers' hypotheses and supporting
decisionmaking. Information base: 'Good guess' drawing upon structured
interviews, document analysis, structural data, survey of international policy
experiences.
page 15
IS
I
F
r
a
u
n
h
o
fe
rIn
s
titu
te
S
y
s
te
m
s
a
n
d
In
n
o
v
a
tio
n
R
e
s
e
a
r
c
h
Stefan Kuhlmann / Jakob Edler:
Tailor-made evaluation concepts for innovation policy learning
Key for evaluation:
understandings the basic concepts of the two progs …
 Cooperation of industry and science for research and innovation
 Assumption: cooperation too low
 Financial incentive for cooperation needed
 Additionality of support for cooperation
 Increase of R&D expenditure of companies
 More R&D results, more risk-taking, speeding-up
 Learn how to cooperate ('behavioural additionality')
 Public policy designed as multi-actor, multi-measures programme (MAP)
page 16
IS
I
F
r
a
u
n
h
o
fe
rIn
s
titu
te
S
y
s
te
m
s
a
n
d
In
n
o
v
a
tio
n
R
e
s
e
a
r
c
h
Stefan Kuhlmann / Jakob Edler:
Tailor-made evaluation concepts for innovation policy learning
Exploration and evaluation of policy rationales,
context and governance – two profiles
Driven by innovation
(Kind/net)
Driven by knowledge generation
(Kplus)
Type
Close to market
Close to basic research
Rationale
Project-oriented
Community of practice-oriented
Purpose of
participation (funding)
Overcome firm-internal
barriers for cooperative
market-oriented R&D
Creation of new cooperation
structures; upgrade and
broadening of research
Cooperation culture
Oriented towards well-known
partners
Oriented towards most
excellent partners
Time horizon
Short-term results
Medium-term, knowledge
creation
page 17
IS
I
F
r
a
u
n
h
o
fe
rIn
s
titu
te
S
y
s
te
m
s
a
n
d
In
n
o
v
a
tio
n
R
e
s
e
a
r
c
h
Stefan Kuhlmann / Jakob Edler:
Tailor-made evaluation concepts for innovation policy learning
Formative, qualitative poles – results of example
 Overall assessment: two different programme approaches justified, to be better
profiled
 Results and recommendations for Kind/net:
 Develop clear profile as innovation programme; adapt funding level (below
research funding)
 Improve programme management (e.g. transparency)
 Results and recommendations for Kplus:
 Provide stable funding and transparent rules
 Involve local authorities
 Extend inter-centre collaboration
page 18
IS
I
F
r
a
u
n
h
o
fe
rIn
s
titu
te
S
y
s
te
m
s
a
n
d
In
n
o
v
a
tio
n
R
e
s
e
a
r
c
h
Stefan Kuhlmann / Jakob Edler:
Tailor-made evaluation concepts for innovation policy learning
Formative, qualitative poles – conclusion from example
 Advanced innovation policy instruments are increasingly complex (MAP)
 Problem: strategic fit of policies – approach, instruments, implementation
 Formative evaluation as a source of strategic intelligence,
 providing evaluative inputs for reflexive, incremental policy-development
 needs qualitative understanding of rationales, context and governance
 including multiple perspectives of different actors and levels
 Formative, qualitative evaluation approaches are indispensable,
 quantitative and summative inputs (e.g. on outputs and performance) are very helpful
page 19
IS
I
F
r
a
u
n
h
o
fe
rIn
s
titu
te
S
y
s
te
m
s
a
n
d
In
n
o
v
a
tio
n
R
e
s
e
a
r
c
h
Stefan Kuhlmann / Jakob Edler:
Tailor-made evaluation concepts for innovation policy learning
General principles of strategic intelligence
 Principle of participation: strategic intelligence realises the multiplicity of actors’ and
stakeholders’ values and interests involved in innovation policymaking (multiple
perspective approach).
 Principle of "objectivisation": strategic intelligence "injects objectivised" information into
the policy arena, i.e. the results of policy/strategy evaluations, foresight exercises or
technology assessment, and also of analyses of changing innovation processes, of the
dynamics of changing research systems and changing functions of public policies.
 Principle of mediation and alignment: strategic intelligence facilitates debates and
"discourses" between contesting actors in related policy arenas, thus mediating and
"moderating", supported by "objectivised" information to be "digested" by the struggling
parties.
 Principle of decision support: strategic intelligence requires forums for negotiation and
the preparation of policy decisions.
page 20
IS
I
F
r
a
u
n
h
o
fe
rIn
s
titu
te
S
y
s
te
m
s
a
n
d
In
n
o
v
a
tio
n
R
e
s
e
a
r
c
h
Stefan Kuhlmann / Jakob Edler:
Tailor-made evaluation concepts for innovation policy learning
Contact:
s.kuhlmann@isi. fraunhofer.de
[email protected]
Info:
www.isi. fraunhofer.de
page 21
IS
I
F
r
a
u
n
h
o
fe
rIn
s
titu
te
S
y
s
te
m
s
a
n
d
In
n
o
v
a
tio
n
R
e
s
e
a
r
c
h
Stefan Kuhlmann / Jakob Edler:
Tailor-made evaluation concepts for innovation policy learning