After Evaluation:
Policy Consequences from Evaluations of
transinstitutional RTD Funding in Austria
Rupert Pichler
Wolfgang Hein
Federal Ministry for Transport, Innovation and Technology
Content
• Context
• Evaluation of FWF and FFF
• Assessment of Competence Centre Programmes (CCPs)
• matching political motivation and ToRs
• findings, recommendations, political consequences and role of
external players
• what remains of the evaluations and why
Intervention + Context = Impact (Pawson/Tilley, 1997)
September 2005
Context
• major evaluations of Austrian RTD system completed
• changing needs and challenges
• key factor of changes is spread of multi-actor programmes and
PPP models through funding institutions
• appropriateness of institutional and regulatory framework as
underlying key issue
•
•
•
•
How are these factors reflected in terms of reference?
Do evaluators get into the feed back loops of policy-making?
How much of the results gets inevitably disposed of?
What prevails in the end, and why?
September 2005
FWF/ FFF: Political Motivation
• design of the two largest funding institutions stemming from the
1960s, remaining virtually unchanged
• obviously no incentive for substantial change by new paradigms of
the 1980s and 90s (collaborative technology programmes, EU FP)
• political pressure for simplification of funding structures added to
the funds since the late 80s
• no evaluation ever carried out in the funds’ history, respective
recommendation of R&D council
• at the same time growing number of evaluations of programmes
implemented outside the funds
September 2005
FWF/FFF: Reflection in ToRs
• evaluation commissioned by BMVIT alone, though reform efforts
imminent at that time are supported by a multitude of players
• BMVIT under pressure to justify efficiency and effectiveness of
existing funding institutions, thus trying to safeguard its influence
• at the same time examination by court of auditors pending which is
clearly understood as to have role in decision making
• positioning of the funds within a future scenario clearly addressed
in ToRs
• design of ToRs outsourced to ISI in light of political weight and
financial dimension
• contractors made aware of “killing fields”
September 2005
FWF/FFF: ToRs – Central Issues
• Are the instruments, procedures and structures adopted by the
funds – according to their mission – appropriate to support the
investigational and innovational behaviour of the relevant actors in
an efficent and effective way?
• What is the position of the two funds in the national, international
and especially European science and innovation system and what
recommendations can be made for future strategies?
September 2005
FWF/FFF: ToRs – specific questions 1/2
• General Approach
Are the objectives and the strategic orientation of the funds appropriate to
pursue the intended effects of research funds? What is the impact of the funds
activities on the corresponding science system and industrial RTDI? How are
the funds positioned in comparison with corresponding international funds?
• Performance/ Management
Do the funds employ appopriate procedures to secure the quality of the
supported project? What are the strengths and weaknesses in the performance of
the funds?
September 2005
FWF/FFF: ToRs – specific questions 2/2
• Cooperation
Is the coordination of the funds with other national research and funding
instruments and institutions suitable to realise feasible synergies? How do the
funds coordinate, cooperate and communicate with the relevant actors of the
science and innovation system, especially with each other and the
corresponding ministries?
• Internationalisation
What are the strategies of the funds to secure their positioning and integration in
the European research and innovation system?
September 2005
FWF/FFF: Findings 1/3 – political context
• beside in-depth analysis synthesis report tackles policy related
questions at length based on the detailed results
• however, intense integration into decision-making process
achieved rather coincidentally while external expertise from court
of auditors competes for political attention
• involvement of evaluators into discussions with reform
stakeholders paves the way to substantial influence
• nonetheless, for a longer time during the process some actors
remain unsure about the exercise’s political usefulness
• multitude of stakeholders makes it easier for evaluators to become
a voice that is heard
September 2005
FWF/FFF: Findings 2/3 – major issues
• merger of the two funds as advocated by substantial stakeholders
not recommended, to the contrary merger of FFF with TIG, ASA,
and BIT
• organical expansion of FWF’s role to tackle use-oriented and
thematic research; a substantial budget increase to cope with this
progress and higher overheads on research grants in future
• both funds require additional resources to generate strategic
intelligence and their own strategies, strengthen their international
roles and improve their communication within the Austrian
funding system
September 2005
FWF/FFF: Findings 3/3 – major issues
• the governance of both funds was criticised as being close to
self-serving with too little strategic capacities, leading to a role
model fairly isolated within the system
• the funds should be transformed into agencies and the power of
their beneficiaries in the governance structures should be limited
• the instruments of both funds need to be improved (more risk,
more networks, simplification)
• the strategic intelligence of both funds was found to weak in
order to cope with implementation of the government’s strategic
goals
• surprisingly little windfall benefits of FFF funding
September 2005
FWF/FFF: Recommendations 1/2
• business of ministries is to make policy, and of agencies to
implement it – each needs to develop strategies at their own level
• the division of labour between ministries and agencies needs to
be transparent and modern
• establishing clear performance contracts between the ministries
and agencies in terms of objectives and how and when they are
measured
• correspondingly, it means absolutely forbidding interference
from the policy or political level in operational matters such as
personnel and project decisions
September 2005
FWF/FFF: Recommendations 2/2
• ministries need to retain enough analytic capability to support their
policy needs
• responding to the policy needs of the ministries the agencies should
propose, design and implement programmes and other instruments
in the long run
• agencies should be able to develop strategies to obtain economies
of scope as well as scale, tackling policy needs of multiple
ministries
• beneficiaries must be important in the governance of agencies, but
not have overall control, which should be exercised by a wider
range of stakeholders on behalf of the taxpayers
September 2005
FWF/FFF: Political consequences
• the merger of the two funds was taken off the agenda after the
evaluation had supplied ammunition to the opponents of the
merger
• the FWF’s governing institutions underwent reform in order to
exclude beneficiaries and institutional stakeholders from
decision-making
• FFF was merged into new FFG (intended anyway) which is a
“normal” state-owned company; FFF lost its autonomy
• in practice, FFF mechanisms still exist as a part of FFG while a
bigger debate over ex-FFF is on hold since it touches sensitive
ground
• while FFG struggles to become a one-stop-shop part of the old
system survives for the sake of continuity
September 2005
FWF/FFG: Remains
• eventually, the evaluation’s expertise prevails over the court of auditors
• beyond, a general willingness to devote attention to non-consultant
style advice became obvious
• the in-depth parts still hold treasures yet to be made use of
• a surprisingly comprehensive debate on role of politics evolved over
the issues of “autonomy” and “self-governance” and led to partial
fulfillment of ambitions to regain political influence as a consequence
of political responsibility
• hypothesis: great visibility of an evaluation exercise can be achieved
when a multitude of equal stakeholders is involved, none of which
being in full possession of the process but each trying to get attention
for own interests; this makes it easier for evaluators to play a role in
decision-making as long as equal distance is kept to all parties
September 2005
CCPs: Political Motivation 1/2
• substantial dimensions and visibility of competence centre
programmes (aimed at bridging the gap between industrial
and basic research) reached in 2003
• two co-existing programmes under responsibility of two
ministries
• funding terms of first centres nearing the end
• widespread discontent with “competition” among ministries
• strikingly different implementation
September 2005
CCPs: Political Motivation 2/2
• development of perspectives obviously necessary
• justification of future investment
• though commissioned jointly by BMVIT and BMWA hidden
agenda develops
• common interest: resist outside pressure to merge and lose identity
• concealed expectations to emerge as “winner” of the evaluator’s
judgement and therefore secure a favourable position for
upcoming battles
September 2005
CCPs: Reflection in ToRs 1/3
support and assistance for decision-making processes
proposals and analysis for strategic orientation
IS
I
F
r
a
u
n
h
o
fe
rIn
s
titu
t
S
y
s
te
m
te
c
h
n
iku
n
d
In
n
o
v
a
tio
n
s
fo
rs
c
h
u
n
g
What kind of progress
Competence Centre Programmes
will probably make in the future?
September 2005
What are the future prospects
of existing
Competence Centers?
CCPs: Reflection in ToRs 2/3
• General approach
Does the conception of the programme correspond to the problem analysis? Is
there missing cooperation between industry and science in research and
innovation projects? Does the programme succeed in achieving its objectives?
What is the impact of the funding activities on the corresponding science
system and industrial R&D?
• Management/ Performance
Is the implementation of the programme efficent? Are the programmes´
instruments, procedures and structures appropriate to the objectives of the
programmes and the needs to be funded?
September 2005
CCPs: Reflection in ToRs 3/3
Perspectives of the CCPs
• curiosity whether or not an empirical justification for two different
programmes could be determined
• vested interests to provoke a quality ranking not expressed in ToRs, though
obvious to the evaluators
Perspectives of existing CCs
• pressing need for ideas about the CCs’ future thoroughly reflected
• Will the centres continue to carry out long-term precompetitive research and
development of high academic and economic relevance?
• Should the centres be restructured and if so in which way?
September 2005
CCPs: Findings 1/2
• nature and dimension of demand for CCPs remains to be
determined
• both programmes address existing specific needs
• implementation is largely consistent with goals, but room for
improvement
• K plus should remain science orientated; appropriate current
funding budget;
• K ind/net should be distinguished as innovation-orientated
programme; decrease of the funding rate; more transparency of the
funding criteria; aspiration for cooperation
• both programmes should provide incentives to create synergies
between the different CCs; cooperation is limited and programme
management should enhance networking activities
September 2005
CCPs: Findings 2/2
four models for the CCs’ future were specified
• privatisation
• termination
• reinstallation in future CCP calls
• institution building
- common platform with public research organisations
- close cooperation between the CCs and universities
- establishing stand alones, possibly through mergers of CCs
September 2005
CCPs: Role of other players
• decision-making on CCPs exceeds the realm of those
commissioning the assessment (BMVIT and BMWA)
• some interests underestimated (Länder, pressure for
institutionalization, R&D council, universities)
• occasionally open criticism is voiced (e.g. concerning comments
on implementation standards)
• at the same time, a soft-spoken OECD study is at hand
September 2005
CCPs: Political consequences
• great relief, no pressure from expert opinions to end seemingly mere political
duplication of programmes
• it is not believed in earnest that the policy field as such is at stake
• the models are immediately fed into ongoing negotiations with other stake
holders by BMVIT
• immediately after completion: assessment as valuable input for negotiations with
Länder, universities and industry
• one year later: design process for CC future (mainly in BMVIT domain) about to
disintegrate, thus reducing the assessment’s immediate impact
• two years later: unexpectedly, a “grand” solution is underway and a short-term
financial aid for outgoing CCs agreed; some of the assement’s implications are
turned upside down
September 2005
CCPs: Remains
• commissioning policy makers never in full possession of the decision process
• awareness of that process and close involvement of evaluators enhances the
exercise’s weight even though by doing so some standard rules are ignored
• general awareness for real need for strategic decisions beyond mere political
interests significantly increased
• input on implementation valid and recognized
• struggle over financial questions concerning the future of existing CCs triggers
agreement to pursue a complete and even more ambitious relaunch which was
not considered an option at the time of the assessment because of apparent
priority of short-term needs
• hypothesis: the more relevant an issue evaluated and the more honest the interest
in the exercise, the greater the probability that findings – even if concrete
recommendations were asked for – serve only as “germs” for further process,
thus eventually achieving powerful effects
September 2005