Disclaimer
The views and comments expressed in this text are the responsibility of Deloitte &
Touche and do not necessarily reflect the opinion of the European Commission.
Acknowledgements
The production of a report of this scope would not have been possible without the
effort, contribution and support of a large number of organisations, experts and
individuals.
This report was written by Isabelle Collins, Richard Doherty, Christopher Allinson,
Sarah Batterbury, and Wouter Van Den Berghe. Responsibility for its content is solely
with the authors.
The authors wish to acknowledge the commitment and professionalism of the expert
team who provided area-based reports, case studies, and valuable inputs into the final
report.
♦
♦
♦
♦
♦
♦
♦
♦
♦
Hanne Shapiro
Paddy Carpenter
Lionel Chaty
Peter Debreczeni
Pedro das Neves
Prof Theo Papatheodossiou
Anna Gammaldi
Gabriele Marchl
Anne Marie De Jonghe
The authors also acknowledge the contribution of Nathalie Belnet (Project Assistant)
during the early stages of the evaluation.
Valuable insights and direction were provided by representatives of the European
Commission - DG Education & Culture and DG Employment & Social Affairs, the
Expert Group on the External Evaluation, the Committee of the Leonardo da Vinci
Programme, Member State Officials, National Co-ordination Units, and CEDEFOP.
The authors would like to thank these individuals for giving freely of their time and
making the evaluation possible.
The production of this report would also not have been possible without the effort of
the many anonymous respondents from the Pilot projects, Mobility projects and
Exchange participants themselves. The authors would like to thank all of the many
individuals across Europe who responded to questionnaires and gave up their time to
contribute in this way to the evaluation.
Deloitte & Touche
1
Executive Summary ...............................................................................................4
1.1 Opening comments ............................................................................................4
1.2 Background and introduction.............................................................................4
1.3 Policy Context....................................................................................................5
1.4 Complementarity................................................................................................6
1.5 Implementation ..................................................................................................6
1.6 Outputs...............................................................................................................7
1.7 Impact ................................................................................................................7
1.8 Conclusions and recommendations....................................................................8
1.9 Overall Considerations.....................................................................................12
2
Background and Introduction ..............................................................................13
2.1 Genesis of Leonardo da Vinci 1.......................................................................13
2.2 Scope of this evaluation ...................................................................................13
2.3 The consultant..................................................................................................15
2.4 Methodological Approach ...............................................................................15
3
Vocational Education & Training Policy Context of Leonardo da Vinci I .........24
3.1 Introduction......................................................................................................24
3.2 The origins of the policy framework ...............................................................25
3.3 Chronology of vocational training and relevant education action programmes
1986-1993 ................................................................................................................26
3.4 Clarification of the legal aspects of vocational training policy .......................27
3.5 Vocational training - EU fora ..........................................................................28
3.6 Other Policy Influences....................................................................................28
3.7 Other EU Policy Influences .............................................................................30
3.8 Launch of Leonardo da Vinci ..........................................................................31
4
Complementarity with other EU instruments ......................................................35
4.1 Relationship with previous programmes .........................................................35
4.2 Relationship with other Community interventions ..........................................36
4.3 Duplication or synergy/value added ................................................................43
4.4 Take-up of lessons by direct funding programmes..........................................45
5
Implementation ....................................................................................................46
5.1 Structures and Procedures................................................................................46
5.2 Project selection and management...................................................................52
5.3 Problem Areas..................................................................................................58
Final Evaluation of Leonardo da Vinci
2
Deloitte & Touche
5.4 Costs, benefits and value for money ................................................................61
5.5 The Interim Evaluation ....................................................................................62
6
Outputs.................................................................................................................63
6.1 What were the outputs .....................................................................................63
6.2 Outputs recorded centrally...............................................................................63
Outputs recorded at national level ...........................................................................72
6.4 Quality of outputs ............................................................................................73
6.5 Products - Dissemination .................................................................................73
7
Impact ..................................................................................................................78
7.1 Approach..........................................................................................................78
7.2 Impact of Leonardo at the EU and national levels...........................................79
7.3 Impact of Pilot Projects....................................................................................79
7.4 Impact of the Mobility Projects .......................................................................85
7.5 Impact for the Participants ...............................................................................86
7.6 Impact in Central and Eastern Europe .............................................................88
7.7 Impact in respect of Equal Opportunities ........................................................90
7.8 Impact of specialist strands..............................................................................91
7.9 Impact with respect to the Social Partners.......................................................92
7.10
8
Impact with respect to sectors......................................................................93
Conclusions and Recommendations ....................................................................96
8.1 The Programme’s objectives revisited.............................................................96
8.2 Types of action.................................................................................................97
8.3 Complementarity..............................................................................................99
8.4 Implementation ..............................................................................................100
8.5 Outputs and Impact ........................................................................................104
8.6 A summary.....................................................................................................107
8.7 Overall............................................................................................................107
8.8 Specific Objectives of the evaluation ............................................................108
Final Evaluation of Leonardo da Vinci
3
Deloitte & Touche
1 Executive Summary
1.1 Opening comments
This evaluation was carried out to assess whether the Leonardo da Vinci Programme
contributed effectively to the achievement of its objectives, chiefly development of
EU vocational education and training policy. In particular the evaluation aims to
contribute to the final report on the programme, which the Commission will present to
the European Parliament, the Council, the Economic and Social Committee, and the
Advisory Committee on Vocational Training later in 2000.
The evaluation is based on the national reports from the Member States, area–based
expert reports; published material from the Commission and from CEDEFOP; and
internal Commission documents. The report is also based on interviews with key
stakeholders and questionnaires sent to pilot and mobility projects and exchange and
placement participants. It therefore represents a process of extensive consultation and
has been shaped and guided by individuals in the expert group, the Leonardo
Committee and our clients in DG Education and Culture.
This document presents an evaluation of the programme’s overall implementation and
results, but is not intended to represent a financial audit of the workings of the
programme.
This Executive Summary contains:
1. Background and introduction to the evaluation
2. An overview of the policy context of Leonardo da Vinci
3. An overview of the complementarity of Leonardo with other EU Instruments
4. An analysis of the key implementation issues which shaped the impact and
delivery of the Leonardo programme
5. A discussion of the key outputs of the programme
6. An analysis of the impact of Leonardo – with particular emphasis being directed
to experiences of the programme for participants and the impact for social
partners, SMEs and in Eastern and central Europe and at sectoral level.
7. Conclusions and recommendations
This format is also followed in the main report.
1.2 Background and introduction
The Leonardo da Vinci programme was adopted for 5 years in December 1994 and
had a key objective in supporting the development of policies and innovation action in
the Member States by promoting projects in the context of trans-national partnerships
which involve different organisations with an interest in training. Leonardo had a
budget of 620 million ECU and was open to the 15 Member States and countries from
the rest of the European Economic Area (Iceland, Liechtenstein and Norway), plus
Cyprus, the Czech Republic, Estonia, Hungary, Lithuania, Latvia, Romania, Poland,
Bulgaria, Slovenia, and the Slovak Republic.
Leonardo da Vinci 1 was established with 19 different objectives in 1995. The
programme's objectives were very broadly drawn and the result was a programme that
Final Evaluation of Leonardo da Vinci
4
Deloitte & Touche
reflected the wide differences in training policy, systems, priorities and resource
levels that characterise the European training scene. This is even more marked in the
case of the pre-accession countries. Accordingly, drawing conclusions in the form of a
European synthesis presents difficulties, and some of the report's detailed discussions
are more relevant to individual, or a small group of, participant countries.
The specific objectives of the evaluation were to evaluate the Leonardo da Vinci 1
programme and provide advice and counselling to the European Commission on five
key issues:
•
the outputs and impacts (results and outcomes) of Leonardo da Vinci 1
•
the effectiveness of the programme in terms of achievement of its own general,
specific and operational objectives;
•
the efficiency of Leonardo da Vinci 1 in terms of management and delivery
structure;
•
Mechanisms for the improvement of the effectiveness and efficiency of Leonardo
da Vinci 1 in future programmes;
•
Mechanisms for the development of complementarity and synergy with other
vocational education and training instruments and policies both at the EU and
Member State levels.
1.3 Policy Context
The Leonardo programme has been formulated in a dynamic policy environment,
consisting of a variety of actions with potential for synergy. The life span of the
Leonardo da Vinci 1 programme coincided with a period of evolution and maturation
of the policies and principles in respect of lifelong learning and its relationship with
the maintenance and growth of employment in the EU. The policy context existing at
the launch of Leonardo da Vinci 1 in 1995 was rather less focused than it was by
1999.
In the run up to the elaboration of Leonardo a debate emerged about the legal base for
European-level vocational training policy. This was resolved in the Maastricht Treaty
in 1992. A Community vocational training policy was introduced, explicitly designed
to support and supplement the vocational training activities of the Member States.
There was, however, no provision for harmonisation at the EU level.
Alongside the development of the legal policy framework concerning vocational
training, a number of European level fora concerned with vocational training policy
and practice have been operating largely independently of each other. The profile of
vocational training in the EU policy arena was also raised by white papers, activities
such as the European year of Lifelong Learning and of course the broader
developments in employment and Information Society policy.
Leonardo was launched in 1995 in this context. It was developed from the
rationalisation of a number of former action programmes: COMETT, PETRA,
LINGUA, EUROTECNET, FORCE, and HELIOS. The 19 different objectives with
which Leonardo da Vinci 1 began were a cumulative listing of aspirations and
principles inherited from this group of preceding programmes. This reflected the
absence of a well-planned and thoughtful approach to a complementary EuropeanFinal Evaluation of Leonardo da Vinci
5
Deloitte & Touche
level contribution to policy-making at a time of change, and the lack of a clear
hierarchy of objectives in Leonardo made implementation problematic.
1.4 Complementarity
At a conceptual level the Leonardo programme was complementary to other EU
programmes but this was almost as a result of accident rather than careful planning.
The different EU programmes tend to operate in parallel rather than in an integrated
fashion.
Leonardo’s distinctiveness vis-à-vis other European programmes can be found in its
provision of opportunities for active testing of innovative ideas through transnational
projects, and it making possible of large-scale mobility programmes for trainees.
The achievement of complementarity between EU interventions could be enhanced if
they were conceived as part of a continuum, or a hierarchy of interventions, which are
complementary to one another in the policy process. Such an approach would enhance
the value of each programme’s contribution by placing it in an integrated context. It
would also illuminate the potential for cross–programme knowledge sharing and
dissemination of good practice.
Continuity between programmes would entail the establishment and utilisation of
information channels between programmes and related policies. This is essential for
optimising knowledge management strategies within the overall vocational
educational and training area.
1.5 Implementation
The Leonardo programme is extremely complex in its formulation and in its
implementation. Many of these complexities are embedded in the Council Decision
establishing the Programme, while others are related to the management systems of
the European Commission. A large proportion of the administrative work was
externalised to a Technical Assistance Office in Brussels and to National Coordination Units in the participating countries. During the course of the programme
there were a number of events which culminated in the closure of the TAO and the
transfer of its functions to a unit within the Commission - the Cellule Leonardo
(CLEO). The underlying principles and processes did not change fundamentally with
the change from the TAO to CLEO.
The complexity of the programme’s structure caused problems for potential promoters
in identifying the appropriate procedures and strands for their projects.
Documentation was also hard to follow and not user-friendly. The requirement for
projects to form partnerships may have biased project selection in favour of projects
with pre-existing networks. The financial circuits for centrally administered projects
were immensely long. This clearly discriminated against SMEs and non-profit
organisations. The co-funding regime was also extremely onerous. Serious questions
arose about the capacity of the Commission to process the number of payments
involved within a reasonable time-scale.
Project selection took a period of almost one year between the publication of the call
and notification of the successful projects. The programme was not able to respond to
changes which took place for the projects in this period. Projects also complain about
the lack of feedback received during and after the process. Backdating contracts was
Final Evaluation of Leonardo da Vinci
6
Deloitte & Touche
also not conducive to project implementation. In addition, no mechanisms were
established to take forward the lessons from the programme at European level. There
was no overall strategy to use the outputs of the programme.
1.6 Outputs
This evaluation uses the term outputs to describe the activities that took place in
Leonardo, and the products associated with the pilot projects. The emphasis on
detailed quantitative reporting on activities at the European level appears to have
resulted in downplaying systematic appraisal of the quality of the Leonardo processes,
of the personal and experiences, or of the products. Our research has indicated that
there was significant satisfaction with the opportunities within the programme for
establishing organisational, networks and European information exchange and with
the personal development opportunities for the participants.
Leonardo suffered major difficulties with the collection and dissemination of outputs,
however. The Commission’s ambitions in this area have been frustrated by the
programme’s emphasis on managerialism and unnecessary bureaucracy and by the
collapse of the TAO.
Leonardo therefore lacked a Europe-wide approach to product dissemination, a
catalogue of products, the provision of good practice material and guidance,
marketing events, and opportunities to build on the potential links with the European
Employment Strategy.
1.7 Impact
The impact of Leonardo extends beyond what can be quantified and measured. It is
equally conditioned by the receptivity of the implementation environment as much as
by the design and quality of the programme. Impact is also heavily contextually
determined and reflects the complex interaction between programme setting and
design in specific places and contexts.
The projects were largely satisfied that their expectations had been met, although
there was widespread dissatisfaction with the delays. In many instances this led to
comments that projects would not be willing to participate in Leonardo in the future.
Trans-nationality and partnerships were welcomed but there is evidence that the
learning curve for the operation of partnerships was fairly steep for the projects.
Evidence of organisational learning may be grouped around:
•
The acquisition of new, substantive, work related skills,
•
Learning about inter-organisational relations,
•
Administrative learning centred on managing complex administrative procedures
and multiple teams from different countries and organisations.
Exchange and placement participants also report widespread positive impact in self
development as the result of their involvement in Leonardo. Both participants and
projects have been increasingly sensitised to the added value that can be obtained by
being open to learning from other parts of Europe. Leonardo has therefore been an
educational experience contributing to the process of Europeanisation of civil society
and raising the level of European consciousness.
Final Evaluation of Leonardo da Vinci
7
Deloitte & Touche
However, it is too early to assess the impact that Leonardo may have had in policy
terms and on the training systems at national and European level. Many projects have
not yet finished and no real assessment has been carried out of the products that have
been delivered. There is no evidence that the Leonardo Committee or the
Commission have engaged in serious analysis or debate of the practical and policy
lessons that undoubtedly have emerged through the projects.
The failure to do this is partly a consequence of inadequate resources and an
inappropriate focus on administration and control with the Commission (to the
detriment of content-related activities). It is also symptomatic of the relative under
development of the European policy system in this area, and is a consequence of the
programme’s lack of clarity about its focus. Was it a demand given funding machine
allowing practitioners to experiment on interesting themes, or was it a co-ordinated
exercise in developing policy through focused funding of agreed priority actions?
The former is much truer than the latter.
No serious attention has yet been given to ensuring that the results will be processed
and considered at European level. Some evidence exists that Member States have
tried to bring national results and lessons together but without direct attributable
impact on policy and or practice at least at this stage.
1.8 Conclusions and recommendations
We have grouped the principal conclusions and recommendations for the report
around the three main themes – complementarity, implementation, outputs and
impact.
1.8.1 Complementarity
Leonardo is embedded in a range of potentially complementary EU policies in
vocational education and training and social integration. The EU needs to
operationalise and optimise the links. Providing the opportunities for the exchange of
best practice and lessons between programmes and departments would entail the
provision of resources and the installation of mechanisms for the flow of information
between programmes.
Although Leonardo did not necessarily achieve great synergy between ministries at
national level, it did begin a process in some Member States which may yield fruit in
the future.
Recommendations: Complementarity
•
High-level political commitment to strategic integration of the whole range of
current EU training activities.
•
Commit to a policy continuum approach involving research, field testing, and
mainstream delivery, to replace current uncoordinated EU interventions.
•
Dissemination of best practices across other EU programmes should also be
undertaken by the establishment of an effective knowledge management strategy
Final Evaluation of Leonardo da Vinci
8
Deloitte & Touche
1.8.2 Implementation
The implementation of Leonardo has arguably been the most problematic area of the
programme. The programme suffered from a number of structural and design
difficulties which impeded effective programme implementation. Both programme
design factors and the delivery procedures and mechanisms shaped the
implementation outcomes.
Programme Design Recommendations
•
A needs analysis should be undertaken to ensure that the programme is
accessible to the target group.
•
Simplification of all procedures is urgently required. Efficient payment systems
must be developed.
•
Administrative practices should be tested early in the programme, changed as
required in the light of experience
•
The actions for mobility projects need to be put into a context where they can
contribute to policy ideas.
•
Dissemination of project results and best practices is essential.
•
Ex-ante targeting needs to be increased.
•
Formative utilisation evaluations should be introduced.
•
Information systems should be defined from the start of the programme to meet
the potential evaluation and exploitation needs, as well as the administrative
processes.
•
A proper monitoring system which is focused on outputs and outcomes and their
use needs to be developed.
The programme was successful in terms of provision of an important tutelage function
and an information brokerage service for the projects. However, the opportunity for
strategic European planning by the Leonardo Committee was lost, and there was no
other effective policy channel for the review of results and the capitalisation thereof.
Leonardo exhibited some over-rigidity, symptomatic of a lack of both organisational
learning and knowledge management. There was a tendency to over-regulate aspects
of the programme. Which was also a constraint on the implementation capacity of the
programme. The programme objectives were lost sight of and were substituted by an
over-emphasis on procedures and the administrative process. Inter-organisational
boundaries between the TAO and the Commission and between administering bodies
and projects were unclear, as they were not adequately and clearly defined.
Decisions about the allocation of resources within the TAO did not always reflect the
existing administration requirements - insufficient resources were allocated to project
selection and proposal management for example. The financial arrangements for the
Final Evaluation of Leonardo da Vinci
9
Deloitte & Touche
TAO, whereby profit was effectively excluded, were extremely questionable and
risky. This was an important factor impacting on the operability of the TAO.
Participants were effectively discouraged from seeking to participate in Leonardo
again. There was no real interest in their environment - the programme suffered from
a very inappropriate management system and there was no strategy for the crisis when
it came. The system was stacked against the supposed target groups. Vicious circles
were set up, such as budget reductions that interfered with the successful completion
of the project work programmes, and setting unrealistic project deadlines as a result of
late starts. For pilot and mobility projects (under selection procedure II) project
quality was secondary to geographic spread.
Recommendations: programme delivery
•
Build on the positive aspects, be willing to revise and modify goals. Recognise
the natural evolution of programme life-course.
•
Succession planning for the post-Leonardo continuation of projects should be an
integral part of project planning and approval.
•
A crisis management strategy should be in place.
•
The roles of the different bodies should ideally be defined through the setting up
of guidelines and agreed conventions.
•
A system for quality management and procedures needs to be developed.
•
In recognition of the partnership character of the projects, selection should not
necessarily be based on the nationality of the lead partner alone.
•
For Leonardo to be a true European programme the Member States must operate
to agreed European guidelines in the selection procedures.
•
Incentive (profit) is necessary for limiting risk, and enhancing motivation and
quality in outsourced management activities. This should be considered when
awarding future management contracts.
1.8.3 Outputs and Impact
Outputs and impact are two aspects of the policy cycle that are intrinsically related
and so the two sets of findings are discussed together. Apart from the fact that many
projects are not yet finished, and that impact could take years to emerge, impact is
difficult to measure because of the absence of necessary data about products and
achievements. The need for this should have been identified at the outset of the
programme. Impact has been conditioned by the organisational dynamics and by the
structural programme characteristics which helped shape the implementation
environment and subsequent attainable impact for the programme.
A number of positive impacts of Leonardo can be identified:
• Project participants created informal networks, initiated learning about partnership
operation and learned about participating in European programmes. Evidence of
organisational learning was identified.
Final Evaluation of Leonardo da Vinci
10
Deloitte & Touche
•
The statistics strand was very useful: the resources provided allowed real progress
to be achieved.
•
The programme allowed space for the projects to remain at the forefront of
technological advances, to adapt to changing market needs, and to allow the
projects to become a platform for innovation. This was more by luck than design,
however.
•
The trans-nationality of the programme contributed to the Europeanisation of civil
society and the growth of European consciousness. A wide network of transnational contacts was built up allowing the exchange of skills, cultures and
learning.
•
The programme provided a useful boost for change in Central and Eastern Europe.
•
Leonardo was responsible for initiation of a number of projects which otherwise
could not have come into existence.
However, the evaluation has also found that the programme outputs and impact were
limited as the result of:
•
•
•
•
•
No attention to policy lessons by Commission of Leonardo Committee.
Absent monitoring of outputs and products and absent dissemination strategies.
Failure to capitalise on project results.
The delays experienced during the programme which served to exclude some
target groups as beneficiaries and constrain the innovativeness of many projects.
The surveys and analyses strand being isolated from the rest of the programme.
Recommendations: Outputs and Impact
•
Data on project services and products should be collected and best practices
disseminated.
•
Mechanisms for ensuring quality of the products should be put in place.
•
Programme design should be transparent to users.
•
Important best practices must be captured, innovative trends pointed up and
strategies for follow through put in place. This is especially important for projects
of significant potential and value.
•
Future Leonardo programmes will need to capitalise on organisational learning
and participant learning through an effective dissemination and knowledge
management strategy. .
•
The eastern and central European experiences must be built on where they were
positive.
•
Appropriate data collection is of critical importance for the subsequent
measurement of impact. Baseline data needs to be collected and performance
measures established.
•
Member States and the Commission should establish a formal mechanism for
assessing results and processing them thereafter.
Final Evaluation of Leonardo da Vinci
11
Deloitte & Touche
1.9 Overall Considerations
We have set out above the recommendations arising from the conclusions of this
evaluation. In general terms we argue that future Leonardo programmes would
benefit from a much greater user-focus.
In the area of complementarity, output analysis and dissemination there is a need for
considerable change. CEDEFOP, the ETF, and new internet-based initiatives could
play a useful role in drawing all of the relevant material together (not just from
Leonardo 2).
We also propose the convening of high level workshops to disseminate key findings
and to roll-out best practice to other related EU programmes. The complementary
links between Leonardo and other programmes should be officially co-ordinated.
Complementarity entails recognition that EU actions in this domain area are
systematically related and need to be integrated at the European level and not just at
the micro level by individual participants and projects.
A high-level think tank, with the imprimatur of the Member States, could rise above
some of the management and allocation functions that the Leonardo Committee
carried out, and set an integrated course for the whole range of training activities that
the EU is now engaged in. This would encompass the selection of major themes for
policy development, research, demonstration projects, events and so on. The
continuum of programme styles could be harnessed to give real power to this work
and to allow each area of Community intervention to contribute in an integrated way.
With regard to evaluation, we strongly urge the establishment of proper monitoring
and data storage systems. In addition - given the need to tailor the programme to the
needs of the target groups and to ensure that evaluation findings are relevant and
would advocate the establishment of formative evaluation structures. This would
entail the decentralisation of the evaluation process to more broadly include the
relevant stakeholders. The Expert Group that followed this evaluation would provide
a good forum for this as it comprises a fairly representative cross-section of
stakeholders, broadening the group to include other important stakeholders would be
helpful, if it were to take on this role.
Final Evaluation of Leonardo da Vinci
12
Deloitte & Touche
2 Background and Introduction
2.1
Genesis of Leonardo da Vinci 1
On December 6, 1994, the Council of Ministers of the European Union adopted the
Leonardo da Vinci programme for the implementation of a Community vocational
training policy (Official Journal L 34O, 29 December 1994).
This programme, adopted for a period of five years (1995-1999) had a key objective
of supporting the development of policies and innovative action in the Member States,
by promoting projects in the context of transnational partnerships which involve
different organisations with an interest in training.
The programme had a total budget of 620 million ECU for the five years and was
open to the 15 Member States, the 3 EFTA States of the European Economic Area
(Iceland, Liechtenstein and Norway), and progressively to; Cyprus, the Czech
Republic, Estonia, Hungary, Lithuania, Latvia, Romania, Poland, Bulgaria, Slovenia,
and the Slovak Republic.
The adoption of the Leonardo da Vinci programme was also aimed at a rationalisation
of Community action in the area of vocational training, providing the basis to enhance
the value of the acquis. Leonardo da Vinci was to facilitate the taking forward of
initiatives successfully developed under COMETT, PETRA, FORCE, LINGUA and
EUROTECNET, and add new dimensions.
The programme came at a time when the Commission’s White Paper on "Growth,
Competitiveness and Employment" forcefully emphasised the crucial importance of
vocational training as a key factor in combating unemployment and strengthening the
competitiveness of European enterprises.
The programme aimed at responding to the demand for new skill needs which are
generated by the evolution of our societies and linking training firmly to solving the
problem of employment in Europe. This was echoed in the White Paper "Teaching
and Learning - Towards the Learning Society", approved by the Commission on
November 29, 1995.
2.2 Scope of this evaluation
The bases for our work were the terms of reference issued by the Commission in May
1999 in conjunction with an open call for tender, adapted and complemented by a
series of discussions with the Commission and its expert group.
This evaluation took place after the disturbance to the programme caused by the
closure of the Technical Assistance Office and the accompanying allegations. Our
mandate was to carry out an evaluation of the programme’s overall implementation
and results, but - as is the nature of an evaluation - not to attempt to perform another
financial audit of the workings of the programme. Nevertheless, we do consider the
role and performance of the TAO over time, and inevitably we also look at the
consequences of its closure for the delivery of the last year of the programme.
The evaluation report provides a mixture of factual information and evaluative
conclusions. This is based on a form of methodological triangulation - in built crosschecks – by employing both qualitative and quantitative instruments designed to
Final Evaluation of Leonardo da Vinci
13
Deloitte & Touche
ensure credibility, rigour and reliability of the findings. It has been produced after a
thorough programme of documentary research, reviews of key output documents,
discussions with representatives of the European Commission, CEDEFOP, Member
States authorities and the National Co-ordination Units (NCUs), as well as a series of
visits to case study projects. Our analysis and synthesis stage draws together these key
findings and has been critical in the generation of conceptual insights into the
functioning and operation of the programme. Our report is structured in such a way as
to use all of these inputs in order to reach the conclusions and output that we offer.
2.2.1 General objective of the evaluation
The general objective of the evaluation was to assess whether the Leonardo da Vinci
Programme contributed effectively to the development of EU vocational education
and training policy. In particular it aims to contribute to the final report on the
programme, which the Commission will present to the European Parliament, the
Council, the Economic and Social Committee, and the ACVT later in 2000.
2.2.2 Specific objectives of the evaluation
The specific objectives of the evaluation were to evaluate the Leonardo da Vinci 1
programme and provide advice and counselling to the European Commission on five
key issues:
•
what are the outputs and impacts (results and outcomes) of Leonardo da Vinci 1
compared to the expectations described in the Decision?;
•
the effectiveness of the programme: does it achieve its own general, specific and
operational objectives?;
•
the efficiency of Leonardo da Vinci 1; how economically did it produce results
and impacts; how efficient was the management and delivery structure?;
•
how to improve the effectiveness and efficiency of Leonardo da Vinci 1 in future
programmes?;
•
how to develop complementarity and synergy with other vocational education
and training instruments and policies both at the EU and Member State levels.
The programme's objectives were very broadly drawn and the result was a programme
that reflected the wide differences in training policy, systems, priorities and resource
levels that characterise the European training scene. This is even more marked in the
case of the pre-accession countries. Accordingly, drawing conclusions in the form of a
European synthesis presents difficulties, and some of the report's detailed discussions
are more relevant to one, or a small group of, participant countries.
Final Evaluation of Leonardo da Vinci
14
Deloitte & Touche
2.3 The consultant
This study was undertaken by the European Services department of Deloitte &
Touche, based in Brussels working in collaboration with an international team of
consultants all of whom have extensive experience in the field of education and
training but were totally independent of any involvement in the Leonardo da Vinci
programme.
2.3.1 The Study team
The study team consisted of a core team based in Brussels responsible for the project
management, survey work and analysis and an expert team based in the participating
states and responsible for the field work.
The core team consisted of Richard Doherty (Project Director), Isabelle Collins
(Project Manager), Christopher Allinson (Expert), Sarah Batterbury (Expert), Wouter
Van Den Berghe (Expert) and Nathalie Belnet (Project Assistant).
The expert team consisted of:
♦
♦
♦
♦
♦
♦
♦
♦
♦
Hanne Shapiro
Paddy Carpenter
Lionel Chaty
Peter Debreceni
Pedro das Neves
Prof Theo Papatheodossiou
Anna Gammaldi
Gabriele Marchl
Anne Marie De Jonghe
Nordic Countries
UK and Ireland
France
Eastern Europe
Iberia
Greece and Cyprus
Italy
Austria, Germany, Liechtenstein
Benelux
2.4 Methodological Approach
The consultants originally planned for this work to be carried out on the basis of a
September start. Unfortunately, due to a series of difficulties outside the control of the
Unit responsible in the Commission, this was not possible, and the contract was
finally signed on 22 December 1999.
In addition to this, a set of thematic evaluations which had been planned to contribute
to this overall evaluation were never undertaken. Additional inputs in the form of the
reports from the Member States are in many cases still not available. However, the
date for presenting the final report was not changeable.
Our methodological approach was designed to respond to the key evaluation questions
set out in the call for tenders and refined in further meetings with the Commission
(see Section 1.2 above). A multi-method approach was employed in order to enable us
to answer questions concerned both with process (implementation, management) and
with measurable outputs. These two categories of question necessitate two different
methodological instruments – qualitative and quantitative. In addition to
methodological triangulation (the use of multiple methods to ensure rigour and
reliability of the data), the evaluation also employed researcher and data-source
triangulation. These provide additional sources of cross-checks which ensure that the
findings are neither biased in favour of the view-point of a single researcher, nor
based on information derived from a solitary source. Triangulation is a standard and
essential aspect of any kind of qualitative research and is employed to ensure rigour,
Final Evaluation of Leonardo da Vinci
15
Deloitte & Touche
reliability and credibility of research through multiple cross-checks of data sources
and in the analysis of these sources. In this evaluation the three forms of triangulation
- methodological, researcher and data source - were combined to ensure quality,
reliability and credibility in the data.
The methodological approach was structured around four main pillars:
•
Methodological design
•
Documentary review
•
Interviews with key informants at European and Member State levels, and case
studies with selected projects
•
Analysis and synthesis
These methodological pillars overlapped during the course of the implementation of
the evaluation as follows:
Nov
Dec
Jan
Feb
Mar
Apr
May
Methodological
design
Documentary
review
Interviews &
project visits
Analysis
synthesis
&
The evaluation also adopted an explicit stakeholder focus, ensuring that the voices of
all key stakeholders were consulted during the collection of data. A number of
specific instruments were designed to hear the voices of all the key stakeholders in the
programme.
2.4.1 Methodological Design
Methodological design was centred on an iterative process with the client. It reflects
the complex nature of the evaluation questions which span both output and process
issues. As indicated above, the research was also founded on the principle of
triangulation - in-built cross-checks serving as a quality control to ensure scientific
rigour for the research. In addition to broad methodological criteria, the process of
methodological design also encompassed the design of specific research instruments.
For this evaluation a number of different instruments were drawn up including:
• An interview proforma and reporting template - to direct and structure the field
visits by the national experts (comprising both semi-structured and structured data
collection instruments)
•
Semi-structured and unstructured interviews designed as a guide for one-off,
bilateral encounters with key informants.
•
Questionnaires for pilot projects.
Final Evaluation of Leonardo da Vinci
16
Deloitte & Touche
•
Questionnaires for mobility projects.
•
For placement and exchange participants.
•
A format for case study records.
The questionnaires were designed to capture process-oriented information which
illuminated the nature and experiences of the programme implementation,
management, and performance, and to gather more descriptive data concerned with
outputs, products and measurable impact. Structured questions were included to
facilitate comparability of responses between projects in a format amenable to
statistical manipulation.
2.4.2 Documentary review / Desk study
In order to examine the role of Leonardo within the European policy context we have
undertaken a review of documentation from the Commission, mainly DG Education
and Culture, and also DG Employment and Social Affairs. This was supplemented
with discussions with CEDEFOP and examination of a number of documents from
them. Finally we have looked at the data provided by CLEO (the Commission’s
Leonardo support office established in March 1999) in order to provide certain
statistical analyses of the programme.
The documentary analysis was a comprehensive exercise involving the sifting of a
considerable quantity of information. We have not summarised all the sources
consulted in this evaluation, as they would be too long to list. Moreover, a
considerable number of sources consisted of unpublished documentation provided to
us by Commission officials and other parties. This information is not amenable for
listing in the form of an annex. We place particular importance on respecting the
confidentiality of programme records and documents. Accordingly, to give an idea of
the breadth of our documentary analysis, we have clustered the documentary sources
into 9 categories which are listed below:
•
National reports1
•
Unpublished documentation supplied by Commission officials and other
interviewees
•
Interim evaluation, 1997
•
Published reports from other related programmes
•
CEDEFOP policy report 2000
•
The White paper on Growth, Competitiveness and Employment
•
Other data supplied from CLEO.
•
Thematic papers from CEDEFOP
•
Web based materials
These sources were an invaluable input into the evaluation as they offered different
perspectives and inputs into our research. While the documentary sources are not
listed above in any particular order, the national reports and informal documentary
1
Not all national reports were available at the time this evaluation was finalised.
Final Evaluation of Leonardo da Vinci
17
Deloitte & Touche
sources provided by the Commission were of the greatest import. Programme records
and documents were used to help identify important issues for the evaluation, to
identify the important processes shaping the implementation of Leonardo, and to
provide a useful source of information - filling gaps that could not be observed or
learned through interviews. Both official and unofficial documents generated by
Leonardo helped to prompt appropriate questions for the interviews and questionnaire
design. They were also an invaluable source of contextual information and cross
checks.
2.4.3 Consultation with Stakeholders
The evaluation also aimed to ensure that the voices of all key stakeholders were
consulted during the collection of data. We placed considerable importance on
inclusiveness in the evaluation process. We therefore employed both interviews and
questionnaires in order to hear the voices of the different key stakeholders in the
programme. The key stakeholders identified are set out in the matrix below.
Stakeholder
Data-collection instrument
The European Commission
Semi-structured interviews
CLEO
Semi-structured interviews
CEDEFOP
Semi-structured interviews
Member State officials
Semi-structured interviews
NCUs
Interview proforma (semi-structured and
structured data collection instruments)
Pilot projects
interview proforma (semi-structured and
structured data collection instruments)
questionnaire
Mobility projects (that had been selected questionnaire
centrally)2
Exchange participants
questionnaire
In addition to bilateral exchanges with key actors (European Commission, CLEO,
Member State officials, NCUs, CEDEFOP), one of the most important groups to
question on their views on the Leonardo da Vinci 1 programme were the projects that
had actually participated. In view of the fact that these projects are dispersed
throughout the European Union, questionnaires were considered to be the best option
to spread the consultation process as widely as possible.
2
Absence of data on mobility projects selected in the Member States limited the sample to those which
had been selected centrally.
Final Evaluation of Leonardo da Vinci
18
Deloitte & Touche
2.4.3.1
Questionnaires
A number of surveys have already been undertaken among projects that have
participated in Leonardo da Vinci 1 . These include surveys undertaken as part of the
Member State reports. Several projects have reported to us that this is the third or
fourth time they have been approached, but there is no evidence to suggest that this
unduly reduced the response rate.
After careful consideration, it was decided that the questionnaire would focus mainly
on the following points:
ƒ
The needs of the projects taking part in the programme.
ƒ
The dead weight of Leonardo da Vinci 1, i.e. would the projects have taken place in some form
even if they had not received financial support from the EU? Would they have attempted to use
other sources of EU funding?
ƒ
The added value of Leonardo da Vinci 1, i.e. are projects aware of other programmes? What is the
difference between these programmes and Leonardo da Vinci 1?
ƒ
The management, content and awareness of the programme, i.e. what is their impression of the
awareness of the programme? What are the most valued components of Leonardo da Vinci 1?
What opinions are there on the quality of the programme?
ƒ
A small number of complementary questions concerning the concrete and intangible results of
Leonardo da Vinci 1 on the projects questioned.
ƒ
In addition, projects were encouraged to share any further comments not specifically covered by
the questions contained in the questionnaire.
The questionnaire was drawn up after studying previous questionnaires drawn up by
ECOTEC as part of the intermediate evaluation. Separate questionnaires were
provided for Pilot projects and Placement and Exchange projects. In addition, the
Placement projects were sent a questionnaire which they were asked to pass on to
participants.
Sampling - The Questionnaires
Stratified sampling was employed in selecting projects and exchange and placement
participants for consultation via the questionnaire instruments.
The sample of pilot projects was broken down (by project leader) as shown in the
matrix below, spread proportionately across pilot projects, placement and exchange
projects and study/analysis projects, and by year of approval.
Final Evaluation of Leonardo da Vinci
19
Deloitte & Touche
Total
B
Belgium
46
BU
Bulgaria
0
DK
Denmark
26
D
Germany
140
GR
Greece
48
E
Spain
157
EE
Estonia
2
F
France
123
IRL
Ireland
35
I
Italy
142
L
Luxembourg
12
NL
Netherlands
34
A
Austria
27
P
Portugal
33
FIN
Finland
29
S
Sweden
29
UK
United Kingdom
140
IS
Iceland
9
FL
Liechtenstein
1
N
Norway
21
EUR
Europe
16
CY
Cyprus
3
CZ
Czech Republic
8
HU
Hungary
6
RO
Romania
10
LT
Lithuania
0
LV
Latvia
1
PL
Poland
6
SK
Slovakia
5
SL
Slovenia
0
TOTAL
1108
Final Evaluation of Leonardo da Vinci
20
Deloitte & Touche
As agreed with the Expert Group on the External Evaluation, no projects were
included from the 1999 call, as they were insufficiently advanced to be able to
comment on outcomes.
As we were constrained to use data provided by CLEO, we only had contact
information for the projects administered centrally. No survey was possible of the
projects administered and selected in the participating states.
As we discussed earlier, a multi-method approach has been the cornerstone of the
evaluation. For this reason, sampling was also carried out in a purposeful fashion,
seeking out key themes and illustrative events which shed light upon the specific
process aspects of the programme. Participants’ responses were particularly useful in
this. Maximum variation was sought across Member State, programme strand, and
individual themes which were generated from the data and firmly grounded in the rich
data we obtained from the participants’ responses.
2.4.3.2
Contacts with National Co-ordinators
The importance of the role of the national co-ordinators of Leonardo da Vinci 1
derives from their role in the marketing and promotion of the programme, their
contribution to the selection of projects and their permanent support function. In
addition their new, more central role in the decentralised Leonardo 2 programme
means that their views are central to the implementation and management of the
programme. They are also key stakeholders in the programme and so their views are
central to the programme’s evaluation.
Therefore, it was important that the views of the national co-ordinators be collected in
a systematic way. It was necessary to design a comprehensive interview template
which would allow us to tap their knowledge and experience to the fullest extent. A
draft template was drawn up by the team for use in interviews, telephone interviews
and group meetings and supplemented by a detailed reporting framework. In order to
ensure a degree of flexibility and context sensitivity the experts used their knowledge
of the participating states to ensure suitability of questions within the broad parameter
specified by Deloitte & Touche.
2.4.3.2
Project Case Studies
A number of projects were selected as mini-case studies in order to shed light on
important issues in more detail.
In particular the case studies focused on themes of central importance to the
evaluation:
•
Complementarity
•
Implementation
•
Outputs
•
Impact
•
Key themes
In selecting the case studies we have attempted to choose a group of projects which
will cover the widest possible number of issues, while ensuring that case studies were
undertaken in all the Member States. This also follows the qualitative sampling
principle of maximum variation purposeful sampling. The distribution of case studies
Final Evaluation of Leonardo da Vinci
21
Deloitte & Touche
across the participating countries is as shown below (again split proportionately by
year and across project types):
Total
%
Case
Studies
Belgium
121
4.30%
2
Denmark
69
2.45%
1
Germany
368
13.06%
7
Greece
126
4.47%
2
Spain
414
14.70%
7
France
323
11.47%
6
Ireland
91
3.23%
2
Italy
374
13.28%
7
Luxembourg
31
1.10%
1
Netherlands
89
3.16%
2
Austria
70
2.48%
1
Portugal
86
3.05%
2
Finland
76
2.70%
1
Sweden
77
2.73%
1
United Kingdom
369
13.10%
7
Iceland
23
0.82%
0
Liechtenstein
3
0.11%
0
Norway
55
1.95%
1
Europe
43
1.53%
2
Cyprus
9
0.32%
0
CEEC
100
3.55%
4
TOTAL
2817
100.00%
56
2.4.4 Analysis and Synthesis
This is often the most difficult aspect of any evaluation. We have taken particular
care therefore to ensure reliability of the data, maintaining multiple triangulation
systems throughout the research process and holding detailed records of all
questionnaire and field notes relevant to the evaluation.
The data has been analysed using a number of standard research techniques,
Final Evaluation of Leonardo da Vinci
22
Deloitte & Touche
•
Statistical aggregation of data enabling trends and measures to be presented in
tabular and graphical form;
•
Coding of qualitative data enabling the distillation of key themes, generation of
conceptual insights and grounded theory;
•
Cross-case comparison as a basis for comparative evaluation research. The
element of comparison is also an essential part of the generation of grounded
insights into the programme. We have analysed the data, thematically selecting
important standard evaluation criteria related to key issues for organisational and
programme performance and impact.
During the analysis process we have sought to identify three major characteristics
within the programme.
•
Areas of maximum variation and difference;
•
Context-determined characteristics; and
•
Areas characterised by similarity and synergy.
In this report we therefore bring together evaluation research which is firmly
grounded in and generated by the data, and illustrative, more descriptive measures,
which have been deduced from the statistical data available to us.
Final Evaluation of Leonardo da Vinci
23
Deloitte & Touche
3 Vocational Education & Training Policy Context of
Leonardo da Vinci I
3.1 Introduction
The EU vocational education and training (VET) policy framework into which the
Leonardo da Vinci programme was launched in 1995 was not a static one-dimensional
structure, but rather a dynamic plural system embracing a variety of actions with the
potential for synergy in contributing to the further development of socio-economic
policy at the European level.
This framework – which has developed over a period of some 30 years and has
continued to develop intensively during the lifespan of Leonardo da Vinci 1 –
includes policies, decisions, opinions and initiatives in the domains of employment
and economic development; education, personal development and lifelong learning;
and social affairs, including equal opportunities. Over the course of the life of the
programme policy framework has been increasingly oriented towards the EU
employment strategy developed through the Luxembourg process.
The coming together of the economic and socio-educational aspects of EU strategy
was clearly articulated in March 2000 at the Lisbon Launch Conference of the
Commission working paper 'Implementing Lifelong Learning for Active Citizenship
in a Europe of Knowledge'. This working paper defines lifelong learning as "all
purposeful learning activity, whether formal or informal, undertaken on an on-going
basis with the aim of improving knowledge, skills and competence". It also reflects
upon the conclusion of the European Council, at the end of the 1996 European Year
of Lifelong Learning, that:
"the principles underpinning a strategy of lifelong learning are derived from the
fundamental aim of education and training to serve individual, cultural, social and
economic well-being and to enhance the quality of life for all citizens."
The working paper goes on to state:
"the Socrates and Leonardo da Vinci programmes, now entering their second phase,
are firmly placed within the context of lifelong learning. The main objectives of
Leonardo da Vinci are closely linked to the first three pillars of the Luxembourg
process with equal opportunities as a horizontal dimension. Within these objectives
the Commission will pay particular attention to proposals aiming to develop new
approaches to lifelong learning, the transnational dimension of the implementation of
the employment guidelines, with a special emphasis on equal opportunities and the
development of generic skills aimed at the new technologies and the environment."
The lifespan of the Leonardo da Vinci 1 programme coincided with an evolution and
maturation of policies and principles in respect of lifelong learning and its relationship
with the maintenance and growth of employment in the European Union. While this
will doubtless benefit successor programmes, the policy context existing in 1995 at
the launch of Leonardo 1 was rather less focused, having developed in the way
illustrated by the process and the chronology below.
Final Evaluation of Leonardo da Vinci
24
Deloitte & Touche
3.2 The origins of the policy framework
The basis of EU vocational training policy is to be found in Article 128 of the Treaty
of Rome (1957), which provides for the Council of Ministers to establish
"general principles for implementing a common vocational training policy
capable of contributing to the harmonious development both of the national
economies and of the common market."
The subsequent Council Decision of 2 April 1963 elaborated ten principles around
which the then Member States agreed to begin to harmonise their actions on
vocational training. The second of these principles included a combination of the
individual social and educational aspects of learning with the economic effects of
skills training
"to broaden vocational training on the basis of a general education, to an
extent sufficient to encourage the harmonious development of the personality
and to meet requirements arising from technical progress, new methods of
production and social and economic developments;
to enable every person to acquire the technical knowledge and skill necessary
to pursue a given occupation and to reach the highest possible level of
training, whilst encouraging, particularly as regards young persons,
intellectual and physical advancement, civic education and physical
development."
Flowing from the Council Decision was a raft of provisions for co-operation between
Member States, together with the setting up of the Advisory Committee on Vocational
Training. Between 1963 and 1983, the development of the framework of vocational
policy in the EC took the form of several non-binding resolutions. The result of the
first of these was the establishment of the European Centre for the Development of
Vocational Training (CEDEFOP) in 1975. The purpose of the centre was to provide
information and expertise on vocational training and to stimulate research, particularly
on the transferability of training standards to support the free movement of workers
within the European Community.
The growth of unemployment in the 1980s led to the raising of the profile of
vocational training as a strategic response to the phenomenon of long-term joblessness
in the EC, and the need to support the restructuring of the labour market, by equipping
workers with the capacity to adapt to technological change. In addition to taking
account of the effects of new technology, labour market policy was faced with new
challenges posed by the substantial growth of unemployment amongst young people
and by the increasing economic activity amongst women.
A Council Resolution of July 1983 emphasised the strategic role of vocational
training as:
"An instrument of active employment policy to promote economic and social
development and adjustment to the new structure of the labour market;
A means of ensuring that young people are properly prepared for working life and
their responsibilities as adults;
An instrument for promoting equal opportunities for all workers as regards access to
the labour market."
Final Evaluation of Leonardo da Vinci
25
Deloitte & Touche
For young people, in particular for those without qualifications, Member States
committed themselves to providing basic training and work experience for at least a
six month period, for those who wished to participate.
While the July 1983 resolution was articulated at the European level, decisions
regarding the method and pace of implementation remained the preserve of Member
States themselves, a principle (following legal testing as described later in this
chapter) that continues to underpin EU vocational education and training policy.
3.3
Chronology of vocational training and relevant education action
programmes 1986-19933
Following some 20 years of non-binding resolutions regarding vocational training, the
European Communities' institutions agreed in the mid 1980s to devise a legal base,
founded upon Article 128 of the Treaty of Rome, for further developments in the
field. As a result of this flowed a variety of action programmes:
COMETT I (1986-1989) Programme on co-operation between universities and
enterprises regarding training in the field of new technology, with a budget of
MECU45. It financed 1,300 projects and established 125 university-enterprise training
partnerships. COMETT II (1990-1994) had a budget of MECU200 - including an
EFTA contribution.
ERASMUS (1987-1995) Programme setting up the European Community Action
scheme for the mobility of university students, with a budget of MECU500.
PETRA (1987-91) Programme for the vocational training of young people and their
preparation for adult and working life, with a budget of MECU40. Participants
numbered some 75,000, along with more than 10,000 teachers and trainers. PETRA II
(1992-1994) had a budget of MECU104.2
LINGUA (1990-1994) Programme to promote foreign language competence in the
European Community, with a budget of MECU153. More than 7,000 language
teachers received in-service training and 33,000 teachers participated in exchanges.
EUROTECNET (1990-1994) Programme to promote innovation in the field of
vocational training resulting from technical change in the European Community, with
a budget of MECU9.2. It provided funding for the networking of innovative projects
in vocational training and for transnational research on specific themes linked to
vocational training.
FORCE (1991-1994) Programme concerned with the quality and quantity of
continuing vocational training, with a budget of MECU88. There were 720 projects,
involving the transfer of innovation and expertise in continuing vocational training.
HELIOS I (1988-1991) and HELIOS II (1993-1996) promoted the social integration
and independent lifestyle of people with disabilities. HELIOS II had a budget of
MECU37, enabling 1,150 organisations working for disabled people to take part in
discussions and exchanges with similar organisations in other Member States, with
particular emphasis on vocational training and preparation for working life.
3
See CEDEFOP: An Age of Learning: vocational training policy at European Level. Thessaloniki 2000
Final Evaluation of Leonardo da Vinci
26
Deloitte & Touche
3.4 Clarification of the legal aspects of vocational training policy
The use of Resolutions based upon Article 128 of the Treaty of Rome, backed up by
Article 235 (which provided legal cover where specific treaty articles did not exist),
disturbed some Member States concerned about the extent of Community competence
in the field of vocational training. A legal action arising from 1989 Commission
proposals to adapt ERASMUS culminated in a judgement by the European Court of
Justice that Article 128 did indeed provide for legal measures that could be binding on
Member States in the field of vocational training. The judgement also stated that a
common vocational training policy could, broadly, include higher and university
education.
The anxieties of Member States concerning the extension of legally-binding
vocational training measures at a European level were confirmed by the Social
Charter of Fundamental Workers' Rights, adopted under the French presidency in
1989. This document, while non-binding in itself, proposed a right of access by all
workers to continuing training systems and to training leave.
The debate on the scope of European-level binding vocational training policies was
resolved in the run up to the Maastricht summit which revised the social chapter of the
Treaty of Rome, including Article 128. In Article 127 of the Treaty of the European
Union, signed at Maastricht in 1992, the previous common vocational training policy
was replaced by a Community vocational training policy, explicitly designed to
support and supplement the vocational training activities of the Member States, but
preventing harmonisation at a European level. Article 126 of the Maastricht Treaty
removed education from the legal ambit of vocational training policy, again excluding
harmonisation of Member State policies and actions. Where actions under Article 128
had required a simple majority, actions under the new Article 127 were to be decided
in Council by qualified majority and after consultation with the European Parliament.
Thus, the formal policy framework into which the Leonardo da Vinci programme was
later to be launched is as follows:
Treaty on the European Union 1992 - Article 127
1. “The Community shall implement a vocational training policy which shall support
and supplement the action of the Member States, while fully respecting the
responsibility of the Member States for the content and organisation of vocational
training.
2. Community action shall aim to:
a) facilitate adaptation to industrial changes, in particular, through vocational
training and retraining;
b) improve initial and continuing training in order to facilitate vocational
integration and reintegration into the labour market;
c) facilitate access to vocational training and encourage mobility of instructors and
trainees and particularly young people;
d) stimulate co-operation on
establishments and firms;
training
between
educational
and
training
e) develop exchanges of information and experience on issues common to the
training systems of the Member States.
Final Evaluation of Leonardo da Vinci
27
Deloitte & Touche
3. The Community and the Member States shall foster co-operation with third
countries and the competent international organisations in the sphere of
vocational training.
4. The Council, acting in accordance with the procedure referred to in Article 189c
and after consulting the Economic and Social Committee, shall adopt measures to
contribute to the achievement of the objectives referred to in this Article,
excluding any harmonisation of the laws and regulations of the Member States.”
3.5 Vocational training - EU fora
Alongside the development of the legal policy framework concerning vocational
training in the European Community/Union, a number of relevant bodies have been
created over the past thirty years, each with a different purpose and composition, but
each concerned discretely with vocational training policy and practice at a European
level.
The Advisory Committee for Vocational Training (1963), established following
the 1963 Council decision. Its task is to provide the Commission with an opinion
before formal vocational training proposals can be made.
The Management Board of CEDEFOP (1975) has the role of managing CEDEFOP
as the EU information and research body in vocational training, with a tangential
relationship to policy developments.
The Governing Board and the Advisory Forum of the European Training
Foundation (1994), which has the task of managing the ETF's work in support of
training developments in Central and Eastern Europe.
The European Forum for the Transparency of Vocational Qualifications (1997)
was established as a result of an initiative by CEDEFOP and the Commission with the
task of progressing the transparency of qualifications across the EU.
The Social Dialogue - working party on education and training, established to
provide a forum for the social partners to meet and to produce common opinions on
vocational training policy and issues.
The Directors General for Vocational Training of the Member States meet once
under each presidency, with an agenda normally associated with the priorities of the
presidency. This is an informal forum for senior civil servants to exchange views and
information without formal outputs but with influence on the progress of vocational
training policy development.
The Committee of the Leonardo da Vinci Programme (1995) has the task of
guiding the Commission in the implementation of the Leonardo programme.
Each body occupies a separate position in the European-level framework of
vocational training, with little, if any, structured dialogue between them.
3.6
Other Policy Influences
Two 1990s White Papers helped raise the profile of vocational training into a central
position in the EU policy arena:
Final Evaluation of Leonardo da Vinci
28
Deloitte & Touche
•
Growth, Competitiveness, Employment - the challenges and way forward into the
21st century (1993)
•
Teaching and learning - towards the learning society (1995)
1993
Growth, Competitiveness, Employment - the challenges and way forward
into the 21st century
This White Paper stressed the importance to the EU of intangible investment in
education and research, proposing a strategy to achieve more employment-intensive
economic growth. The paper argued a role for vocational training in addressing issues
of unemployment and competitiveness. In the light of inadequacies identified in
existing vocational education and training systems the white paper proposed the
adoption of policies and practices of continuing training and lifelong learning. The
European Year of Lifelong Learning, one result of the paper, took place in 1996 with
the overall aim of promoting:
− "personal development;
− the integration of individuals into working life and society;
− the participation of individuals in the democratic decision-making process;
− the ability of individuals to adjust to economic, technological and social change;"
With a number of sub themes advocating, for example, the provision of vocational
training for young people, continuing education and training, co-operation between
commerce and education for SMEs, measures aimed widening and increasing
opportunities for lifelong learning, and the development of a European dimension in
initial and continuing education .
The Council conclusions on the strategy for lifelong learning proposed the following
framework of areas for further development:
− "challenges for the school system;
− economic and social considerations;
− continuing education and training;
− pathways and links between general and vocational education;
− access, certification and accreditation;
and
− teachers and adult educators."
1995 Teaching and learning - towards the learning society
The Competitiveness White Paper themes of lifelong learning were further
emphasised in this second White Paper, which noted three major developments
affecting the European social economy: the globalisation of trade, the emergence of
the information society and, the progress of science and technology.
Final Evaluation of Leonardo da Vinci
29
Deloitte & Touche
The White Paper proposed that education and vocational training policies should be
aimed at broadening the knowledge base and improving individual employability. The
paper identified five objectives towards a learning society:
− "to encourage the acquisition of new knowledge through the introduction of a
European system of accreditation of technical and vocational skills
− to bring about closer co-operation between schools and business through
apprenticeships and training
− to combat social exclusion by offering second-chance education to those affected
− to ensure proficiency in three Community languages;
and
− to equalise treatment of capital and training investment."
3.7 Other EU Policy Influences
In addition to the economic and employment-related elements mentioned above, the
European Union is based on several overarching principles, a number of which are of
particular relevance to the Leonardo da Vinci programme. The principle of
subsidiarity has particular relevance for education and training, as we have shown in
describing the debate about legal base. The Amsterdam Treaty (1997) reinforced the
principles of European citizenship, a commitment to tackle social exclusion, and the
development of an integrated approach to the development of a common employment
strategy. The Leonardo programme was conceived and implemented within the aegis
of these main principles of European Integration, as well as building on the
“traditional” Community approach of transnational demonstration projects, and
upgrading significantly the opportunities for personal mobility.
The matrix below summarises these key themes, showing the way in which Leonardo
has translated these key EU principles into specific programme elements.
Principles of the EU relevant to
Leonardo
Feature of Leonardo
Interaction between European citizens
Subsidiarity
Trans-nationality
National Co-ordination Units and national
project selection
Development of the concept of
European citizenship
Development of a common strategy
on employment & co-ordination of
national strategies
Development of a Community
vocational training policy
Mobility, placement & exchange
Attempt to develop a European Vocational
Training policy
Linkage of programme to labour market
policy.
Final Evaluation of Leonardo da Vinci
30
Deloitte & Touche
Commitment to tackle social
exclusion
Targeting of specific marginalised groups
Promote equality between men and
women
A stated objective
Relevant Principles derived from the Structural Funds approach
Additionality
Co-financing
Partnership
Transnational projects & NCU role
Important Policy Domains
SMEs
All present in Leonardo
Social Partners
Eastern & Central Europe
Innovation & technology
We will return to these different themes at various points throughout the report.
However, it is worth noting at this stage that Leonardo accorded greater value to some
rather than others during its implementation.
3.8 Launch of Leonardo da Vinci
By 1995 the EU vocational training and education policy and activity framework had
evolved to:
•
provide an agreed revised legal basis for Community actions in this field, not
seeking harmonisation but respecting the autonomy of Member State provisions
while providing for value added at the European level;
•
encompass the twin themes of development of the individual as citizen and the
development of the individual as economic actor through the principles of lifelong
learning;
•
support several permanent vocational training policy/practice discussion forums;
•
acquire a practice base through a range of funded programmes which would be
substantially brought together under the new Leonardo programme
On 6 December 1994 the Council approved the Leonardo da Vinci programme as the
first Community Vocational Training Action Programme since the revised TEU of
Maastricht resolved the legal framework.
3.8.1 Leonardo da Vinci I – Common framework of objectives
Leonardo da Vinci began with a common framework of objectives provided for in
Article 1 (3), the aim of which was to contribute to the achievement of the objectives
of Article 127 of the Treaty by pursuing the 19 different objectives in the Decision. In
accordance with the recommendations of the expert group convened by the
Final Evaluation of Leonardo da Vinci
31
Deloitte & Touche
Commission to guide this evaluation, we can cluster the objectives around four
themes:
• European Union level objectives
•
National level objectives
•
Target group objectives
•
Horizontal objectives
The matrix below shows how the 19 Leonardo Objectives may be clustered around
these themes:
Clusters
Objectives
European Union level
objectives:
•
developing the European dimension in vocational
training and vocational guidance
•
supporting vocational training policies in such a way that
all workers in the Community have access to continuing
vocational training throughout their working life without
any discrimination
•
promoting the gradual development of an open European
vocational training and vocational qualifications area,
particularly through the exchange of information and
experience on obstacles to application of the free
provision of services by training bodies. This is also a
horizontal objective
•
-promoting co-operation on skill requirements and
training needs, and
- encouraging the acquisition and transparency of
qualifications and an understanding of the key skills
relevant to technological development and the
functioning of the internal market, including the free
movement of goods, services, persons and capital, the
competitiveness of undertakings and the requirements of
the labour market. This is also a horizontal objective
National level
objectives
Target group
objectives
•
improving the quality and innovation capacity of
Member States' vocational training systems and
arrangements
•
giving all young people in the Community who so wish
the possibility of one or, if possible, two or more years of
initial vocational training after their full-time compulsory
education, leading to a vocational qualification
recognised by the competent authorities in the Member
State in which it is obtained. This is also a target group
objective
•
promoting lifelong training so as to encourage ongoing
adaptation of skills to meet the needs of workers and
undertakings, contribute to reducing unemployment and
Final Evaluation of Leonardo da Vinci
32
Deloitte & Touche
facilitate personal development
Horizontal objectives
•
encouraging specific vocational training measures for
adults without adequate vocational qualifications, in
particular adults without adequate education
•
promoting vocational training for young people and
preparing young people for adult and working life,
taking account of the requirements of society and
technological change
•
encouraging specific vocational training measures for
disadvantaged young people without adequate training
and in particular young people who leave the education
system without adequate training
•
promoting equality of access to initial and continuing
training for persons disadvantaged by socio-economic,
geographical or ethnic factors or by physical or mental
disabilities; special attention must be given to persons
affected by several risk factors likely to cause their social
and economic exclusion;
•
promoting equality of opportunity as regards access for
migrant workers and their children and the handicapped
to vocational training and their effective participation
therein;
•
promoting the development of vocational guidance
facilities with a view to providing every individual with
the opportunity to have lifelong high-quality vocational
guidance;
•
enhancing the status and attractiveness of vocational
education and training and promoting parity of esteem
for academic diplomas and vocational qualifications;
•
promoting equality of opportunity as regards access for
men and women to vocational training and their effective
participation therein, in particular so as to open up new
areas of work to them and encourage them to return to
work after a career break
•
promoting vocational training in the light of the results
of technological research and development programmes,
particularly by means of co-operation between
universities and undertakings in the sphere of training in
technologies, their application and their transfer
•
fostering the development of methods of self-training at
the workplace and of open and distance learning and
training, in particular to facilitate access to continuing
vocational training;
•
encouraging the development and integration of key
Final Evaluation of Leonardo da Vinci
33
Deloitte & Touche
skills in vocational training measures, with the aim of
promoting the acquisition of flexible qualifications and
personal skills necessary to worker mobility and the
needs of undertakings
•
supporting activities aimed at developing linguistic skills
as part of vocational training measures
In addition to the large number of objectives there were also 4 strands:
•
support for the improvement of vocational training systems and arrangements in
the Member States
•
support for the improvement of vocational training measures including university
– enterprise co-operation concerning undertakings and workers
•
support for the development of language skills, knowledge and the dissemination
of innovation in the field of vocational training
•
support measures,
In addition to this list of desirable Leonardo activities, a more focused set of five
priorities was established for the programme in 1996 and 1997:
1. The acquisition of new skills;
2. Forging closer links between educational/training establishments and enterprises;
3. Combating exclusion;
4. Promoting investment in human resources;
5. Promoting access to skills through the information society in the context of
lifelong learning.
This evaluation provides an opportunity for stakeholders to assess the degree of
coherence in the above context, within which Leonardo da Vinci I was implemented.
The large number of objectives, priorities, and strands created a confused picture for
potential applicants. There was no clear hierarchy of objectives, some of the
“priorities” were - in reality – objectives, and many of the “objectives” were, in real
terms, more akin to activities.
In Leonardo 2, the number of objectives has been reduced to three. However the
picture remains confused. Priorities, strands and objectives provide a problematic
intersection which translates with some difficulty into practical project proposals and
activities. Those closely involved with Leonardo in the Commission also report some
difficulty in unravelling the hierarchy of objectives. Goal-based evaluation is also
extremely difficult in the absence of a clear hierarchy of objectives.
Final Evaluation of Leonardo da Vinci
34
Deloitte & Touche
4 Complementarity with other EU instruments
In this chapter we outline the main EU programmes which are related in their
objectives and purpose to Leonardo. This enables us to situate Leonardo within the
broader programming environment and thereby demonstrate its particular niche and
complementarity with other programmes. We show that Leonardo can be seen as
belonging to a hierarchy of programmes in training and social investment, and that its
distinctiveness vis-à-vis other European programmes can be shown by its offering of
large-scale opportunities for trainee mobility and of funding for active testing of
innovative ideas. By understanding the way in which Leonardo is embedded within
the broader programme context we are able to demonstrate the potential for crossprogramme knowledge sharing and dissemination of good practice.
The Leonardo programme came into being alongside a number of related EU
programmes. In this chapter we give particular focus to the European Social Fund, the
ADAPT and EMPLOYMENT initiatives, the Research and Technological
Development Framework Programmes, and the related parallel programme
SOCTRATES. In the paragraphs which follow, however, we first situate Leonardo in
its context by giving a brief overview of the different programmes which were
integrated into the Leonardo programme.
4.1 Relationship with previous programmes
Initiatives in the area of vocational training across Europe are numerous. A brief
search in the CORDIS4 database of EU programmes found 46 programme records
related to the key words “vocational training” from 1989 to now. While a small
number of these concern broader policy issues such as EU-US co-operation, the vast
majority are EU programmes with a direct involvement in vocational training.
Equally, the database is not exhaustive and will not pick up some of the smaller EU
actions such as that on the Third System and Employment or the ESF Article 6 pilot
projects. This gives, however, a broad indication of the range of activities being
carried out on this topic within the Commission and an indication of the task required
to ensure complementarity
The Leonardo da Vinci programme was aimed at a rationalisation of Community
action in the area of vocational training. It took forward initiatives developed under
COMETT, PETRA, FORCE, LINGUA and EUROTECNET and added new
dimensions.
In very broad terms these programmes were respectively concerned with the
development of University/Education Training partnerships, the transition from
school to work, re-qualification of the workforce, the development of language skills
and promoting innovative actions in the field of vocational and continuing training.
Each of these prior programmes can be clearly identified within the strands of
Leonardo da Vinci.
The Leonardo da Vinci Technical Assistance Office was also responsible for the
transition from the programmes to Leonardo da Vinci. One of the initiatives they
undertook in this area was the development of a database of products from the past
4
Community Research and Development Information Service (www.cordis.lu)
Final Evaluation of Leonardo da Vinci
35
Deloitte & Touche
programmes. However, this is no longer accessible due to the difficulties that have
beset the programme.
4.2
Relationship with other Community interventions
4.2.1 the Structural Funds
The majority of the EU's investment in the Member States is done through the
medium of the Structural Funds (The European Regional Development Fund, The
European Social Fund, The European Agricultural Guarantee and Guidance Fund and
the Financial Instrument for Fisheries Guidance). While all these funds contribute in
some form to training, a significant proportion of this entails investment in physical
infrastructure. The investment in the human infrastructure is generally carried out
through the European Social Fund.
4.2.1.1
The European Social Fund (ESF)
The ESF is the EU’s main financial instrument for investing in people. Its mission is
to help prevent and fight unemployment, to make Europe's workforce and companies
better equipped to face new challenges, and to prevent people losing touch with the
labour market.
In the programming period 1994 to 1999, the ESF priorities were expressed in
Objectives 3 and 4 of the Structural Funds.
For Objective 3 the fund supported actions which:
•
facilitated the occupational integration of long term unemployed persons,
•
facilitated the occupational integration of young people in search of employment,
including up to two years initial vocational training
•
promoted integration of people excluded from the labour market
•
promoted equal opportunities for men and women on the labour market
The types of actions supported included
•
vocational training, pre-training, guidance and counselling
•
temporary employment aids
•
the development of training, employment and support structures.
Objective 4 was aimed at facilitating the adaptation of workers to industrial change
through:
•
anticipation of labour market trends and vocational qualification requirements
•
vocational training and retraining, guidance and counselling
•
assistance for the improvement and development of appropriate training systems
with a focus on the specific needs of SMEs.
Thus the ESF and Leonardo da Vinci had similar goals, especially in terms of
promoting and improving training, education and counselling and the development of
vocational qualifications. The major difference lies in the immediate target
beneficiary of the projects, the approach and the types of project assisted.
The ESF channels EU money into helping the Member States. Thus the focus and
outcomes are designed to have an impact at a national, regional, or local level. The
Final Evaluation of Leonardo da Vinci
36
Deloitte & Touche
ESF has wider objectives than Leonardo da Vinci covering support for infrastructure,
improving vocational education and training systems, and the promotion of a culture
of training. The ESF’s greatest impact is arguably on labour market structures and
institutions. Leonardo da Vinci, conversely, was designed with a more specific EU
focus in the sense that it represented an opportunity for the formulation of EU-wide,
transnational Vocational Education and Training approaches. This transnational
dimension was unique to Leonardo da Vinci.
In terms of scale of resources, the ESF between 1994 and 1999 had some 57191
million ECU at its disposal.5 (i.e. nearly 100 times more than Leonardo da Vinci 1)
4.2.1.2
ADAPT AND EMPLOYMENT
Community Initiatives are intended to act as a kind of laboratory within the ESF for
the development of new approaches for helping people access, and retain work. The
projects within the two Initiatives Adapt and Employment were all based on five key
principles:
• working with partners in different Member States;
• developing innovative approaches;
• local involvement;
• creating a multiplier effect with results;
• complementarity with other EU programmes.
There were two Community Initiatives in the field of Human Resource Development
(HRD) which had a relationship to the Leonardo da Vinci area – ADAPT and
EMPLOYMENT. Of these, ADAPT had the closer relationship.
The ADAPT Initiative (with a budget of ECU 1.4 billion), drawing on previous
experience with EUROTECNET and EUROFORM, complemented Objective 4 of the
Structural Funds, assisting innovative approaches to the challenges of preparing
workers for industrial change. It required that projects be undertaken in a context of
transnational collaboration between linked projects.
A new priority within the ADAPT Initiative -ADAPT-Bis- was added to address the
labour market implications of the Information Society.
A reserve budget was drawn on to finance the new strands; this provided an additional
ECU 460 million for EMPLOYMENT and ADAPT. ADAPT projects were designed
to fall into four categories of measures:
♦ Supply of training, counselling and guidance
♦ Anticipation, promotion of networking and new employment opportunities
♦ Adaptation of support structures and systems
♦ Information, dissemination and awareness actions
5
Source:'The impact of structural policies on economic and social cohesion in the Union 1989-99'.
European Commission 1997.
Final Evaluation of Leonardo da Vinci
37
Deloitte & Touche
The number of projects funded under the ADAPT Programme was roughly similar to
that funded by Leonardo da Vinci. However as the budget was over 20 times larger, it
can be seen that the average size of the projects was significantly greater.
ADAPT projects were promoted and organised by partners who reflected the whole
range of companies, organisations and representative and intermediary bodies which
make up the contemporary labour market and those who provide it with training,
information, and other resources and services. Member States developed their own
Operational Programmes and indicated their own priority promoters, in some cases
excluding others from direct eligibility, but generally the following types of
organisations were involved in active partnerships:
•
companies (especially SMEs), groups of companies, sectoral organisations,
chambers of commerce, crafts and agriculture;
•
trade unions and other worker organisations;
•
public and private training organisations and universities;
•
local and regional authorities;
•
local development agencies;
•
non-governmental and voluntary organisations;
•
organisations concerned with equal opportunities.
Thus there was a clear overlap in terms of target project promoters between the two
programmes.
As in Leonardo da Vinci, innovation was a key issue for ADAPT projects, defined in
terms of making changes and introducing new practices in the face of technological
and other changes and transforming the ability of workers themselves to respond to
them.
Although both ADAPT and Leonardo da Vinci had a core element of
transnationality, this was implemented in different ways. Within ADAPT the
partnership took the form of linked projects involving at least 2 Member States and
requiring approval of each project by the Member State concerned. Under Leonardo
da Vinci the projects were required to be true partnerships – one of the factors which
the participants reported as most valuable in Leonardo da Vinci. ADAPT did not
exhibit or require this to the same extent during the selection process. The third
important element for ADAPT was the notion of mainstreaming. This was less
central for Leonardo da Vinci.
In terms of dissemination, it was acknowledged by the ADAPT programme that this
was one of the key elements of the programme and the intention to relate the products
of innovation to mainstream practice had to be built into the projects from the start.
The EMPLOYMENT Initiative approved in 1994, with an ESF budget of ECU 1.5
billion, contained three interdependent strands, two of which were developed from the
original NOW and HORIZON Initiatives:
•
NOW - to promote equal opportunities for women;
Final Evaluation of Leonardo da Vinci
38
Deloitte & Touche
•
HORIZON - promoting pathways to employment for persons with disabilities
and others threatened with exclusion;
•
YOUTHSTART - integrating young people under 20 into the labour market,
especially those without adequate training or qualifications.
The structure was modified with the introduction of a new strand, INTEGRA,
specialising in the employment barriers faced by immigrants, refugees and other
people at risk of exclusion. HORIZON continued to serve the needs of people with
disabilities. All of these focused more on direct employment projects rather than on
training.
4.2.2 The Research and Technological Development Framework
Programmes
EU Research and development expenditure is carried out in the context of multiannual Framework Programmes, comprising a series of research programmes. Under
the Fourth Framework Programme (1993-1997), and the current Fifth Framework
Programme (1998-2002), there were two main areas of research funded which had a
close relationship to the work of Leonardo da Vinci. These approximate to actions
relating to the human element and those relating to the technology/innovation
element.
Under the Framework Programmes research is funded on a shared cost basis for
projects proposed by transnational partnerships similar to the type of partnership
required for Leonardo da Vinci projects. Also, as in Leonardo da Vinci there is the
possibility to involve the associated states, leading to a wider public than just the EU
Member States. As an element of their applications, projects are required to present a
Technology Implementation Plan which details how the outputs from the project will
be exploited, an element which was not present in such a detailed form under
Leonardo da Vinci.
In terms of the human elements, the most relevant programme under the Fourth
Framework Programme was Training and Mobility of Researchers (TMR), especially
the Marie Curie Fellowships. This programme involved a relatively small number of
participants, highly targeted in the field of scientific research. The placements were on
average longer than Leonardo da Vinci placements and there is little real overlap.
The Targeted Socio-economic Research Programme (TSER) had a key action on
improving the socio-economic knowledge base. This produced some 200 projects on a
range of themes such as innovation in education and training via technology, Human
resource development and competence development. As an example the theme of
innovation in education and training via technology produced 7 projects, all of which
could in essence have been funded under Leonardo da Vinci.
Under the Fifth Framework Programme this area has been integrated into the new
Programme improving human research potential and the socio-economic research
base. This works in 3 areas:
♦ Improving the human research potential (which contains the mobility actions)
♦ Improving the socio-economic research base, and
♦ Support for scientific and technological development policies in Europe.
The Programme has a budget of €1280 million over 5 years.
Final Evaluation of Leonardo da Vinci
39
Deloitte & Touche
In terms of technology and innovation, under the Fourth Framework Programme, the
Telematics Application Programme had a strand on Telematics for Education and
Training looking specifically at aspects of technology in training and education. This
programme also had a supporting measure focusing specifically on the dissemination
of the results of the projects.
Under the Fifth Framework Programme this has been incorporated into the
Information Society Technologies (IST) Programme, which also absorbed the
ESPRIT and ACTS programmes. PROSOMA is a multimedia showcase for the
results of these programmes, also aimed at promotion of the take-up of the outputs.
Also under the Fifth Framework Programme, the JRC workplan budget line
"underpinning European Competitiveness" is financing an action at the IPTS in
Seville regarding the development of a "European Observatory of Knowledge"
designed as the core activity to combine all essential sources of information on
education, training and skills available in the European Union.
Its objectives are:
•
Inventory and networking of existing observatories on "Education", "Training"
and "Youth". Improvement of data compatibility, retrieval and information
exchange procedures. In an advanced stage of development the Observatory will
be able to provide access to statistics, to indicators on the quality of education, to
contribute to the analysis of the current state of the art of the European educational
system and to consensus generation on best practices.
•
Monitoring organisational changes in public administration and industry, the
evolution of the utilisation of new types of services, the impacts on the
management of supply chains and the related emerging demands for new skills in
society.
•
Development of benchmarking tools to analyse the impacts of current skills
shortages on competitiveness and innovation at company level (focus on SMEs)
and in entire supply chains at regional, national and EU level.
•
Analysis of the dynamic links between the emerging demands for new skills and
the need for a corresponding timely and flexible adaptation of the education and
training programmes.
•
Analysis of the impact of emerging educational, training methods and techniques
on human knowledge acquisition ability.
•
Monitoring, analysis and testing of emerging data mining techniques and
knowledge management systems.
•
User friendly dissemination and interactive exchange of information on skills,
education and training for citizens, SMEs, R&D providers & users,
National/Regional policy decision-makers. Particular focus will be devoted to
disabled citizens.
As such it offers an opportunity for the dissemination of the results of Leonardo da
Vinci in the future. CEDEFOP is a partner in the project and thus there is the start of a
co-ordination mechanism in place, should the Commission decide to use it.
Final Evaluation of Leonardo da Vinci
40
Deloitte & Touche
4.2.3 SOCRATES
SOCRATES and Leonardo da Vinci were in many senses parallel programmes. The
former focused on education and the latter of vocational training. In reality these
distinctions are less clearly maintained. Approximately 18% of Leonardo da Vinci
projects were held by universities for example. This issue is discussed in more detail
in Chapter 6.
The issues on complementarity, synergy and possible overlap between SOCRATES
and Leonardo da Vinci can be considered from different perspectives, such as:
♦ the content of the programmes;
♦ the actors and beneficiaries involved;
♦ the structure and management of the programme.
Each of these will be briefly discussed below.
As regards the content, it is quite obvious that SOCRATES and Leonardo da Vinci
could never have been designed in a fully complementary way. Indeed, the distinction
between the concepts which are at the heart of both programmes - “education” for
Socrates and “vocational training” or “training” for Leonardo da Vinci - has become
increasingly blurred over the last decades – and moreover covers different realities in
different countries. It is clear that this situation provides opportunities for both
synergy and overlap.
A clear example of the possible overlap is the funding for activities in the field of
“Adult education” (SOCRATES) and “Continuing (vocational) training” (Leonardo
da Vinci) where the distinction has never been clarified. We did not find any evidence
of a clear and accepted policy – either at European or at Member State level – that
would ensure complementarity. In fact, under SOCRATES 2 and Leonardo da Vinci 2
this parallelism has been continued and even reinforced. Other areas where
complementarity was unclear concerned open and distance education, language
training, industry-university collaboration and certain mobility activities.
On the other hand, it appears to us that little overlap has actually occurred in the field
– but also little synergy. This is first because of the uneven access policy, whereby
the formal “education” community could take part in both, but where SOCRATES
was restricted to them alone. Further, because of the activities supported by the two
programmes only addressed a very tiny part of the possible target groups.
Another characteristic of complementarity concerns the organisations involved in both
programmes. There, the boundary line is, in principle simple. In SOCRATES, with a
few exceptions, only education institutions can participate; in Leonardo da Vinci in
principle any organisation can participate - with a strong preference for organisations
in need of vocational training. In reality, it is the education institutions that often take
the lead in Leonardo da Vinci (in some countries up to 90% of projects are coordinated by them). This holds in particular for higher education institutions, who
often pursue not so much a “programme logic” (“what can we get out of a
programme”), but rather a “portfolio logic” (“what do we want to do, and where can
we get funding for it”). There is little evidence that this practice leads to a lot of
“double funding” or “overlap” between projects supported, but institutions often tend
to optimise the deployment of their human resources by applying to different funding
mechanisms.
Final Evaluation of Leonardo da Vinci
41
Deloitte & Touche
It should also be noted that, although SOCRATES is almost exclusively involved with
education institutions, in certain actions, other types of organisations can be involved
(e.g. Thematic Networks; or Mobility of Students which may include a practical
placement in a company).
Complementarity and synergy could also be considered from a structural level, i.e. the
way the programmes have been organised, managed and implemented. From this
perspective, synergy has been poor. Both at European (Committees, Commission and
TAO) and national levels (with a few exceptions), management and organisation of
the programmes was not closely co-ordinated, with virtually no contact at policy and
implementation level. This can be seen in simple operational things like different
funding mechanisms and rules for similar activities, or different timing and formats
for selection procedures. But also little structural contact occurred at policy level, for
instance in the decisions on what types of actions ought to be centralised /
decentralised, and in the design of both SOCRATES 2 and Leonardo da Vinci 2. The
people who are knowledgeable about SOCRATES and Leonardo da Vinci - in the
Commission, in the TAOs, in the Agencies, … - are very few in number. Some
beneficiaries, with a long tradition in obtaining European funding, appear to be better
informed about the similarities, differences and scope for overlap and synergy than
the policy makers themselves.
The distinction between SOCRATES and Leonardo (and also YOUTH) is therefore
predominantly one of organisations and policy-makers, rather than one of
beneficiaries and target groups. From that perspective, there is scope for both
complementarity and overlap. In practice, some complementarity was achieved in the
field, rather than at policy level – and overlap was not high. But synergy between both
programmes has been weak, at all levels: European policy level, Member State level,
and organisational level.
4.2.4 Other related activities, such as SME programmes and Social Fund
pilot projects
In addition to the mainstream programmes there are a number of small actions
financed by the Commission which have some relationship to vocational education
and training.
For example, under Article 6 of the current ESF Regulation, 1% of the ESF budget is
set aside to provide funding for pilot projects that develop innovative ideas for
promoting employment and social integration. These projects are selected centrally
and there is a programme of network animation associated for the exchange of
experience and accompaniment of the projects. Equally in the case of the budget line
for pilot projects in the field of the Third System and Employment, there was a
required element of transnationality but also a system of network animation and
valorisation of the outputs. Both these elements are relatively small in terms of the
number of projects supported.
Equally some pilot actions undertaken in the framework of the multi-annual
programme for SMEs have an element of training for SMEs. Again these are
relatively small and often do not have any mechanism for the dissemination of the
results.
DG Enterprise’s expertise and level of resources available for its own developmental
work in the training field are too limited to allow it to make a real impact. The
Final Evaluation of Leonardo da Vinci
42
Deloitte & Touche
importance of the SME dimension of training policy is not to be underestimated (as
the Leonardo da Vinci Decision noted), but the actions in the SME Programme are too
limited to contribute in a real way to European thinking. There was a genuine lack of
knowledge among stakeholders in the Member States about the training part of the
SME Programme. Nor is there evidence that the managers of Leonardo da Vinci
really included or benefited from that work in their own planning, despite their DG’s
role as the lead agency for training policy development in the Commission. In the
other direction, the influence of the lessons of Leonardo da Vinci in the SME area was
not easily visible in the planning for DG Enterprise’s Communication on training for
SMEs, despite former DG XXII’s view that greater discussion between the two
Directorates General would have been valuable. It seems this dialogue now longer
exists at all.
4.3 Duplication or synergy/value added
According to DG Employment and Social Affairs:
“Policy development and implementation under the Structural Funds are also linked
to other EU programmes, especially Leonardo da Vinci Da Vinci, which supports and
complements actions in the Member States to improve the quality of training policy
and practice through transnational projects. Close synergy has been developed with
ESF operations, especially under the Employment and Adapt Initiatives. Socrates
which is intended to raise teaching standards, adapt training systems to new
technologies and promote life-long learning is also close to ESF Objectives.
The prevention of social exclusion, a main theme of the ESF, also forms part of the
Leonardo da Vinci and Socrates objectives and has been supported by other EU
programmes in this field such as the actions initiated under HELIOS to promote the
integration of people with disabilities."6
In practice we found very little evidence of these admirable intentions being
implemented. At a European level, and in terms of the strategic relationships between
the programmes, internal Commission structures, pressures on officials, and the sheer
administrative weight of managing their programmes, reduced the capacity for
Commission officials to engage in inter-departmental collaboration. Despite some
formal procedures (such as the sharing of national Structural Fund draft programmes
with DG EAC before finalisation), the evidence of effective synergy is limited. This
is not surprising: for many years this incidence of low synergy between DG EAC and
the major spending departments has been pointed out as a significant weakening of
the Community-level potential in the training field.
In addition the implementation processes were very different – centralised for
Leonardo da Vinci and decentralised for EMPLOYMENT and ADAPT - which also
had a significant impact on the potential for synergy both at European and national
levels.
Indeed at Member State level an example of this lack of synergy can be illustrated by
the experience of Sweden which had located the co-ordination of all the relevant
programmes in one single office, hoping to benefit from synergies. In fact none were
forthcoming and so for the new generation of programmes there will be two offices –
6
Commission website http://europa.eu.int
Final Evaluation of Leonardo da Vinci
43
Deloitte & Touche
one for Leonardo da Vinci and SOCRATES and one for the rest, as this structure is
felt to be more in keeping with the different requirements.
The only example of co-operation we identified was the joint call for multi-media
projects in 1997 involving also the Community Initiatives and the Framework
Programme.
In terms of duplication, this is very difficult to assess, as the mechanisms for checking
are quite rudimentary. Nevertheless a system of consultation on the selection of
projects does exist and the opportunity is there to signal any examples. Whether the
managers of the large programmes have the capacity to identify overlap is not clear –
certainly there is no mechanistic way of doing this, and it relies on the experience of
those involved.
Having said this, there is no evidence of any significant overlap. Project promoters
seem to be quite sophisticated in selecting appropriate funding mechanisms. No
application processes are simple and a considerable investment is required from the
promoters. It is not, therefore, in their interest to apply indiscriminately.
In the opening paragraphs to this chapter we observed that Leonardo da Vinci was
located within a hierarchy of EU programme initiatives in the vocational training field
and related sectors. The concept of a continuum between the various funding regimes
is a particularly useful one when we consider the issue of complementarity. This
would categorise programmes as follows:
Nature/aim of intervention
Specific programme
Experimental
ESF Article 10, Third System, SME
Programme
Research/Innovation
Leonardo da Vinci
Development/Test Marketing
Community Initiatives (now EQUAL)
Mainstream implementation
ESF
In each case the lessons learned from a programme should be built on in the next
programme at a different scale. Continuity between programmes in fact represents a
policy opportunity that, if accurately harnessed, can yield valuable conduits for the
dissemination and spillover of best practice and both strategic and operational lessons.
Encouraging the utilisation of conduits of information is a recognised pathway for
optimising systemic knowledge management strategies within the overall vocational
and educational training polity.
Recognition of the hierarchy of programmes and the opportunity for continuity and
information channels also has an implication for the selection and evaluation criteria
for each type of programme. Under Leonardo da Vinci, projects are supposed to be
innovative and to test out new ideas. This implies an element of risk, and of freedom
to take risks - the space to innovate. A conclusion that an idea does not work need not
imply that the project has been a failure – it may have successfully tested an idea with
the conclusion that it is not feasible to continue with the approach. In addition ideas
which do not work well in one context may always be of use in different contexts. The
application of this kind of learning entails well-structured and effective systems of
knowledge management to be put in place.
Final Evaluation of Leonardo da Vinci
44
Deloitte & Touche
4.4 Take-up of lessons by direct funding programmes
There is little evidence of take up of lessons by direct funding programmes at
European level. However, unless and until programmes such as Leonardo da Vinci are
in a position themselves to draw lessons from their experiences it is difficult to see
how mainstream programmes can be expected to do so.
Thus, if the purpose of Leonardo da Vinci is to act as a laboratory of innovation, the
mechanisms need to be developed to describe in a useful and synthesised manner, the
outputs of the programme and the lessons learned. This requirement needs to be built
into the programme from the beginning as it affects the management, monitoring and
reporting requirements and process design.
In contrast, there was some take-up of lessons reported at participant state level
particularly, but not exclusively, in those states where the vocational training system
was less well developed.
Final Evaluation of Leonardo da Vinci
45
Deloitte & Touche
5 Implementation
The Leonardo da Vinci programme is extremely complex both in its formulation and
in its implementation. Many of these complexities are embedded in the 1994
Decision, while others are related to the management systems of the European
Commission. A large number of stakeholders are involved, but an equally large
number of entities are involved in the decision making and administration procedures.
5.1 Structures and Procedures
In order to administer the programme effectively a number of organisations were set
up to assist the Commission. This reflected an externalisation of a large proportion of
the administrative work both to a Technical Assistance Office (TAO) in Brussels, and
to National Co-ordination Units within the participating countries.
This mirrors to a large extent the systems used under the previous generation of
programmes, and was a relatively common model within the Commission, not only
for Programmes within the field of vocational education and training but for
programmes in other Directorates General.
The TAO was set up as the result of an open call for tenders in accordance with both
the European Procurement Directives and the Commission's own internal procedures.
The designation of the National Co-ordination Units was the responsibility of the
participating states. Despite the fact that the EU budget provided much of their
funding, the Commission had no decisive power in their selection. On the other hand,
the contractual relationship was between the Commission and the individual NCUs.
During the course of the Leonardo da Vinci programme there were a number of
events which culminated in the closure of the TAO and the transfer of their functions
to a unit within the Commission – overseeing the Cellule Leonardo (CLEO) which
took over the administrative responsibilities of the TAO. In the course of this
evaluation we have obviously had to take into account the factors leading to these
developments and the impact of these on the operation of the Programme. However,
we have attempted to distinguish those issues which related to Programme design,
those which related to institutional factors at an EU or national level, and those which
related specifically to the events surrounding the problems of the TAO.
It should be made clear that the underlying principles and processes of managing the
projects did not change fundamentally with the change from the TAO to CLEO.
In addition to these various structures, there were two different application processes
according to the different strands of the programme.
The remainder of this section looks at the various structures and their function, the
processes associated with the implementation of the Programme, the risks or critical
areas, the costs and benefits of the system, and finally examines the comments of the
interim evaluation and the extent to which these have been taken into account.
We also comment briefly on the introduction of Leonardo 2 in the context of
highlighting those issues which have already been addressed in the implementation of
the new programme phase.
Final Evaluation of Leonardo da Vinci
46
Deloitte & Touche
5.1.1 Technical Assistance Office
The Technical Assistance office was set up following an open call for tenders at the
end of 1994. It took the form of a five year service contract renewable annually,
financed from A-budget lines (ie not included in the operational budget of the
Programme). The contract started in June 1995, and was terminated unilaterally by the
Commission in January 1999.
Under the previous programmes 7 separate Technical Assistance Offices had existed
and their functions were transferred to the new TAO with the launch of the new
Programme, and to the TAO for Socrates and Youth which was set up following a
similar procedure.
5.1.1.1
The Contractor
The Contract for the TAO was awarded to Agenor, an organisation set up specifically
to manage this type of operation at the start of the FORCE programme, with
shareholders from a range of organisations with an interest in the field of vocational
training. The major shareholder was CESI, a major French vocational training
organisation through its associates SECOF and RESOURCES SA. The remainder
were DIHT (Deutscher Industrie und Handelstag), AIFPI of Greece, FPE of Spain,
Sistemi Formativi Confindustria of Italy and LO Danish Confederation of Trade
Unions – all organisations with an involvement in vocational training.
Agenor had previously held the contract for the Technical Assistance Office for the
FORCE Programme, and thus could reasonably be expected to be competent in the
administration of European Programmes of this type.
5.1.1.2
Role of TAO (terms of reference)
The work programme of the TAO was set out each year in an annex to the contract.
Initially the tasks fell into seven broad areas:
♦ Setting up the infrastructure for the Programme, including all the documentation
for the Programme
♦ Assistance in the calls for proposals
♦ Assistance in selection of the projects according to the relevant selection process
(see below)
♦ Administrative and financial management of the contracts
♦ Actions other than calls for proposals
♦ Information actions
♦ Transition (building on the experience of previous programmes), planning,
monitoring and assistance with evaluation actions not performed by the TAO.
These tasks were gradually refined over time reflecting the different stages of the
Programme and the maturity of experience of both the Commission and the TAO.
At the end of the TAO’s existence the work programme had developed to cover:
♦ Administration of the projects (including project selection procedures)
♦ Information
♦ Dissemination
♦ Meetings
Final Evaluation of Leonardo da Vinci
47
Deloitte & Touche
♦ Information Technology
♦ Closure of contracts from the previous programmes
The work programme had a clear focus on: contracting (including payments and
monitoring), preparation of information for use in the final evaluation of the previous
programmes, provision of an effective tool for monitoring and management of the
projects (by the IT system of the TAO), and preparation for the involvement of the
pre-accession countries.
5.1.1.3
Structure of TAO
The TAO had approximately 75 staff in a number of departments dealing with the
various aspects of its work. This staffing complement was reinforced by external
experts for processes such as the selection of projects.
Figure 5.1 The Structure of the TAO
IT
4 people
NCU Network animation
Evaluation and
Monitoring
6 people
9 people
Management
4 people
Contracts and Finance
Dissemination/Virtual
Products Centre
10 people
6 people
Logistics and planning
4 people
Personnel/Administration
Analysis and thematic
reports
5 people
5 people
Information and
Publications
7 people
Two criticisms that have been made of this structure are that insufficient resources
were dedicated to the area of contracts and project management given the workload
involved, and that the structure was not flexible enough to deal with the evolution of
the programme and changing needs as it proceeded along its life-cycle.
Final Evaluation of Leonardo da Vinci
48
Deloitte & Touche
To a great extent this is a reflection of the Commission’s contracting procedures in
that in response to a need to adhere closely to initial requirements, defined before full
knowledge of the situation was possible, external contractors are not able to be as
flexible as necessary in the execution of contracts. This situation is not in any way
unique to the Leonardo da Vinci TAO, but does illustrate a contradiction in
attempting to gain additional flexibility by externalising work, but then constraining it
such that the potential is lost.
The initial structure also reflects a certain lack of clarity in terms of the boundaries of
responsibility between the Commission and the TAO, particularly as concerns the
policy aspects of the Programme.
5.1.1.4
Budget
The initial budget of the TAO was 9.1 million ECU for the first year. This gradually
reduced over time, 8 MECU in the second year, 7.3 MECU in the third and 5.9
MECU for the final year. This was the budget for the operation of the TAO, all of
which was claimed from the Commission on the basis of actual expenditure with the
exception of a small management fee of about 100,000 ECU per annum.7
In addition to this funding from Budget line A, the TAO was responsible for the
management of some 98 million ECU per annum from budget line B, being the
subventions to the actual projects and which passed through a TAO bank account.
This implies that the Agenor consortium incurred costs and managed funds totalling
over 100M ECU per year, employing 75 people, with only a small management fee of
100 000 ECU per annum to cover its corporate risks. This is highly unusual, to say
the least.
5.1.1.5
Special circumstances, and the establishment of CLEO
In the normal management procedure of the TAO, at the end of each year the final
accounts were checked by the Commission before the balance of their claim for that
year was paid. At the end of 1997 it became apparent that some of the experts
working for the TAO, primarily in the evaluation of project applications, worked for
companies which were involved in projects under the Leonardo da Vinci programme.
In order to be sure that there were no potential conflicts of interest, this information
was sent by DG XXII to DG XX and UCLAF to enable a check to be carried out on
this. An audit was thus undertaken in the TAO between February and April 1998
where all the project evaluators were checked against all the partners in the projects.
As a result of this process 4 projects were questioned but subsequently accepted.
Following this DG XX audited a selection of 30 projects and no problems were found.
During this period no progress could be made on the signature of the 1997 Projects or
on the renewal of any 1995 projects, which led to a backlog of administrative work.
The contracts were eventually cleared for signature in mid April (726 projects) and
mid May (the remaining 4 projects).
Subsequent to this audit, allegations of irregularities arose which were investigated in
an audit by DG XX and UCLAF which was undertaken in the first half of 1998. This
was completed in July, and finalised in October following the responses from DG
XXII. This was submitted to the Parliament at the end of December 1998.
7
Source: Commission Audit Report
Final Evaluation of Leonardo da Vinci
49
Deloitte & Touche
This report recommended a number of changes to the management of the TAO which
were not implemented by the contractor. Therefore, following discussions, the
Commission decided not to renew the contract between the EU and Agenor SA. At
the same time the Belgian Courts decided to pursue the investigation of the fraud
allegations, which led to the sealing of the files held by the TAO.
This meant that there was a another period of some 6 months between January and
June 1999 when no activity could take place on Leonardo projects managed centrally
while the files were reconstituted and the new Cellule Leonardo da Vinci (CLEO) was
established.
This Cellule was set up within the Commission and took on a proportion of the
original TAO staff (although not the management staff). The structure of the CLEO
was designed to focus on the processing of payments and the final rounds of project
selection. The staffing of CLEO is organised as follows:
Figure 5.2: The Structure of CLEO (12/99)
Content
Logistics
17 people
13 people
Finance
Contracts
Co-ordination
4 people (inc Head of
16 people
8 people
Unit B.2)
Reconciliation
Other
5 people
3 people
In the context of this evaluation, the only question of relevance in the terms of
reference of the audit report related to the promotion of the calls for proposals and the
selection procedures to ensure the best possible selection of projects.
5.1.2 National Co-ordination Units
In each Member State there was a National Co-ordination Unit. These bodies were
nominated by the Member State and contracted to the Commission. The NCUs were
co-funded from the Leonardo da Vinci budget, the rate of subvention initially varied
Final Evaluation of Leonardo da Vinci
50
Deloitte & Touche
between 30 and 60% in different Member States although there has been a
convergence around 50% recently. The cost of the NCUs represents some 8% of the
Leonardo da Vinci budget (Budget line B only), or about 11.2 MECU per annum.
NCUs were also set up in the pre-accession countries.
5.1.2.1
Structures
There was no model structure for a National Co-ordination Unit. In fact, there was a
single contact point only in 7 Member States, the others having several either with a
geographical responsibility (eg Belgium) or a functional responsibility by strand (eg
the UK). The incidence of a single NCU was much more marked in the pre-accession
countries.
There was equally no single model for the form of the NCUs. These varied from
private or other independent organisations (eg NGOs) contracted to provide the
services, to Ministries or special inter-ministerial units set up specifically for the
purpose.
In some cases the Leonardo da Vinci NCU was based with other EU Programme
management units (eg ADAPT or Socrates) but in many instances this was not the
case.
Despite the fact that in some instances it was difficult for newcomers to the
Programme to locate the NCU in their countries, the level of satisfaction with the
support from NCUs was very high in the projects we sampled. These were all projects
administered by the TAO in terms of payments etc, so we cannot comment on the
views of the Mobility projects managed directly by the NCUs.
Exhibit 5.1 Project Satisfaction with the NCUs
•
“National co-ordinating unit was very helpful. Not much contact with central
TAO. Vademecum more distant, of interest at the very initial stage” - Finnish
project
•
“National Co-ordinating Unit very helpful and very quick turnaround at all times.
Central technical assistance unit – decision making process far too lengthy” - UK
project
•
“The support of our National Co-ordination Unit was fundamental for the
successful development of the project – this was greatly appreciated by all of the
partners” - French Project
5.1.2.2
Functions
In each of the participating countries, the National Co-ordination Unit had three main
functions:
♦ An information role, responsible for the promotion of the Programme within the
country concerned
♦ A selection and management role for the projects selected under Procedure 1
♦ An advisory role for the projects selected under Procedure 2.
In addition they had the role of smoothing the transition from the previous generation
of programmes.
Final Evaluation of Leonardo da Vinci
51
Deloitte & Touche
In terms of the information role, the NCUs were crucial in order to ensure that
information on the programme reached the widest possible audience, particularly all
the key players in the country. Although the European calls were published in the
Official Journal, this was not necessarily a source of information readily accessed by
some of the key actors targeted by the Programme.
In addition to this, the NCUs provided active help and background material on the
application process, which was quite complex and in most cases also provided
guidance and support during project implementation. Once the programme was more
established many NCUs also undertook a range of activities aimed at promoting the
results of the programme. These included seminars, publications and databases on
websites.
Some of the websites were quite rich in information (eg UK, Italy) while others were
extremely basic.
The role of the NCUs in the selection process is described in section 5.2 below.
5.1.3 The Leonardo Committee
Article 6 of the Decision provides for the establishment of a Committee composed of
two representatives from each Member State, chaired by the Commission. The Social
Partners also participate in the Committee as Observers. This Committee is
responsible for deciding on the general guidelines for the financial support to be
supplied (amounts, duration and recipients), on matters relating to the internal
breakdown of the programme, the procedures for selection, follow-up, evaluation,
dissemination and transfer of results.
It is also responsible for setting the priorities for the programme.
Thus the principal decisions taken by the Committee relate to the setting of the annual
priorities – a not-insignificant task, given the complexity of the programme – and the
approval of the proposed projects from each annual call for proposals.
With the extension of the Programme to the Pre-Accession countries, major questions
need to be asked about the capacity of the Committee to guide the Programme due
simply to the sheer size of the committee and its resultant unwieldiness. Even the
extensive use of sub-committees does not enable the degree of flexibility which would
be desirable.
We encountered much criticism of the Committee for its narrow focus on the issue of
distribution of Leonardo Da Vinci resources “between” the Member States, rather
than playing a useful role in guiding the programme in terms of thematic focus,
linkages with other activities and so on.
5.2 Project selection and management
Projects were selected and managed on the basis of two different procedures
depending on the type of project. These were organised under the responsibility of
either the Member States (Procedure 1) or the Commission (Procedure 2) as follows:
Final Evaluation of Leonardo da Vinci
52
Deloitte & Touche
Procedure 1
Procedure 2
Member
State
Strand
(mobility)
Commission
Remainder of
Strand I
Commission
I Support for the improvement of
vocational training systems and
arrangement in the Member States
Strand III.1
Development of Language Skills
Strand III.3.a
Multiplier-effect projects
Strand II
Support for the improvement of
vocational
training
measures,
including university/industry cooperation, concerning undertaking
and workers.
Strand III.2
CEDEFOP
Strand III.3.b
Transnational Study Visits
The projects concerned with Transnational Study Visits were administered by
CEDEFOP.
In general calls subject to both selection procedures were published at the same time
for maximum coverage and transparency.
5.2.1 Procedure 1
This procedure is overwhelmingly concerned with the placement and exchange
projects previously funded under the PETRA programme. Under this procedure a call
for proposals was organised by the Member State. The Member State would produce
an operational plan for the programme which would form the basis of their budgetary
envelope and the selection of the projects.
Project promoters sent an application to the National Co-ordination Unit of behalf of
the participating state, which then managed an assessment of the projects and
produced a list of pre-selected projects with a supporting report on the selection
process. This list was then discussed with the Commission (who also received copies
of the applications and assessed them), and a final draft list was produced which was
submitted to the Leonardo da Vinci Committee for approval.
This procedure covered approximately 1700 projects, with states selecting between 1
and 40 projects per annum, with a maximum total number of projects being managed
by a participating state of 214 over the 5 years.
Projects funded under this procedure represented approximately one third of the
Programme budget.
5.2.2 Procedure 2
The remainder of the projects (some 3000 projects) were selected using procedure 2.
Under this procedure the Commission published a call for proposals in the Official
Journal. The call was also promoted through the NCUs and any other available
Final Evaluation of Leonardo da Vinci
53
Deloitte & Touche
networks. Calls were generally published in December with a final decision taken by
the Commission in November of the following year.
Under this procedure, project promoters would send copies of their applications to the
TAO and to the NCU. On reception each application was evaluated following a
detailed evaluation framework by Commission officers and by 2 independent external
experts nominated by the participating countries. This evaluation framework was
gradually refined over the life of the Programme. The applications were also
evaluated by the participating countries.
The Council Decision sets out a division of budget between different thematic
categories:
Criterion
Minimum %
Initial vocational training
40%
Continuing vocational training
32%
Lifelong learning
13%
Pilot Projects
35%
Mobility Projects
30%
In addition to this a balance was sought between the participating states in terms of
number of projects and budgets. Although no formal quotas were established per
participating state, an informal system existed and formed the basis of the Leonardo
Committee’s decisions.
Following the evaluation of the projects they were awarded a mark between 1 and 4,
with 1 being “excellent” and 4 being “not acceptable”.
On this basis a list of proposed projects to be accepted was drawn up and
communicated to the participating states. There then followed a process of bilateral
negotiation where the results of the central and national evaluations were discussed
and a final draft list was drawn up for presentation to the Leonardo da Vinci
Committee for approval. The rate of acceptance of the projects varied from 16.49% in
the first year to 35% for 1999, although overall it was in the region of 30%.
An inter-service consultation also took place within the Commission to try and ensure
that there was coherence between the projects selected and projects selected under
other EU Programmes managed elsewhere in the Commission. Chapter 4 has
discussed the degree of success of this approach.
Once the opinion of the Leonardo da Vinci Committee was received, the written
procedure was launched for the formal approval of the Commission.
Final Evaluation of Leonardo da Vinci
54
Deloitte & Touche
The Timetable for Leonardo da Vinci project selection, 1999
ID
1
Task Name
Call for Proposals
2
Proce dure I
3
Receipt, data input, analysis and preselection
4
exchange of proposals EC/NCUs
5
Lists of eligible proposals to Commission
6
Proc I eval report and preselection list to EC
7
Receipt, input and evaluation by Commission
9
Prep of NCU report to Commission
10
Exchange of lists of proposals EC/NCUs
11
NCU Evaluation report to Commission
12
Analysis of proposals by DG XXII
21/05
21/05
Evaluation of Proc II proposals by experts
14
Evaluation of Proc I andII eval reports by DG XXII
15
Se lection of propos als
16
Prep of files for bilateral meetings
17
Meeting of Leonardo Committee
18
Bilateral Meetings
19
Establishment of draft selection list
20
Interdepartmental consultation on the list
21
Draft list to Leonardo Committee
22
Meeting of Leonardo Committee
23
Written Procedure in the Commission on list
24
Decision by the Commission on the selection list
25
12/05
Proce dure 2
8
13
December January
February March
April
May
June
July
August
September October
November December January
February Ma
30/11
07/12
14/12
21/12
28/12
04/01
11/01
18/01
25/01
01/02
08/02
15/02
22/02
01/03
08/03
15/03
22/03
29/03
05/04
12/04
19/04
26/04
03/05
10/05
17/05
24/05
31/05
07/06
14/06
21/06
28/06
05/07
12/07
19/07
26/07
02/08
09/08
16/08
23/08
30/08
06/09
13/09
20/09
27/09
04/10
11/10
18/10
25/10
01/11
08/11
15/11
22/11
29/11
06/12
13/12
20/12
27/12
03/01
10/01
17/01
24/01
31/01
07/02
14/02
21/02
28/
19/10
24/11
Contract negotiation
Final Evaluation of Leonardo da Vinci
55
Deloitte & Touche
Thus the selection process took somewhat over 8 months, with contracting procedures
taking a further 3-4 months with up to 7 months being not unknown. By way of
comparison, the procedure for selection of projects under the Fifth Framework
Programme takes 3-4 months (including 4-6 weeks for Committee opinion) with a
further 6 weeks to 4 months for the contracting process. As with Leonardo da Vinci,
some aspects of the timing on the contracting process lie in the hands of the projects
themselves.
Looking at the timetable shown above, it suggests that by altering the timing of the
call slightly, it might be possible to avoid the two-month hiatus over the summer
period.
5.2.3 Contracting and payments
Once the list of projects was approved the successful promoters would be informed
and the contracting process would begin. In the first four years of the programme it
was normal for projects to be accepted but for the budget allocation approved to be
significantly lower than that originally proposed. Thus the "contractualisation"
process often involved a renegotiation of the project and may have led to quite
significant changes in the project’s work programme. This practice was criticised by
the auditors, and had adverse effects in that it was not unknown for projects to inflate
their budgets artificially to take account of the expected reduction. In the final year of
the Programme a decision was taken to fund projects 100% of the requested budget,
except in cases where the budget was clearly shown to be overstated in the evaluation
process, or on the specific advice of the participating state.
Although there was an intention to complete the contractualisation process before the
end of the calendar year (see flow chart above), it was often difficult for the projects
to complete the complex forms and budget revisions in time for this to take place and
normally contracts would be signed in the first quarter of the following year. In the
case of 1998 and 1999 projects the situation was further complicated by the events
surrounding the TAO.
Projects that were not successful did not receive any feedback as to why their project
was not accepted.
Exhibit 5.2 Delayed Start Dates For The Projects
•
“Lots of beneficiaries mention the fact that they have only received the first
instalment/payment. The missing money in many cases blocks the project.”
(German project)
•
“The decision about the grant arrived far too late, the written decision arrived 7
months after the beginning of the first project phase”.(Austrian project)
This lengthy process had two major problems – firstly it was not uncommon for
projects to find that situations had changed significantly between the design of the
proposal and the point where the project could start. Secondly, a practice, much
criticised in the audit reports, of backdating contracts, sometimes by as much as 7
months, had grown up. This made life extremely difficult for projects who faced a
decision between starting the project before the contract was issued (thereby incurring
significant risk), or of starting after signature of the contract, but being unable to meet
the timetable.
Final Evaluation of Leonardo da Vinci
56
Deloitte & Touche
Each project would then receive an advance payment. The final payment would be
made on the basis of the final report and detailed financial statements. In the case of
projects with a duration of more than 12 months, an interim payment was also payable
based on an interim report and financial statement.
These reports and financial statements were the subject of close scrutiny and
evaluation, and detailed reports on them were drawn up by the TAO.
5.2.4 Follow-up and animation
This is acknowledged as one of the weakest areas in the central administration of the
Programme. To some degree this is a result of the events surrounding the closure of
the TAO in that the resource to undertake much of this work was lost just at the time
when projects were coming to fruition.
Paradoxically, the establishment of CLEO has given the possibility to maintain a Unit
dealing with the administration of the projects beyond the paper life of the overall
programme. Technical assistance offices normally end with the end of the Programme
leaving the closure and dissemination phases to be undertaken as an incidental activity
by the TAO for the successor programme. This situation seems unlikely to change in
the light of current discussions on the future of TAOs.
Contact, other than purely administrative, was not maintained with the projects during
their life-times, leading to a feeling of isolation reported by our sample.
In the participating states, and even between groups of states (eg the Nordic countries)
a number of animation activities were undertaken, but this was not consistent across
the countries and was not drawn together at European level in any meaningful way.
5.2.5 Strand III3b – Study Visits
Strand III3b of the Leonardo programme – the programme of Community study visits
for vocational training professionals – is handled separately from the Procedures 1
and 2 above, being managed by CEDEFOP, the European Centre for the Development
of Vocational Training. The Community study visits programme pre-dated Leonardo
by some 10 years, with the aim of activating information flows and exchange of
experience amongst vocational training specialists of Member States and associated
countries. The study visit programme is funded by means of a ring-fenced allocation
in the Commission’s annual grant to CEDEFOP. This money was not part of the
Leonardo budget.
Selection of participants for the study visits is the responsibility of National Liaison
Officers, civil servants nominated by their governments to work with CEDEFOP on
the implementation of guidelines and organisational procedures.
CEDEFOP, as a Community body established by a Council of Ministers’ decision of
1975, has a formal role as a source of information and centre of reference in
contributing to the development of vocational training in the European Union.
CEDEFOP does not have a formal role in respect of the Leonardo Programme’s
operations or pedagogical objectives beyond strand III3b.
Final Evaluation of Leonardo da Vinci
57
Deloitte & Touche
5.3
Problem Areas
5.3.1 Structural
The structure of the programme was extremely complex. This caused problems for
potential promoters in identifying which procedure, which strand was relevant for
their project. This situation has not been improved by the supporting documentation
which was also extremely hard to follow, especially for project promoters working in
languages other than their first language.
Some of the NCUs produced additional supporting information which made the
procedures much clearer, but in general it must be said that the documentation was not
“user friendly”. This need for clearer documentation was repeatedly expressed in our
survey.
There is a need for project promoters to build partnerships when drawing up
proposals. However, in the case of the pilot projects, very little assistance was given
in this process other than the partner search database (which came on stream quite late
in the life of the programme) and the annual fair (“contact and information days”) in
Brussels. An active role could be played by the NCUs in exchanging information on
potential partners. The current system is biased in favour of applicant organisations
which have existing networks (not that these should be discouraged) but makes the
development of new and sustainable partnerships comparatively difficult.
There is no real mechanism either at European or national level that is specifically
designed to take forward the lessons from the Programme. Activities have taken place
in some participating states but this has been lacking in co-ordination at a European
level. This relates not only to assistance/insistence on the diffusion or
commercialisation of products, but more generally to liaison with the wider
constituency of business, government and trade unions to mainstream the valuable
learning outcomes.
This lack of any real strategy at any level, from project to the overall programme, to
use the outputs of the programme is a serious weakness and, in terms of the long term
impact of the programme, probably the major one.
The administrative burden of the programme has meant that the negative aspects
associated with it have become the dominant factor, rather than a facilitating process.
It is not an exaggeration to state that many potential applicants for EU support are not
prepared to endure the delays and bureaucracy that the current systems display, and
are thus refusing to participate in the future.
5.3.2 Financial
The financial circuits of the programme for the centrally administered projects were
immensely long. Dossiers had to be verified by officials in three different departments
prior to any payment being made (TAO, DG XXII, and DG XX). This meant that the
time taken to pay claims was extremely long, even without the additional delays
caused by the investigations into the TAO.
Exhibit 5.3 Payment delays
•
“We have been waiting for the outstanding payments since the end of the project.
We do not understand the continued absence of written replies about this matter
in spite of the countless written requests we have made” (Italian project)
Final Evaluation of Leonardo da Vinci
58
Deloitte & Touche
•
“Due to loony periods of bureaucratic work in Brussels we have had an 8 month
delay! The interim report was not evaluated in accordance with the agreement (60
days) That constitutes a breach on the Commission’s side (it took 4 ½ months)”
(Danish project)
•
“Immense delays, through excessive bureaucratic procedures, especially
payments. Unfulfilled promises of payment dates only appears to be in the interest
of financial aspects, little if any in the technical aspects” (UK project)
The length of time taken to pay claims is extremely important as it has a strong
influence on the type of organisation which is able to act as a project promoter. SMEs
and non-profit organisations often simply do not have the cash flow to withstand this
type of problem, and thus improving their participation rate in the programme will be
extremely difficult until this issue is resolved.
Over the life of the Programme approximately 11500 payments had to be processed
centrally. Currently CLEO has a target of clearing 300 dossiers per month whereas the
TAO was managing to deal with approximately 100. It should be noted that payments
will need to be processed until December 2001.
Despite the fact that CLEO has achieved this level of processing, there are still two
next steps of the process – the verification and payment by the financial services
departments of DG Education and Culture and DG Financial Control. There has been
much internal debate between the services concerned. However, it seems that now
some progress has been made on reducing the backlog although there appears to be a
level of about 200 payments which is unlikely to be reduced given the resources
available. This suggests forcefully that the designers of programmes should have a
more realistic understanding of the resource implications of implementation, and that
processes should be designed to be as streamlined as possible.
Payments to the projects initially passed through the TAO. This added an extra stage
to the process. The TAO was thus charged with making payments to bodies with
which it had no direct contractual relationship. Further, questions were raised during
the audit as to whether the money advanced to the TAO was being managed in the
most productive manner. These issues were addressed following the audit report and
substantial changes were made. This situation (of allocating funds to the TAO to
manage) obviously no longer applies. It is doubtful whether it would apply to any
future technical assistance regime.
The whole co-funding regime was extremely onerous, requiring minute details in
justification of expenditures which were not always easily identifiable (eg additional
salary costs). In addition the involvement of the pre-accession countries complicated
matters in terms of financial reporting as it was necessary to account separately for all
related expenditure. We understand that steps have been taken in Leonardo 2 to
simplify the co-funding regime to attempt to reduce this problem. In addition,
advances in the enlargement process have reduced the administrative burden of
dealing with the pre-accession countries.
5.3.3 Procedural
In terms of the procedures, the main areas of weakness seem to lie in the following
areas:
Final Evaluation of Leonardo da Vinci
59
Deloitte & Touche
The Selection Process
By working with the NCUs it seems that information on the calls reaches a wide range
of operators and that the information is easy to access. Having received the
information, however, the complexity of the application process may discourage some
potential participants, especially those who have not previously been involved in EU
programmes.
The main concern however, lies in the fact that it took (in cases where no special
circumstances intervened) a period of almost one year between the publication of the
call and the notification of the successful projects. During this time many changes can
take place in the circumstances of the project/partnership, and what was innovative at
the time of the proposal may be less relevant at the end of the process. While there is
clearly a need for a transparent evaluation process, the degree of consultation and
negotiation involved in this case led to this extremely extended timetable.
This was made worse in the eyes of the projects by the fact that no feedback was
received, either during or after the process. It may be worth noting that in the case of
the Fifth Framework Programme calls, attempts have been made to introduce a
process of rapid response so that applicants are informed in a short time period
whether their application has passed the eligibility criteria and is going forward for
consideration.
Finally the practice of backdating contracts is not conducive to the proper
implementation of projects.
Bilateral negotiations
The outcomes of negotiation in the bilateral meetings meant that although projects
were initially selected on the basis of objective criteria, the ensuing discussion
introduced a number of other factors more related to national priorities. In some cases
this may have added clarification to the expert analysis but the process moved away
from the principle of total transparency, introducing a set of non-documented criteria.
Category 4 projects
An extreme example of this is the case of the Category 4 projects. As described
above, the evaluation process graded projects between 1 and 4, with 4 being classed
as “unacceptable”. However, in the course of the bilateral negotiations a number of
these projects were selected for inclusion on the final list. This clearly casts doubt on
the process of independent evaluation and was criticised in the audit report. These
projects may have been selected for their strategic importance, but this is not clear,
and leaves the whole selection process open to criticism. Should the evaluators refrain
from grading any projects as type 4 to avoid this criticism, the whole process would
be undermined.
Reduction of budgets
Once projects were selected, it was unusual in the 1995-1998 cycles for the full
budget requested to be allocated to the project. This then resulted in a process of
negotiation which could mean that the project as implemented was substantially
different from the project as evaluated – not a desirable situation. The alternative of
trying to implement the project as proposed but with reduced resources is equally
undesirable. A third negative effect of this practice is that in subsequent applications
projects deliberately overstate the financial requirements as they expect to be reduced
if successful, leading to an elaborate game and wasting time and resources.
Final Evaluation of Leonardo da Vinci
60
Deloitte & Touche
For the 1999 call this process was changed with only those projects with clearly
inflated budgets having their allocations reduced.
Lack of feedback and contact
The problem of lack of feedback and contact mentioned above in the selection process
was equally an issue for projects administered by the TAO. It is our understanding
that this was less of an issue for the decentralised projects, which is understandable as
the scale of the management task was much smaller.
Reporting
Although the projects produce very detailed final reports, and these are extensively
analysed, this analysis is undertaken in a very mechanical way and there is no
mechanism for recording or using the results of this analysis. This is discussed in
more detail below.
5.3.4 Institutional
Commission ability to process payments
Quite apart from the capacity of the TAO/CLEO to process the payment and
contracting elements of their tasks, serious questions also arise about the capacity of
the Commission itself to process the number of payments involved within a
reasonable timescale. This issue relates to both the resources of the Commission and
the internal processes which are currently acknowledged to be long and slow.
The structures for programme administration need to be designed with the capacity of
the Commission itself in mind to avoid the creation of bottlenecks. We have seen
examples of other programmes where the reporting and verification requirements are
much less cumbersome, while remaining open and transparent.
Responsibility of Member States/Commission
The division of responsibility between the Commission and the participating states is
the cause of several limitations in the programme implementation. Apart from the
cumbersome consultation procedures, at a practical level it has led to a series of
weaknesses, for example:
♦ Data is not consolidated and held centrally. This makes it difficult to analyse and
build on the results of the decentralised projects. This weakness was compounded
by the lack of a well structured data management system at the centre in which to
keep this data, had it been collected.
♦ While some interesting work was done by the NCUs in terms of animation and
dissemination, this was not co-ordinated or built on in any significant way. There
was no feeling of an implementation partnership, but more of a separation of
powers. This meant that many interesting opportunities were lost.
5.4 Costs, benefits and value for money
In total, and not including the costs of the Commission personnel involved in
programme management and financial control, the EU contribution to the
administration of the Leonardo da Vinci programme came to nearly 20 MECU per
annum, which represents over 13% of the Programme budget.
Final Evaluation of Leonardo da Vinci
61
Deloitte & Touche
According to the DG XXII Audit report of the TAO, the average cost of a person at
the TAO was 83 000 € compared to 108 000 € for a Commission official8. Thus, in
principle, the externalisation of the programme management represented a significant
saving for the Commission.
One can analyse the programme in terms of cost per participant, or other yardsticks.
However, it was not possible for us to make a detailed analysis of the outputs (see
chapter 6 below). However, the cost effectiveness of producing something that is not
exploited beyond the actual project has to be estimated as low.
5.5 The Interim Evaluation
The interim evaluation9 made a number of comments on the selection process, on
monitoring and evaluation and on the overall management of the programme. These
remained largely unaddressed in the remainder of the programme, and indeed in some
cases the problem was intensified rather than resolved. While this was often due to the
extraordinary circumstances surrounding the programme, nevertheless there is little
evidence of any attempt to take the recommendations of the evaluation on board.
The evaluation made a number of recommendations regarding dissemination,
highlighting already that this was a weak point in the programme, but for various
reasons, including crisis management and resource shortages, these were not
implemented.
8
SEC(1999)881 Vade Mecum Relatif aux Bureaux d’Assistance Technique
9
First external evaluation of the Action Programme for the implementation of a European Community
Vocational Training Policy, Ecotec Research and Consulting Ltd, March 1997
Final Evaluation of Leonardo da Vinci
62
Deloitte & Touche
6 Outputs
6.1 What were the outputs
This evaluation uses the term ‘outputs’ to describe the activities that took place in
Leonardo da Vinci, where, when, involving whom, and at what cost. This information
is presented below in tabular form. In addition to the activities listed, it considers
those outputs described as ‘products’ with particular respect to the pilot projects.
6.2 Outputs recorded centrally
The majority of the tables relate only to the data held centrally – thus do not include
information on projects operated under procedure 1. This information is reviewed in
6.3. Final data on the 1999 procedure 1 projects is not yet available, as the projects
have only just finished and the reporting cycle is not yet complete.
Over the period 1995 to 1999, some 3745 projects were financed, at a total cost to the
EU of €487.69 million. A further €151.04 million was spent on placements and
exchanges administered by the participating countries, and €87.78 million on support
and other measures (including the NCUs), giving an overall total of €726.7 million.
The total investment in Leonardo da Vinci, including the project participants’ own
contributions, was € 801,190,597.
Thus the EU contribution represented
approximately 66%. In total the projects involved approximately 3000 contracting
organisations. Overall involvement in the partnerships totalled 77,274 organisations
altogether. This figure includes some duplication (especially in the case of the
mobility projects where organisations may figure as potential hosts in several
projects) but is nonetheless impressive.
In terms of priorities, it is difficult to present the data, as there was a change in
priorities between 1995 and the remaining years. However, figure 6.1 below shows
the breakdown for the years 1996-1999.
Final Evaluation of Leonardo da Vinci
63
Deloitte & Touche
Figure 6.1: Number of Projects by Priority
350
300
250
96
97
98
99
200
150
100
50
0
1 Acquisition of
new skills
2 Closer links
between
education and
training
establishments
and enterprises
3 Combating
exclusion
4 Investment in
5 Use of
Human Resources information tools
and technologies
In terms of the breakdown by strand10, the situation for 1995-1999 was as follows:
Figure 6.2: Distribution of projects by strand
25.00%
20.00%
15.00%
% of Budget
% of Projects
10.00%
5.00%
10
see end of chapter for a list of the strands
0.00%
I.1.1.a
I.1.1.b
I.1.1.c
I.1.1.d
I.1.1.e
Final Evaluation of Leonardo da Vinci
II.1.1.a II.1.1.b
II.1.1.c II.1.1.d II.1.2.a II.1.2.b
II.1.2.c
III.1.a
III.1.b
III.2.a
III.3.a
64
Deloitte & Touche
Strand II.1.2.a – in-company training placements for university students and young
graduates had the largest number of projects (and budget), although the pilot projects
on training to cope with technological change and investment in continuing vocational
training for workers also represented a significant proportion of the resources. The
detailed expenditure by strand is shown in table 6.2.1 below.
In terms of distribution by country, a broad pattern is clear, with France leading the
largest number of projects. However, there is some variation between years,
indicating that there was at least marginal flexibility in the distribution of projects:
Figure 6.3: Distribution by country by number of projects
18%
16%
14%
1995
1996
1997
1998
1999
12%
10%
8%
6%
4%
2%
Be
lg
D ium
en
m
G ark
er
m
a
G ny
re
ec
Sp e
ai
Fr n
an
c
Ire e
la
nd
Ita
Lu
ly
xe
m
N bo
et
he ur
rla
n
Au d
st
P o r ia
rtu
g
Fi al
nl
an
Sw d
U
ni
ed
te
en
d
Ki
ng
Ic do
el
Li
an
ec
d
ht
en
st
N ei
or
w
a
C y
C
yp
ze
ru
ch
s
R
ep
u
Es b
to
H nia
un
g
Li ary
tu
an
i
La a
tv
Po ia
la
nd
R
Sl
ov oma
ak
ni
a
R
ep
Bu ub
lg
a
Sl ria
ov
en
ia
0%
Final Evaluation of Leonardo da Vinci
65
Deloitte & Touche
A similar representation on the basis of budget shows that although France had
consistently the most projects, the greatest budget went to Germany, with France,
Italy and the UK at similar levels following them.
Figure 6.4: Division of budget by country by year
16%
14%
12%
10%
1995
1996
1997
1998
8%
6%
4%
2%
D
Be
lg
iu
en m
m
G ark
er
m
an
G y
re
ec
e
Sp
ai
n
Fr
an
c
Ire e
la
nd
Lu
xe Ita
m ly
b
N
et our
he
g
rla
nd
s
Au
st
r
Po ia
rtu
ga
l
Fi
nl
a
n
U
ni Sw d
te
d ed
Ki en
ng
do
m
Li Ice
ec
la
ht nd
en
st
ei
N n
or
w
a
C
ze C y
ch yp
R rus
ep
ub
li
Es c
to
ni
H
un a
ga
Li ry
tu
an
ia
La
tv
i
Po a
la
Sl
nd
R
ov om
ak
a
R nia
ep
ub
lic
0%
Final Evaluation of Leonardo da Vinci
66
Deloitte & Touche
TABLE 6.1 MEASURES GLOBAL
1995
1996
1997
1998
1999
TOTAL
Measures
Number
of
Projects
LEONARDO
Grants (M€)
Number
of
Projects
LEONARDO
Grants (M€)
Number
of
Projects
LEONARDO
Grants (M€)
Number
of
Projects
LEONARDO
Grants (M€)
Number
of
Projects
LEONARDO
Grants (M€)
Number
of
Projects
LEONARDO
Grants (M€)
I.1.1.a
93
10.90
79
11.84
71
10.22
83
12.64
82
8.82
408
54.42
I.1.1.b
58
6.55
60
9.24
51
7.60
69
9.07
57
5.24
295
37.71
I.1.1.c
20
2.02
21
2.47
22
2.89
25
3.34
18
1.74
106
12.46
I.1.1.d
18
1.99
14
2.09
12
1.80
18
2.47
6
0.67
68
9.02
I.1.1.e
38
4.26
56
7.83
41
6.25
47
6.86
40
4.45
222
29.65
II.1.1.a
127
14.31
103
14.90
78
11.94
71
11.66
45
4.58
424
57.39
II.1.1.b
81
9.72
110
16.67
88
13.00
83
11.79
56
6.21
418
57.39
II.1.1.c
57
8.49
69
10.13
45
7.07
38
6.60
35
3.20
244
35.49
II.1.1.d
20
2.33
19
2.74
12
1.71
13
2.20
12
1.14
76
10.12
II.1.2.a
102
13.75
144
20.59
150
23.11
137
16.59
180
23.58
713
97.62
II.1.2.b
11
0.38
11
0.37
21
0.74
13
0.27
21
0.63
77
2.38
II.1.2.c
5
0.10
12
0.46
13
0.45
11
0.43
14
0.31
55
1.75
III.1.a
43
5.05
30
4.39
38
5.79
40
5.39
23
2.63
174
23.25
III.1.b
3
0.10
3
0.03
4
0.32
1
0.00
2
0.10
13
0.56
III.2.a
47
6.85
41
6.03
47
7.92
35
5.42
0
0.00
170
26.22
III.3.a
26
2.86
21
2.34
37
4.54
37
4.19
161
18.37
282
32.30
TOTAL
749
89.66
793
112.14
730
109.22
721
94.24
752
81.67
3745
487.69
Source: European Commission, DG EAC
Final Evaluation of Leonardo da Vinci
67
Deloitte & Touche
TABLE 6.2 COUNTRIES GLOBAL (all amounts in €)
Countries
199511
Number of
Projects
Belgium
Denmark
Germany
Greece
Spain
France
Ireland
Italy
Luxembourg
Netherlands
Austria
Portugal
Finland
Sweden
United Kingdom
Iceland
Liechtenstein
Norway
Bulgaria
Cyprus
Czech Republic
Estonia
Hungary
Lithuania
Latvia
Poland
Romania
Slovenia
Slovak Republic
EUR projects
TOTAL
11
1996
1997
1998
1999
TOTAL
LEONARDO Number of LEONARDO Number of LEONARDO Number of LEONARDO Number of LEONARDO Number of
Grants
Projects
Grants
Projects
Grants
Projects
Grants
Projects
Grants
Projects
31
29
143
59
112
151
32
137
11
40
30
45
32
30
138
7
2
21
3,358,755
2,734,395
16,168,646
4,993,941
10,228,773
13,687,112
3,331,803
13,724,022
1,448,200
4,412,364
3,078,035
4,004,224
3,066,287
3,037,980
14,015,851
750,925
210,000
2,100,542
24
2,323,010
1,074 106,674,865
24
20
103
35
86
117
27
105
8
25
20
27
20
23
105
5
1
15
3,649,178
2,381,300
16,822,573
5,157,900
11,214,134
14,521,429
3,617,458
14,308,125
1,237,850
4,463,321
3,220,000
3,985,925
3,117,673
3,191,798
14,311,789
626,398
170,000
2,190,165
27
3,890,806
793 112,077,822
26
16
86
28
82
113
21
94
7
19
20
22
19
18
87
7
1
15
3,810,651
2,154,060
15,818,667
4,523,570
10,583,897
14,034,766
2,939,157
13,022,925
1,246,103
3,747,813
3,912,653
3,768,937
3,115,539
2,944,805
13,054,088
814,426
154,000
2,255,210
28
16
85
22
70
100
18
81
7
19
22
20
18
17
81
5
1
11
3,820,684
1,999,042
12,290,358
3,349,009
8,690,802
11,097,282
2,194,438
10,701,513
882,337
3,103,405
2,884,610
2,648,775
2,517,726
2,378,410
10,795,192
513,365
160,000
1,525,021
4
2
613,311
343,600
14
2,492,406
2
273,600
5
19
4
14
6
3
17
25
431,282
2,078,612
467,751
1,422,995
808,774
520,900
2,747,663
3,118,764
27
3,446,259
730 109,070,443
14
0
728
1,755,505
0
94,904,215
25
14
81
26
76
104
16
82
4
22
20
24
25
16
72
5
1
9
9
2
18
2
15
10
5
22
26
5
17
0
753
2,709,605
1,804,304
12,899,744
3,595,110
8,864,066
11,172,558
2,313,430
10,518,441
590,626
3,285,593
2,951,394
3,115,464
2,448,373
2,300,256
10,641,608
543,213
100,000
1,072,837
1,114,429
265,050
1,548,474
296,226
1,448,365
775,409
598,150
2,875,780
2,589,492
509,495
1,638,593
0
94,586,085
134
95
498
170
426
585
114
499
37
125
112
138
114
104
483
29
6
71
9
11
39
6
43
16
8
39
53
5
31
78
4,078
LEONARDO
Grants
17,348,873
11,073,101
73,999,988
21,619,530
49,581,672
64,513,147
14,396,286
62,275,026
5,405,116
19,012,496
16,046,692
17,523,325
14,265,598
13,853,249
62,818,528
3,248,327
794,000
9,143,775
1,114,429
1,309,643
3,970,686
763,977
5,363,766
1,584,183
1,119,050
5,623,443
5,981,856
509,495
3,394,098
9,660,075
517,313,430
Includes 1995R project renewals
Final Evaluation of Leonardo da Vinci
68
Deloitte & Touche
TABLE 6.3 Number of beneficiaries for the selected projects by country and by
strand
Country
Strand I
Strand II
B
DK
D
EL
E
F
IRL
I
L
NL
P
UK
S
A
FIN
N
IS
FL
RO
HU
CZ
CY
PL
SK
EE
LT
LV
BG
SI
EUR – Projects
TOTAL
2.059
789
16.434
1.045
4.525
6.381
656
6.743
89
1.425
1.518
6.550
1.042
1.521
864
995
224
13
819
357
464
480
69
154
255
54.513
1.059
559
7.103
1.016
4.214
7.108
1.498
5.058
263
2.012
1.299
4.351
1.049
1.595
1.270
585
255
Strand III
164
4
37
175
107
40
12
61
122
45
17
282
40.616
205
Total
3.118
1.348
23.701
2.061
8.739
13.489
2.154
11.805
352
3.437
2.817
10.938
2.091
3.116
2.134
1.580
479
13
994
464
504
12
541
122
69
199
272
282
95.334
For Strand I : only figures for 1995 – 1998
II and III :figures for 1995 - 1999
Source: European Commission, DG EAC
Final Evaluation of Leonardo da Vinci
69
Deloitte & Touche
TABLE 6.4 Number of selected projects by country and by strand
Country
B
DK
D
EL
E
F
IRL
I
L
NL
P
UK
S
A
FIN
N
IS
FL
RO
HU
CZ
CY
PL
SK
EE
LT
LV
BG
SI
EUR – Projects
TOTAL
Strand I
200
116
1.940
71
373
471
92
289
18
116
173
404
117
123
120
132
67
1
75
37
59
16
8
18
53
4.994
Strand II
23
21
99
17
11
171
32
94
4
22
28
75
20
23
33
15
8
Strand III
1
Total
4
1
224
137
2.043
88
484
644
124
384
22
138
201
484
137
146
153
147
75
1
90
44
63
1
1
20
8
22
54
13
622
13
5.829
4
2
1
5
15
7
4
1
1
4
13
For Strand I : only figures for 1995 – 1998
II and III :figures for 1995 – 1999
Source: European Commission, DG EAC
Final Evaluation of Leonardo da Vinci
70
Deloitte & Touche
Training organisation
Figure 6.5 Contractors by type of organisation
U niversity
1%
1%
2%
2%
U niversity enterprise training partnership
2%
O ther organisations
2%
2%
Public authorities
3%
27%
Small and medium sized enterprise ( less than 2 5 0
workers)
3%
Research centres or institutes
3%
Professional organisations/ federations/ groupings
Chamber of commerce, industry, agriculture
4%
Trade union organisation
Employer organisation
4%
Large enterprise ( 2 5 0 workers and more)
Group or association of companies
5%
Joint body
18%
7%
Regional consortium
Sectoral organisation
European organisations
7%
9%
Final Evaluation of Leonardo da Vinci
O rganisations concerned with certification and
recognition of qualifications
71
Deloitte & Touche
Figure 6.6 Type of organisation as partner
Small and medium sized enterprise (< 250
workers)
University
University enterprise training partnership
Training organisation
Large enterprise (2 50 workers and more)
O ther organisations
Public authorities
Research centres or institutes
Professional organisations/ federations/ groupings
Chamber of commerce, industry, agriculture
Group or association of companies
Trade union organisation
Employer organisation
Regional consortium
Sectoral organisation
Joint body
O rganisations concerned with certification and
recognition of qualifications
6.3 Outputs recorded at national level
For the period of 1995-199912, placement and exchange projects under Strand I were
co-financed with an amount of €162.49 million (€61.36m for young people, €83.74m
for young workers and €17.38m for instructors). The beneficiaries were 51,047 young
people, 21,486 young workers and 10,750 trainers, giving a total of 83,283
participants from about 6,516 projects.
Under Strand II young graduates, staff from universities or enterprises and intercompany exchanges, received €105.10 million in co-financing, for 822 projects
involving 40,616 beneficiaries.
Strand III for language instructors involved only 205 beneficiaries from 13 projects at
a cost of €0.99 million.
Thus in total nearly 125,000 beneficiaries participated in these mobility actions, at a
cost of some €270 million. More than a third (35%) of these projects were led from
Germany – 1243 out of a total of 5829, with France and the UK having the next
highest proportion at 644 and 484 respectively.
No information was available on where the participants undertook their placements,
although this should be available from the national reports.
However, although the NCUs have data on participants leaving their country and
destinations, they have no information on participants arriving in their countries from
12
1999 figures are estimates subject to confirmation
Final Evaluation of Leonardo da Vinci
72
Deloitte & Touche
elsewhere – a useful piece of information for which there were apparently no central
collection facilities foreseen.
6.4 Quality of outputs
The emphasis on detailed quantitative reporting at the European level appears to have
resulted in downplaying systematic appraisal of the quality of the Leonardo da Vinci
processes (managerial/networking etc); of the personal experiences (placement and
exchange); or of the products (pilot projects). The evidence gathered from
questionnaire responses and from structured interviews suggests, however:
•
a significant level of satisfaction amongst promoters with the opportunities
provided by Leonardo da Vinci for development of project management skills,
establishment of networks and information exchange at the European level;
•
a significant level of satisfaction amongst participants, irrespective of age, in the
personal development opportunities afforded by the mobility projects;
•
a feeling amongst promoters of pilot projects that the quality and utility of their
products was neglected by Brussels, which paid far more attention to
administrative and procedural considerations.
6.5 Products - Dissemination
From the launch of the Leonardo da Vinci programme emphasis was placed by the
Commission on the importance of dissemination of those outputs described as
products. Each pilot project was required to specify anticipated products of the
training /development activity in the framework of a dissemination plan. There was no
definitive ex-ante classification of products, but Commission guidance provided in the
course of Leonardo da Vinci 1 suggested that the term “can be taken to mean the
tangible outcomes of projects funded by the European Community. Examples are
training manuals, training course curricula, training software etc., or surveys and
analyses funded as research”. (In the terminology of this evaluation the term
'outcomes' is used to refer to benefits accruing to a target group or groups as a result
of intervention; the tangibles are termed 'outputs' as noted at the beginning of this
section.)
The guidance (produced in 1997) went on to offer pointers regarding issues
concerning the commercial exploitation of products. The Commission view was that
dissemination also applied to the processes involved in pilot projects – managerial
lessons learned, be they positive or negative (in the spirit of the programme as a
'laboratory of innovation'). Equally, in mobility projects, the guidance considered that
there were 'experiences, methodologies and lessons' worthy of dissemination. Thus
each type of project would be expected to have some tangible message to relay to the
outside world.
Methods of dissemination suggested by the Commission included:
•
press
•
newsletters
•
publications
•
fairs and exhibitions
Final Evaluation of Leonardo da Vinci
73
Deloitte & Touche
•
conferences and seminars
•
internet
Such dissemination was to be supported by the Commission's own communications
strategy, in association with the central TAO, including:
•
databases of past projects, products and results which 'can be used by promoters
past, present and future, to see what other work is being done that connects with
their work, thereby creating the potential for links between projects and avoiding
repetition. For example, the Leonardo da Vinci Technical Assistance Office has
developed a database of products from past programmes (ie PETRA, COMETT,
FORCE, EUROTECNET and LINGUA) which can be consulted at the
Multimedia Centre”;
•
product fairs at a European level;
•
a Multimedia Centre, established by the TAO “where visitors can look at products
developed under previous programmes (and, when the projects are completed and
products received, also from the Leonardo da Vinci programme), and examine the
pedagogical, methodological and technical aspects of product development”.
The Multimedia Centre was able to take account of the tendency amongst promoters
to change their product media in the wake of fast-moving ICT developments: initial
emphasis on audio and videocassettes for example, gave way to proposals for CDROM products. Expense and quality issues relating to the latter led to the later interest
in the development of web sites.
Problem Areas
The Commission's ambitions with respect to the development, classification and
dissemination of Leonardo da Vinci products appear to have been frustrated by the
programme's emphasis on managerialism and unnecessary bureaucracy and by the
collapse of the Technical Assistance Office. Survey respondents have commented, for
example:
'Nobody seems interested in the quality of the work' (private sector promoter, UK)
'The administrative demands detract from matters of substance' (professional
association promoter, Denmark)
'Only appears to be interest in the finance aspects; little, if any in the technical aspects'
(private sector promoter UK)
'Dissemination of results by the Leonardo da Vinci administration is very inadequate'
(training organisation promoter, Spain)
The Commission's own Leonardo da Vinci communications strategy was interlinked
with the work of the collapsed TAO, so that, for example:
•
the envisaged product databases are not available to visitors;
•
the multimedia centre no longer exists;
•
for those Leonardo da Vinci projects which have finished, product evaluation is
limited to verifying technical conformity with contract;
Final Evaluation of Leonardo da Vinci
74
Deloitte & Touche
•
there is no complete catalogue/database of Leonardo da Vinci products so far;
•
there has been only one product fair at European level. A second, planned for
January 2000, was cancelled because 'a suitable room' could not be found in
Brussels.
CLEO, the successor technical assistance unit, is under pressure to resolve the
substantial remaining administrative backlog in relation to projects and does not have
the resources to continue work on product classification and dissemination. Earlier
initiatives, such as one to publish up to 20 examples of good practice based upon
research and analysis projects, have had to be abandoned. As a result of their work on
evaluation/verification of final reports CLEO staff know broadly what exists in
respect of outputs from finished projects and take the view that there are high quality
products amongst them. It appears that there is, however, neither the operational
priority nor the organisational resource to do anything about them.
The effects
As it is, the Leonardo da Vinci programme appears to have lacked (or lost):
• a Europe wide approach to the dissemination of products
• a catalogue of products, even as a simple list
• the provision of good-practice case material and guidance
• high profile marketing events
• a realisation of the growing political importance of vocational training
particularly in respect of the EU employment strategy
The loss of leadership and resource in classification and dissemination of Leonardo da
Vinci products at a European level has further adversely affected the programme's
credibility amongst promoters and NCUs.
Perhaps more important, it could diminish the authority and capacity of the
programme to inform and influence the development of Vocational Education and
Training policy at the EU level at a time when vocational training is recognised more
and more as a crucial input to the Union's socio-economic policies, particularly in
respect of the maintenance and growth of employment. It is no exaggeration to state
that not having even a basic directory of the products and outputs of a programme that
absorbed over €500 million of public money is a completely unacceptable lapse.
The need
In order to help re-establish the professional potential of the Leonardo da Vinci
programme there is a need for the Commission to lead a new initiative, perhaps in
conjunction with CLEO, CEDEFOP, the NCUs and the Leonardo da Vinci networks,
to produce a capitalisation strategy for the products of both Leonardo da Vinci 1 and
II. This initiative could include
•
a properly segmented marketing plan at a European level;
•
a comprehensive, inventory of Leonardo da Vinci products, classified by category;
Final Evaluation of Leonardo da Vinci
75
Deloitte & Touche
•
a resumption of work on good practice models and guidance;
•
a re-instatement of the European Leonardo da Vinci product fair;
•
a programme of promotional events, including a joint Commission/Parliamentary
seminar to emphasise the political importance of Vocational Education and
Training.
Given the obvious interest in vehicles like the CEDEFOP Electronic Training Village
(over 7000 registered residents to date) and the new initiative for a European Learning
Gateway, it would be logical to ensure that the fuller development of these tools make
provision for an effective capitalisation of Leonardo’s results.
Final Evaluation of Leonardo da Vinci
76
Deloitte & Touche
Leonardo da Vinci 1 – The strands
Strand 1
support for the improvement of vocational training systems and arrangements in the
Member States
I.1.1.a
Projects to improve initial training systems
I.1.1.b
Projects to improve continuing training systems
I.1.1.c
Projects concerning vocational information and guidance
I.1.1.d
Projects to promote equal opportunities
I.1.1.e
Projects to combat exclusion
I.1.2.a
Placement programmes for young people in initial vocational training
I.1.2.b
Placement programmes for young workers
I.1.2.c
Exchange programmes for training programme instructors, planners and
administrators.
Strand 2
support for the improvement of vocational training measures, including
university/enterprise co-operation, concerning enterprises and workers
II.1.1.a
Projects concerning innovation in vocational training for workers
II.1.1.b
Projects concerning investment in continuing training content and methods
II.1.1.c
Projects concerning industry/university co-operation
II.1.1.d
Projects to promote equal opportunities
II.1.2.a
In-plant training placement programmes
II.1.2.b/c Exchange programmes
Strand 3
support for the development of language skills, knowledge and the dissemination of
innovation in the field of vocational training
III.1.a
Language training projects
III.1.b
Projects for the exchange of language teachers
III.2.a
Survey and analysis projects
III.2.b
Exchange of comparable data
III.3.a
Multiplier projects for the dissemination of prior results
III.3.b
Programme of study visits
Final Evaluation of Leonardo da Vinci
77
Deloitte & Touche
7 Impact
7.1 Approach
In this chapter we outline the impact of the Leonardo programme. The chapter
examines impact at each of the different levels of the project: European, Member
State, projects, and participants. In addition, the impact of Leonardo is discussed for
different target groups (women, social partners, SMEs, marginalised groups).
We can employ a number of different approaches to understanding the impact of
Leonardo at these different levels. In Chapter 6 we have already identified and
described the outputs of the programme. Programme outputs are an important aspect
of impact, constituting the measurable and quantifiable side of impact analysis. As we
saw in Chapter 6 it has been possible to quantify the numbers of activities undertaken
by different projects, to measure the numbers of participants in the different strands
and so on. Impact, however, extends beyond what can be measured and quantified. A
programme like Leonardo can have a far-reaching impact on the lives of the
individual participants, and can lead to unexpected side effects and spillovers into
separate but related areas of activity for the projects. At the level of the Member State
and EU, the impact of Leonardo may reshape new policies and programmes and may
extend to other policies and programmes. In this sense the impact of Leonardo reflects
an intrinsic process of organisational learning. Impact is therefore equally conditioned
by the receptiveness of the implementation environment as much as by the design and
quality of the programme.
In the sections that follow we therefore draw on data which reflects the
understandings of the key stakeholders as to their experiences of the programme. We
employ a holistic approach, drawing on multiple aspects of the programme
environment in order to put together a comprehensive picture of the impact and
dynamics of the programme. By drawing on the qualitative indicators of impact we
can direct greater attention to context, and to the development of in-depth
understandings of the complex interaction between programme setting and
programme design. This is important as it shapes and determines the nature and depth
of programme impact at the different levels and spheres of operation.
In later sub-sections we give particular emphasis to the experiences and
interpretations of Leonardo held by the projects and participants. The experiences of
Leonardo tend to be shared experiences, subtly differentiated by context but –
nevertheless - meaningful for the participants and projects that have engaged directly
with the projects. In the paragraphs which follow we therefore stress the shared
elements of experiences of the programme at the different levels of its operation. This
chapter, in conjunction with our analysis of programme outputs (Chapter 6), provides
us with a complete and comprehensive understanding of impact which is built on both
quantitative measures and qualitative holistic data.
Before considering impact at the level of projects, participants and target groups, we
first turn our attention to the European and Member State levels of programme
operation. At these levels there is clear evidence that the programme impact has been
directly conditioned by the organisational dynamics which have shaped the
implementation environment as well as by the structural characteristics of the
programme which we have already outlined in Chapter 4.
Final Evaluation of Leonardo da Vinci
78
Deloitte & Touche
7.2 Impact of Leonardo at the EU and national levels
Time has shown that the capacity of Member States to operate European programmes,
coupled with their varying ability to integrate European activities into national policy
making, and the tendency to allow programmes with major budgets to dominate
attention, all played key roles in reducing Leonardo’s ability to command serious
attention at high level within the Member States. In addition, an influence on policy,
or general practice can be achieved through direct access to senior officials (assuming
that they are really in a position to develop innovative policy and to change systems)
or through the spreading of high quality concepts and ideas at grass roots level. In the
case of Leonardo da Vinci, it is too early to assess if such a cause and effect
relationship exists.
We have referred above to the rather disappointing role played by the Leonardo
Committee.
Not surprisingly, many of the project representatives state their view that the
programme has influenced the national environment (part of a justifiable sense of
pride in their work, perhaps?). Those NCUs who were in a position to promote the
themes and initiatives of LEONARDO (such as Germany) also feel that the
programme had an impact in their Member State. But this is not (yet) discernible in
policy or system terms. Given the fact that many projects are not yet finalised, and
that the products have not been effectively presented, it would be illusory to expect
much evidence at this stage. There remains the reality that the more individuals,
organisations and policy makers work in a European Framework, the more likely it is
that European-level message will filter through into their national environment. This
process is given a useful push by the debate surrounding the creation of the Euro, the
convergence of economic policies and the Luxembourg employment process.
7.3
Impact of Pilot Projects
7.3.1 Motivation and Expectations
It is interesting to compare the main motivations and expectations held by the projects
at the start of their engagement with the Leonardo Da Vinci programme, with their
reported experiences of its impact. On the whole the projects appear largely satisfied
that their objectives have been met, while there is widespread dissatisfaction with the
delays and financial costs of the project.
Although there was general dissatisfaction with the financial mechanisms of Leonardo
we are reminded by many of the projects that the prospect of financial support was
instrumental in ensuring the start up and continuation of many of the projects - as one
project says:
“the project would not have been implemented without the support of Leonardo”
(Dutch project)
It is regrettable that stakeholders who clearly welcome and often depend on the
Community financial contribution are so often subjected to worry and delay because
of a dysfunctional administrative system.
Final Evaluation of Leonardo da Vinci
79
Deloitte & Touche
7.3.2 Perceptions of Innovative Character
One impact of the Leonardo da Vinci programme, which was possibly not anticipated
by the original designers of the programme, concerns the widespread perception held
among the pilot projects that their projects were particularly innovative. This is an
inevitable result of the selection processes applied to the pilot projects which placed
emphasis and standing on the projects which were ‘innovative’. Perhaps not
surprisingly, the pilot projects consistently reported that their projects were
‘innovative’.
It is a common place in European development programmes that the definition of
innovation is something which is particularly hard to achieve and is determined very
considerably by implementation context. We find a similar phenomenon in the
vocational and educational training domain, and in Leonardo projects particularly. For
this reason it becomes very difficult to assess the genuine innovativeness of projects,
as what may be considered innovative in one context is less so in another.
Projects gave a number of reasons for the innovative character of their activities.
These may be summarised as follows:
•
The training methods, products and materials were perceived to be intrinsically
innovative
•
IT was used in the projects
•
The trans-national element of the project was perceived to be innovative,
•
The training material was felt to be innovative because it was the first time it had
been applied in the domain in which the project operated
•
The project was held to be innovative because it was inherently flexible
•
The project was perceived as innovative because it responded to need
7.3.3 Trans-nationality and partnerships
“a good learning opportunity although frustrating at times” (UK project)
One of the most important outcomes of the Leonardo programme was, as we have
seen earlier, the institutionalisation of a wide network of trans-national contacts and
the sharing and exchange of skills working practices and cultures across the European
arena. This must be regarded as a highlight of the programme which has almost
universally been applauded and welcomed by both individuals and pilot projects alike.
The success of the trans-national element of Leonardo may be grouped around a
number of causal factors:
1. The projects benefited from exchange of experience and co-operation with other
projects i.e. learning new skills
2. The trans-national dimension entailed opportunities for learning new cultures and
this was also highly valued
3. The involvement of the projects in both national and trans-national partnerships
was also highly valued
Final Evaluation of Leonardo da Vinci
80
Deloitte & Touche
The projects make many statements about the benefits they have gained from
participating in the partnerships and transnational exchanges which are inherent to
Leonardo. Exhibit 7.1 overleaf gives exemplars of these reported experiences. The
words of the projects themselves perhaps illustrate this best. We can translate these
reported experiences into statements about impact:
1. There is evidence of organisational learning in terms of:
ƒ
acquisition of new substantive, work related skills;
ƒ
learning about inter-organisational relations – negotiation, bargaining, cooperation, team working, the value of and skills required for networking;
ƒ
administrative learning centred on managing complex administrative
procedures and multiple teams from different countries and organisations.
2. Projects have been increasingly sensitised to the added value that can be obtained
by being open to learning from other parts of Europe. Leonardo has therefore been
an educational experience which has contributed the process of “Europeanisation”
of civil society and has raised the level of European consciousness.
3. A process of partnership capacity building and enhancement has been initiated or
continued among the Leonardo projects. Understanding and successfully
participating in partnerships is something which requires time to learn and
develop. The Leonardo project has contributed to continuing learning about
partnership in the Vocational and Educational training sector.
Exhibit 7.1: Transnational Partnerships in Leonardo
1. Benefit of exchange of experience and co-operation with other projects i.e.
learning new skills
• “the company has gained considerably from working with the partner
organisations” (Irish project)
•
“Leonardo has improved and increased the definition and the exchange of
products and methodologies at European level” (Italian project)
•
“The co-operation with the partners was especially important for us. It has been
possible during the project implementation to collect new knowledge and
exchange experience about professional training in Europe”. (Austrian project)
•
“Results of the project: new methods/approaches in connection with new situation
at work, for training and in the social field were developed. & Important
international networks exist now”. (German project)
2. Trans-national dimension learning new cultures also valued
•
“all our advisers increase your expectatives about European dimension” (sic)
(Spanish project)
•
“Leonardo funding made it possible to create a common project with five nations
represented. This was a necessary condition to achieve a “pan – European”
Final Evaluation of Leonardo da Vinci
81
Deloitte & Touche
standard for securing of loads” (Swedish project)
•
“Intensification and increase of the meaning of European topics in training and
education. For this, following condition is required: Chambers of Commerce as
responsible/competent body for European projects”. (German project)
3. Partnership element valued
•
“we have created a consortium to co-operate in new projects thanks to the
Leonardo programme” (Spanish project)
•
“We found now stable and reliable partners with whom we want to implement and
manage other European projects.” (Austrian project)
The pilot projects also frequently observed that partnerships were also burdensome to
administer and to manage. Complaints were made about absent partners who left the
burden of activity on the shoulders of the remaining partners. However, given what
we already know about the developmental trajectory necessary for the acquisition of
partnership capacity, we can see these complaints as demonstrating evidence of the
early stages of the formulation of partnership capacity.
•
“there was difficulty with partners from different countries to keep to deadlines
and maintain momentum” (UK project)
•
“in theory it was a partnership of equals but the ineffectiveness of one partner
upset the equilibrium of the partnership”(Irish project)
•
“From the beginning it was a partnership of equals but from the ‘Leonardo affair’
most of the partners lost interest and the project was dominated by the lead
organisation” (Swedish project)
7.3.4 Managerial issues: Institutional performance
In spite of the very positive message coming from the projects about the valuable
impact of Leonardo in stimulating organisational leaning and Europeanisation at the
project level, when we consider the impact of Leonardo in the sphere of managerial
and financial matters the experience is very different.
“The strategy and objectives of the Leonardo programme as a concept is brilliant. The
way it is organised is dreadful” (Irish project)
Almost without exception projects have spoken of:
•
staff intensive management & high administration costs;
•
complicated administrative and reporting requirements;
•
administration not handled effectively – massive delays;
Final Evaluation of Leonardo da Vinci
82
Deloitte & Touche
The questionnaires and project case studies provide ample testimony to the difficulties
experienced.
™ Staff intensive management & High administration costs
Inevitably this has not been without significant financial and time costs to projects.
Inability to predict the cost has been a major issue for the projects.
“The energy required for management of the project is at least the double as foreseen
to reach the fixed objectives.” (German project)
™ Complicated administrative and reporting requirements.
Here projects suggest different solutions to the problem. Some ask for a help desk,
others suggest streamlined management in favour of the projects, the provision of
guidelines and models for the completion of the administrative documents and so on.
•
“the requirements on the reports to Brussels were far too complicated” (Swedish
project)
•
“involvement of our organisation in the Leonardo programme was very
worthwhile… any simplification of administration/paperwork would be most
welcome!” (Irish project)
•
“the financial administration of the project was trial and error. There should be
more adequate instruction on this, perhaps even a help desk” (Dutch project)
™ Administration not handled effectively – massive delays
Projects make specific complaints about both the Commission and the TAO. The
projects speak very directly on the subject of payment delays.
•
“speed of response has been slow and always has been. However I feel staff have
been both fair and thorough” (UK project)
•
“Our dissatisfaction is due to the delays in payments, the excessive bureaucracy,
and the consequences which arose out of the intervention of the Belgian Court of
Justice.” (Spanish project)
•
“The management of the budget has become intolerable for the contractor”
(Belgian project)
•
“Percentage funded by Commission? - We will tell you as and when the
Commission pay the last tranche they are 2 years late.….we are currently
envisaging legal action if a solution to our final payment is not found rapidly”
(French project)
The operation of the TAO has also been the subject of pointed criticism by the
projects:
Final Evaluation of Leonardo da Vinci
83
Deloitte & Touche
•
“One administrative task of TAO (especially the payments) is extremely delayed,
requests for information are not answered only until we decided to call in a
collection agency (Dutch project)
•
“The TAO worked very inefficiently, contacts were very late. No transparency in
the way reports were processed, little feedback. Payment late” (French project)
Conversely, a smaller group of projects have applauded the excellent support of the
NCUs (see Chapter 4).
The chief impact of the operational difficulties experienced by the Commission in
keeping to deadlines and meeting its payment obligations is that many projects report
that they would not repeat the experience of Leonardo da Vinci because of the delays.
The same projects that say that they will not participate in Leonardo again also report
the benefit they have obtained as the result of the trans-national exchanges and
partnership arrangements which were a feature of Leonardo. It follows that many
projects who would not become involved again with Leonardo in its current form,
would nevertheless be interested in involvement in subsequent programmes which
draw out the positive aspects of the Leonardo experience without repeating the
negative administrative experiences.
7.3.5 The Flexibility and Adaptability of the Programme
Projects allude to the flexibility of the Leonardo programme. This has undoubtedly
been a direct benefit to the pilot projects. We can identify at least three ways in which
the innate programme flexibility has impacted beneficially on the activities of the
projects:
1. it has created space to ensure that the projects are able to remain at the forefront of
technological advances;
2. it has allowed the projects to adapt to changing market needs;
3. it has created the preconditions to allow the projects to become a platform for
innovation.
7.3.6 Achievements
A significant number of the projects report that Leonardo allowed development of
new competencies and expertise that would not have been achieved otherwise. The
projects provide testimony to advances in vocational and educational training which
they attribute directly to their involvement with Leonardo. This is an important
finding because it enables us to conclude that in spite of the serious operational
difficulties experienced by Leonardo and in spite of delays in payments, the projects
gained in terms of:
1. partnership capacity enhancement and organisational learning, and
2. significant advances and innovation have been made in terms of the particpants’
own working areas.
In the words of the projects:
Final Evaluation of Leonardo da Vinci
84
Deloitte & Touche
•
“Leonardo added a new dimension to our work that otherwise would have taken
years to develop”(Irish project)
•
“we are grateful to the fact that the Leonardo gave us the chance to create an
initial vocational training specialisation in Greece which can also be offered at a
distance”(Greek project)
•
“participation in the Leonardo programme has given us a strong impulse to
develop the important part of our educational activity and to strengthen its
European dimension; we began some research projects connected with our tasks
within Leonardo, work on the Leonardo project have interested our staff” (Polish
project)
•
“Leonardo has been a huge support allowing us to achieve significant advances in
our work integrating underprivileged young people into the labour market”
(Spanish project),
7.4
Impact of the Mobility Projects
The mobility projects record many similar experiences to the pilot projects. Impact
has not therefore been dissimilar for this group of projects. The mobility projects have
similar goals to the pilot projects but, not surprisingly, an additional, important, goal
also entails the aim of setting students and trainees on track to start their professional
careers and working lives. In addition, the mobility projects also aim to enhance the
employability of students. Other goals are broadly shared with the pilot projects as
they were set out in 7.4.1 above.
As with the pilot projects, the trans-national dimension of Leonardo was also highly
prized by the mobility projects. Value was placed on the international side of
Leonardo as it facilitated the broadening of opportunities, learning and employability
of the students and placement participants.
•
“The programme responds to a current need for students of demanding
professional training. The companies who host the trainees can make contacts
with other regions and other European high schools” (German project)
As with the pilot projects, the satisfaction of the mobility projects was also somewhat
lessened by the difficulties associated with delays and operational difficulties of
Leonardo. Projects note that:
•
“The programme should be absolutely managed with more flexibility, for the
interest of the beneficiaries” (Austrian Project)
•
“the delay in payments issue is of extreme concern to us” (Irish project)
•
“The problems with the technical assistance management caused a lot of
problems with the management of the project The problems even exist today”
(Dutch project)
Final Evaluation of Leonardo da Vinci
85
Deloitte & Touche
The notion of quality control is an important one for the mobility projects. This is
founded on projects conferring status and trust to the regulatory bodies (Commission,
Universities, other bodies). It is therefore important that damaging administrative
delays are not allowed to damage and tarnish the position of the Commission in this
respect.
“Moreover , to act under the shelter of a European programme is a guarantee for
sending and hosting organisations , and an added value for all participants” (Italian
project)
In line with the experience of the pilot projects, the mobility projects also claim to
have been innovative, to have appreciated partnership but found it difficult to manage,
and to have been highly motivated by the Leonardo grant money. They credit
Leonardo with facilitating actions and developments that would not have been
possible without the support of the programme.
•
“We were able to offer the trainees funding for their practical placement periods,
which would not have been possible without Leonardo” (German project)
•
“As a direct result of the Leonardo activities a degree of knowledge of the labour
market and effective links with European companies have been formed”
(Portuguese project)
•
“It enables all our students to gain international placements in sectors of industry
and commerce which would not otherwise have been open to them. It allowed
them to be far more mobile physically and intellectually than before” (Irish
project)
The above reflects general project – level satisfaction with the mobility projects,
which allowed tens of thousands of people to have a transnational training experience.
But there is little evidence of any framework to collect the lessons of these projects,
and channel them into a policy context. Why is mobility favoured? Is it merely to
give a broader perspective to the participating individuals? Where are the policy
analyses that should flow from this great flood of projects?
7.5
Impact for the Participants
As we outlined in Chapter 2, we consulted as widely as possible in the data collection
phase of the research. We therefore sent out a questionnaire to placement and
exchange participants. Their responses can be grouped around 3 key benefits:
•
the trans-national dimension was greatly appreciated and provide the basis for
enhanced learning, cultural awareness and reflection about Europe;
•
Leonardo provided the basis for enhanced maturity, independence and self
confidence;
•
Leonardo is perceived to enhance the employability of participants;
Final Evaluation of Leonardo da Vinci
86
Deloitte & Touche
As with the pilot projects, the trans-national dimension of Leonardo was very
important for the exchange and placement participants. They report that the scheme
has:
•
broadened their horizons – quite often they note that they would like further new
work experiences in other countries;
•
enhanced their cultural awareness – under Leonardo they have developed better
understanding of different European countries;
•
expanded their opportunities for communication with other professional people
from abroad;
•
enhanced their understanding of the supra-national dimension of the EU; and
•
enhanced their reflection about Europe and its diversity.
The participants observe, for example, that:
•
“after this experience I will try to communicate to others this sense of Community
unity which few people feel” (Spanish man, to Italy)
•
"a placement in a foreign country makes us more mature and responsible.
Moreover, I got the opportunity to discover a new culture" (French woman, to
Spain)
•
"Each time I had the possibility of exchanging knowledge with colleagues from
different European countries, the possibility for enhanced technological
understanding was also increased" (French man, to Spain)
•
"Exchange between countries is obligatory to achieve a better understanding of
the remaining countries" ( Spanish man, to France)
The Leonardo programme has undoubtedly impacted significantly on the lives of the
participants. The benefits are clearly age related. Young adults report that they have
gained in employability, maturity and self-confidence whereas more mature
participants tend to record enhancement in work related skills.
There were, however, some responses critical of aspects of the placement and
exchange projects, particularly that that quality of the experience was adversely
affected by the lack of preparation (knowledge and awareness of the host country) of
participants and the lack of systematic monitoring by the organisers. Amongst the
concerns of participants themselves were:
•
a mismatch between work and learning needs (problem of time available for
mentoring), more training wanted.
•
a lack of choice about placement for participant/ more appropriate placement/
more information in advance of starting;
•
an imbalance of too much work, not enough learning;
•
the grant was too small; and
•
the placement was too short to fit in all the learning;
Final Evaluation of Leonardo da Vinci
87
Deloitte & Touche
Nonetheless, on this level we must regard this strand of Leonardo as a success. The
vast majority of the participants record that they will recommend Leonardo to their
friends. It is important to emphasise that the role of the European Commission and of
the institutes / universities in Leonardo are regarded as quality control mechanisms for
the programmes by the participants
"placements under an European programme offer guarantees and quality. It
reassures students that every detail is under control, backed up by university, and the
European Commission"
7.6
Impact in Central and Eastern Europe
While it is too early to estimate the actual impact of the CEE countries’ involvement
in the Leonardo programme it is possible to discern the following:
Bulgaria
•
a very direct impact on vocational education and training policy and the
harmonisation of training and employment policies in a general framework
strategy;
•
a change in the general culture how to work; a change in the style and mentality in
vocational education and training (in the former system people went to a
vocational education and training school, have got a training provision but they
were not aware of their opportunities how to use the knowledge and skills they
acquired, and efficiency and efficacy was not a goal;
•
people learned in vocational education and training that evaluation and
dissemination of results must begin at the very beginning of a project not at the
end.
Czech Republic
•
a new awareness and enthusiasm in the secondary school/vocational training
sector for the benefits of mobility/exchange activity previously confined to the
university sector;
•
a major impact on individuals: there are currently 100 or so mobility projects in
the Czech programme;
•
an influence, at the very least inferred, upon the development of the Czech
vocational training policies and practice;
•
an opening of the Czech vocational training field to EU ideas.
Hungary
•
The most important impact for the country is that the Hungarian vocational
education and training now is present on the European “education market”, the
Hungarian promoters and the outputs of the joint projects as products and services
are present on international fairs, conferences and symposia.
•
Another important factor was the acquisition of know-how of the new ways of
project management in vocational education and training institutions (higher
education institutions went through the same within TEMPUS and later World
Bank projects). These institutions had got some of these experiences during the
Final Evaluation of Leonardo da Vinci
88
Deloitte & Touche
PHARE vocational education and training and the World Bank projects, but in
those cases there was always an external partner responsible for the management
of the projects (donor/recipient relations in assistance programmes), but now
projects have management responsibility themselves.
•
Despite the classical “academic” orientation of the actors in vocational education
and training institutions, a relatively large number of persons tried to run a project
as promoter. Out of 150 applicants 43 projects were approved with Hungarian
promoters, and another 247 institutions work as partners in projects with a
promoter from another European country.
•
A further important impact what one could call a kind of having a “Leonardo
consciousness” among the applicants or projects’ participants. Having the label
“participated in Leonardo da Vinci 1” is a source of pride for most of the
vocational education and training institutions.
Poland
The majority of pilot project promoters believe that their projects will have the
strongest impact in the following areas:
•
development of curricula designed to facilitate young people’s adaptation to
changing conditions of the labour market;
•
development of co-operation between SMEs on the one hand, and vocational
education and training institutions on the other;
•
growing involvement of social partners in vocational education.
Romania
Even though at national level the priorities for the country’s economic and social
development has been the development of continuous vocational training and regional
development, most of the projects dealt with the acquisition of new professional skills
(mainly within the frame of mobility and placements strands), development of new
curricula, of social partnership and build institutional capacities. According to the
NCU director this is explained by the fact, that there is no relevant legal regulations
still, and also that the newly established institutional structures serve to solve this
problem, and therefore the maybe promoters are concentrating on the national project
more for the moment.
Concerning the projects they focus considerably on initial training, as this was
regarded as having a “solid base” at operational level, due an effect of the PHARE
vocational education and training projects, which started before Leonardo da Vinci .
The Leonardo programme served as a catalyst for change in all the CEE countries,
facing the huge problem of transforming their economic and political systems in a
much shorter period than the present EU Member States had. The challenges of
globalisation and the rapid changes in technology and knowledge require a raising of
the skills of the labour force in order to create a learning society.
All these countries had a long and well-established vocational education, mainly on
the basis of the European continental traditions, which was broken up with the
introduction of centrally planned economic and one party based political systems
Final Evaluation of Leonardo da Vinci
89
Deloitte & Touche
Within that system closed links were developed between vocational education and
training schools and economic actors (state owned industrial and agricultural
enterprises), but one must not hide the fact that this system was rather efficient and
serving well the needs of that economic system. When the systems collapsed, the
whole education and training sectors were influenced. Neither the school system, nor
the employment policy within the planned economic system were able to respond to
the challenges.
The Leonardo programme could not help to foster change on a large scale, but could
catalyse the processes, and via the co-operation and working together with different
partners helped in setting up a “treasury of knowledge” about each other’s vocational
education and training systems, and shared common values about working methods
and different qualities of life.
Now the main concern of the CEE countries in their development strategies lies in the
development of human resources, or linking education policy with labour market and
employment policy. But due to the very rapid changes in the labour market and the
traditional and somehow inherent slowness of reform in education, the task is not a
simple one. From this point of view, even the rather late participation of the CEE
countries in Leonardo da Vinci , as a “laboratory for innovation”, provided an
excellent opportunity for complement their overall system level reform processes.
Leonardo da Vinci ’s priorities were designed to meet the needs of the EU Member
States, but one can find matching priorities in the CEE countries.
From the interviews carried out in the CEE countries it is clear that there is both
institutional and operational capacity to take part in the Leonardo Da Vinci
programme. However, there has been and there are still problems in the financial
capacity of CEE countries. In most of the countries, at the policy-makers level, the
financial contribution to the programme is considered as an intangible investment. As
an indicator of trust one must pay for it, because the outputs and outcomes bring
added value worth more than the cost. This is especially true in the light of the belief
of all the countries concerned possibly this co-operation helps them to take one more
step towards accession to the European Community.
Considering the “EU policy” elements of Leonardo, in the CEE countries at policy
makers’ level all the responsible people are very well informed and aware about
those. However, there is a lack of information at school and social partners level, “at
policy level there are very nice ideas, but at implementation level not so much” as was
said in one of the Ministries. But if we look at the transnational character of Leonardo
da Vinci and that it is considered as a laboratory for innovation these two elements
may have a rather good impact even at the implementation level, due to the simple
fact, that working together with others and exchanging ideas and confronting
problems in a different system can develop the capacity of the partners to see their
system from an outsiders’ point of view, and by this try out new approaches and may
be change or modify his own system. This way even at the implementation level, at
schools and at social partners the program had a real impact (or better to say will have
an impact, because most of the projects are not yet finished, but there are some
indicators that this will happen).
7.7
Impact in respect of Equal Opportunities
Since the time of the launch of the Leonardo 1 programme in 1995 the Commission
has developed its stance on equality of opportunity as a horizontal aspect of funded
Final Evaluation of Leonardo da Vinci
90
Deloitte & Touche
programmes, as indicated in the earlier chapter of this evaluation report concerning
the policy context. However, in Leonardo 1 the issue was simply addressed in two
ways:
•
a question to promoters on their plans for ensuring equal opportunities within their
projects;
•
the funding of fieldwork projects on the equal opportunities theme.
On the first, there is no record of monitoring procedures or follow-up of this question.
Hence there is no data available for this evaluation.
On the second, the Commission has published a Leonardo da Vinci Compendium of
Projects 1995-1996-1997 on Equal Opportunities for Men and Women in Vocational
Training, detailing 118 projects (1.57 per cent of the total number of projects), 49 of
which were procedure 1 and 69 procedure 2 selections. The Compendium states:
" ..most are pilot projects intended mainly to yield training tools for women in growth
areas and cross-sectoral skills. There is also a significant number of survey and
analysis projects which usually focus on investigating training needs at regional and
sectoral level, with a view to adjusting training and vocational guidance for women,
and examining the career obstacles for women within companies."
The Compendium notes the unevenness in the geographical distribution of equal
opportunities projects, 38 percent of which were accounted for together by the UK (26
projects) and Italy (19 projects). These results are attributed to certain national
characteristics and
"also reflect the different national sensibilities and meet the priorities established by
the Member States for each annual call for proposals."
On January 16-17 1998, the Finnish Leonardo Centre organised a transnational
thematic seminar on "Equal Opportunities in Vocational Training" in Tampere, in
conjunction with the Commission and the Dutch and UK Leonardo NCUs. A
significant part of the content of the event was a presentation of the projects as in the
Compendium above.
7.8
Impact of specialist strands
The programme featured a number of “specialist” strands which deserve brief analysis
in addition to the above. These were:
•
statistics actions;
•
surveys and analyses
The statistics action is largely welcomed. A key player in European training statistics
said “most of the data available now is there because of Leonardo’s contribution”.
For a modest financial outlay, the results are impressive and useful. Already there is
evidence of results through publications such as the CEDEFOP 1997 publication on
statistics, the forthcoming publication on Initial Training, and CEDEFOP’s input to
policy-making through various papers which now benefit from a better statistical base
than before. These publications top CEDEFOP’s sales lists, and the collaboration with
EUROSTAT has proved very fruitful.
Final Evaluation of Leonardo da Vinci
91
Deloitte & Touche
The surveys and analyses strand, on the other hand, has not been as successful as
might have been hoped. There was little or no linkage between this strand and the
actual projects in the field, thereby minimising the opportunity to test some of the
research theme results in live projects. The strand may also have suffered from
confusion regarding the relative roles to be played by the EU’s Research &
Development programmes and this activity within Leonardo. This area of Leonardo
could have built on some of the useful track record of programmes like Eurotecnet in
championing high-quality research on relevant themes, and bringing experts together
to achieve exchange objectives. Once again, the Commission’s resource constraints
meant that the results have not (yet) been disseminated or used for further work. The
Leonardo 2 Programme has effectively suppressed this strand, perhaps a sign that the
R&D programmes will now be the prime place for this kind of work.
7.9
Impact with respect to the Social Partners
The involvement of the social partners as project promoters or major contributors
within the Leonardo programme took place, with the support and encouragement of
the Commission, within the framework of the European Social Dialogue on Training.
Since the 1985 Delors move to revitalise the European social dialogue, vocational
training has been expressed as a priority by the social partners at both
interprofessional and sectoral levels.
For its part, the Commission has supported, for example, the independent dialogue
between the social partners on education and training set up by the Social Dialogue
Committee and also the involvement of the social partners in sectoral survey activity
in both the FORCE programme (1990-1994) and the Leonardo programme.
In terms of practical involvement of the social partners in the Leonardo programme,
the available data show that of the 2,272 projects resulting from the 1995,1996 and
1997 calls for proposals, a total of 413 (18 per cent) were identified broadly as social
partner projects in the light of three criteria:
1. Projects directly submitted by social partners, in which the contracting/coordinating organisation was a trade union, employer, or joint organisation. Under
this criterion 164 (7 per cent of the total number) projects were identified.
2. Projects submitted by equivalent organisations, such as trade union research
centres or training bodies.
3. Projects promoted by other organisations in which social partners play a key part
as parties to the social dialogue and/or disseminators or multipliers of project
results and/or where the social partners are the target group.
Of the 164 ‘strictly’ social partner projects the majority was contracted to trade union
organisations, followed by employers’ associations with joint bodies in third place.
Most of the social partner projects were pilot projects submitted under strand II,
covering “support for the improvement of vocational training measures … concerning
undertakings and workers”. The projects undertaken reflected the following priorities
of the social partners themselves, developed within the support framework of social
dialogue on training between 1994 and 1996:
•
Planning of in-company training
•
Training in SMEs
Final Evaluation of Leonardo da Vinci
92
Deloitte & Touche
•
Development of work organisation and training
•
Specific occupations
•
Integration of young people
•
New forms of regional co-operation
•
Validation/recognition of skills/qualifications
•
Training materials and methods
•
Allocation of training resources
•
New responses to individual needs
to which four further themes were added after 1996:
•
Lifelong learning
•
Vocational guidance for young people and adults
•
Qualifications, including the anticipation and analysis of needs, validation and
certification
•
Resources, including financing, joint responsibility and joint investment
In the estimation of the Commission, the body of the 413 social partner projects in the
Leonardo programme reflected all the above priorities and concerns, demonstrating
the value of the Leonardo da Vinci programme for the social partners.
7.9.1 Issues
It is clear that the purposes of the Leonardo da Vinci programme were consonant with
the principles of the European social dialogue framework and the involvement of the
social partners has been an important element in the overall quality of the programme.
However, while the range and variety (if not the number) of social partner projects in
the Leonardo programme may have satisfied the Commission of the value of the
programme to the social partners. ETUC suggested, inter alia, that
a) the overall trade union involvement in Leonardo da Vinci had been ‘unsatisfying
and unsatisfactory’
b) Bureaucratic procedures deter both trade unions and management side of
enterprises from participation
c) Actors at the enterprise level are unused to procedures such as are involved in
applying for projects: this provides hurdles to participation
d) SMEs have difficulty, for example, because of limited resources and problems
with worker release, in integrating vocational training within work organisation.
7.10 Impact with respect to sectors
It is very difficult to make any substantial comment on the impact on various sectors
by Leonardo da Vinci since the necessary data is lacking. For the centrally funded
projects we can break down the projects by sector as shown below:
Final Evaluation of Leonardo da Vinci
93
Final Evaluation of Leonardo da Vinci
Other Community, Socail and Personal
Service Activities
Health and Social Work
Education,
Public Administration and Defence
Reasl Estate, Renting and Business Activities
Financial Intermediation
Transport Storage and Communication
Hotels and Restaurants
Wholesale and Retail Trade
Construction
Electricity, Gas and Water Supply
Manufacturing
Mining and quarrying
Fishing
Agriculture Hunting and Forestry
Extra-territorial organisations and bodies
500
450
400
350
300
250
200
150
100
50
0
Extra-territorial organisations and
bodies
Figure 7.2: Breakdown by sector (not including training)
Private Househlds with Employed Persons
If a similar exercise is carried out excluding the training sector, the results show a
distribution as follows:
Private Househlds with Employed
Persons
Other Community, Socail and
Personal Service Activities
Health and Social Work
Public Administration and
Defence
Reasl Estate, Renting and
Business Activities
Financial Intermediation
Transport Storage and
Communication
Hotels and Restaurants
Wholesale and Retail Trade
Construction
Electricity, Gas and Water Supply
Manufacturing
Mining and quarrying
Fishing
Agriculture Hunting and Forestry
Deloitte & Touche
While this serves to give some indications, the dominance of training organisations
leaves a large number of projects where we have no information on the sector
addressed.
Figure 7.1: Number of Projects by Nace code
1400
1200
1000
800
600
400
200
0
94
Deloitte & Touche
While this is a far from perfect measure, it shows that there is a strong domination by
the service sector, including IT. There is also a large involvement in activities in the
area of community, social and personal services.
It would be useful to analyse this with more complete data – which should be possible
under Leonardo da Vinci 2.
Final Evaluation of Leonardo da Vinci
95
Deloitte & Touche
8 Conclusions and Recommendations
In this chapter we set out our conclusions and recommendations. For ease of reference
this is done sequentially, with the main findings presented in subsections relating to
the individual chapters. Recommendations are also linked to these findings. In the
paragraphs which follow we begin by revisiting the objectives and the different
actions in Leonardo da Vinci before discussing the key findings and recommendations
arising from the main evaluative chapters concerned with the themes of
complementarity, implementation, outputs and impact. The chapter concludes with an
overview of the key evaluation themes that have guided and shaped the evaluation:
outputs and impacts, effectiveness, efficiency, complementarity and synergy. Generic
programme - level recommendations are also provided.
8.1
The Programme’s objectives revisited
It is generally accepted amongst the stakeholders that the 19 different objectives with
which Leonardo da Vinci 1 began were little more than a cumulative listing of
aspirations and principles inherited from the group of programmes which preceded
Leonardo da Vinci 1 . There is some suggestion that, had the ratification of the
Maastricht Treaty not been delayed, the Commission might have had the time, and
opportunity, to reduce and refine the range of objectives of the Leonardo da Vinci 1
programme to something more coherent. The adoption of the Council decision
probably more reflected the Member States’ wish not to lose any of the elements of
the previous generation of programmes, rather than a well-planned and thoughtful
approach to a complementary European-level contribution to policy-making at a time
of change.
The external interim evaluation of Leonardo suggested that the multiplicity of
objectives would make any final evaluation impossible. We agree in part. The fact is
that because the 19 'objectives' statements were not clear objectives, targeted and
measurable in terms of performance, it is not feasible to attempt to provide any
assessment of their achievement in quantitative terms.
In the light of that, this evaluation has concentrated upon what, as far as we can learn,
was actually achieved by the Leonardo expenditure, with what effect and to what end.
In looking towards the future, we anticipate that the operation and evaluation of
Leonardo da Vinci 2 is likely to benefit from the reduction of programme objectives
from 19 to 3, enabling much clearer performance review.
Recommendations: Programme Objectives
•
Programme objectives must be patently measurable and achievable; should be
distinguished from aspirational aims and should conform to the SMART criteria
described later in this chapter.
•
There should be a clear distinction between process type objectives, e.g.
concerned with the incremental development of an EU level vocational training
policy, and task-type objectives, e.g. concerned with numbers of training places
created for excluded groups.
Final Evaluation of Leonardo da Vinci
96
Deloitte & Touche
8.2
Types of action
During Leonardo different types of action took place:
•
Policy development actions
•
Pilot projects or demonstration actions
•
Mobility actions
•
Publications, Databases and other Information actions
Of course, it is not always possible to completely distinguish between the four types
of action in practice. This reflects the sophisticated nature of many of the
interventions in Leonardo. Actions with multiple dimensions make typologies of
interventions difficult to apply rigorously. In many cases, for example, pilot actions
are an integral part of the policy process where they can be used to test policy
responses to problems, which have been identified through policy analysis and
research. Some publications are linked to other actions as well. Nevertheless, the
distinctions are generally useful, in our opinion, for identifying the different types of
activity undertaken under the programme.
8.2.1 Policy Development
Under “policy development” we include all measures from research actions through
conferences and seminars and the production of statistics, research reports and
analyses of specific areas. It also includes the analyses of outputs from other pilot and
project-type actions.
Given the economic policy context, it is clear that the EU dimension has an important
role to play in the elaboration of training policy for Member States. The European
dimension to policy research is of enormous value, because it allows for exchanges of
experience between Member States and analysis on a European scale. “Europe” can
compare and contrast the very different experiences encountered in a range of
locations and then help to analyse which solutions have proved effective. Such
analysis (and the results of the research itself) must however be disseminated widely
and discussed with the Member States in order to maximise its value. This is because
the appropriate policy response will in all cases have to be delivered at national or
regional level.
Recommendations: Policy Development
•
Analysis and comparison of best practice across Europe should be undertaken
•
This must be disseminated and discussed with the Member States to facilitate
national and regional level take up of best practices.
8.2.2 Pilot Projects
As “pilot projects” we include all test or demonstration actions funded by the
programme. The direct quantitative impact of pilot actions is typically localised and
Final Evaluation of Leonardo da Vinci
97
Deloitte & Touche
limited in relation to the potential number of beneficiaries in the EU as a whole. The
main value of the actions lies in the potential to influence a policy debate and to be
replicated by other actors, or by the Commission itself, if they are successful. They
are usually used to test different policy responses to identified problems and are often
called “demonstration projects” because of their ability to demonstrate suitable
courses of policy action for the future.
There is a strong case that pilot actions should only be used to test potential policy
responses (rather than merely funding “interesting” ideas on a demand-driven basis),
and only those which are not already being tested elsewhere. This means that they
should only be launched after an exhaustive analysis of the relevant policy options.
Our own view is that pilot actions must indeed be linked to pioneering innovative
policy development and not just launched in the absence of this strategic focus.
Furthermore, the results of the pilot action must be examined in great depth and
communicated to all relevant policy making bodies and interested parties. Finally, the
objectives of the pilot actions must be clear and if possible, performance indicators
should be designed to measure the success or otherwise of the pilot, before it is even
launched. Leonardo has not (yet) demonstrated that it has followed these principles.
Recommendations: pilot projects
•
Pilot actions should only be used to test potential policy responses not being tested
elsewhere. Exhaustive analysis of relevant policy options is therefore first
required.
•
Results of pilot actions must be disseminated.
•
Performance indicators should be established for the pilot projects from their
inception.
8.2.3 Mobility actions
Mobility actions are those which involve actors in a project that forms part of the
general delivery of existing programmes, rather than being a special action with a
hoped-for larger effect. There is no fundamental problem with the Commission being
involved in such actions, as long as there is a clear identification of a need at
European level and a clear strategy as to objectives, quality assurance and use of
results. Obviously, EU actions should be integrated as far as possible into other
mechanisms in order to gain greater visibility and greater efficiency and to avoid
duplication and wastage of resources.
Such actions must, in our view, be mounted on a scale which is sufficient to make a
real impact. This is necessary to achieve economies of scale and administrative
efficiencies. In order to achieve the necessary volume of activity and market
penetration, these actions should also, in our opinion, be well marketed and resourced
and planned on a multi-annual basis for sustained impact. The mobility actions under
Leonardo, while they succeeded in “moving” a large number of people, were
generally rather isolated in the sense that they occurred without close monitoring or
capitalisation of their results.
Final Evaluation of Leonardo da Vinci
98
Deloitte & Touche
Recommendations: Mobility Actions
•
Analysis needs to be carried out of the desired outcomes for Leonardo’s mobility
actions
•
Analysis of what works is also needed. This requires the establishment of
monitoring systems and procedures that focus on desired outcomes and best
practice. This will enable the mobility interventions to be focused on innovative
areas and avoid the risk of duplication of effort with other EU and national
programmes.
8.2.4 Publications, Databases and other Information Actions
Most of the publications and other information actions produced by the programme
are directly related to its dissemination strategy. This type of action should be very
focused on specific policy purposes, if they are to have real value. The studies,
surveys and analyses work of Leonardo was mixed in terms of its real contribution in
this sense. This comment ignores the obvious lack of widespread dissemination of
results of the Leonardo projects.
Recommendations: Publications, databases, other information actions
•
8.3
Ensure actions are focused on specific policy purposes, aimed at specific target
audiences and adequately disseminated
Complementarity
At a conceptual level the Leonardo programme was complementary to other EU
programmes as we saw in Chapter 4, but almost as a result of accident rather than
careful planning.
However, we have noted that:
•
The other training-related programmes funded by the EU budget (principally ESF
and R&D) had budgets greatly superior to that of Leonardo. They inevitably
occupied Member State authorities (and national training bodies) more than
Leonardo did.
•
Socrates and Leonardo were not co-ordinated at official level, either in Brussels or
in the Member States. Given the high incidence of participation in both
programmes by formal education institutions, some complementarity was secured
at actor level. But this was not widespread.
Final Evaluation of Leonardo da Vinci
99
Deloitte & Touche
•
The incidence of effective knowledge sharing through inter-departmental contact
within the Commission remains low. This is a regrettable handicap on the
potential for the large number of EU-level training-related initiatives to combine
and yield added value for the benefit of all.
The different EU programmes tend to operate in parallel rather than in an integrated
fashion. Providing the opportunities for the exchange of best practice and lessons
between programmes and departments would entail the provision of resources and
installation of mechanisms for the flow of information between programmes. This
implies the necessity for programmes to have a degree of porosity in boundaries
which would enable the communication channels to work effectively. We make a
number of recommendations below, which would assist the establishment of
mechanisms for effective knowledge management (policy think tank, profile raising
activities, workshops, dissemination functions, use of Internet and CEDEFOP). Such
devices would assist the Commission to maximise the benefits of having
complementary programmes by ensuring integration and a more holistic approach to
vocational education and training.
We have noted that continuity between programmes in fact represents a policy
opportunity that, if accurately harnessed, can yield valuable conduits for the
dissemination and spillover of best practice and both strategic and operational lessons.
The establishment and utilisation of information channels between programmes and
related policies is essential for optimising knowledge management strategies within
the overall vocational and educational training polity. Although Leonardo did not
necessarily achieve great synergy between ministries at national level, it did begin a
process which may yield fruit in the future.
The notion of seeing different European interventions as a continuum, playing really
complementary roles in a policy process is a key conclusion in this context.
Recommendations: complementarity
•
High-level political commitment to strategic integration of the whole range of
current EU training activities.
•
Commit to a policy continuum approach involving research, field testing, and
mainstream delivery, to replace current uncoordinated EU interventions.
•
Dissemination of best practices across other EU programmes should also be
undertaken by the establishment of an effective knowledge management strategy
8.4
Implementation
The implementation of Leonardo has arguably been the most problematic area of the
programme. As we saw in Chapter 5 the programme suffered from a number of
structural and design difficulties which impeded effective programme delivery.
Final Evaluation of Leonardo da Vinci
100
Deloitte & Touche
In the area of implementation we can cluster the main conclusions around two
separate phases of the policy process: programme design and programme delivery.
Programme delivery relates to the mechanics and structures of implementation. An
implementation experience is conditioned equally but the nature of programme design
(i.e. its implementability) and by programme delivery structures and procedures. Both
programme design and programme delivery therefore have a direct impact on the
implementation experience. Conclusions and recommendations arising from our
analysis of the implementation of Leonardo are therefore grouped around the related
themes of programme design and programme delivery.
8.4.1 Programme design
In the area of programme design, our conclusions are centred on four key points:
•
The programme lacked a user-focus. It therefore did not succeed in
accommodating or responding to the particular needs of the targeted participants
and groups;
In Chapter 5 we saw that target groups were adversely affected or even excluded
by the design structures. For example SMEs often lacked the necessary resources
to wait months for payment. Projects without pre-existing networks struggled to
create partnerships. Delays constitute bias against target groups. Complexity of
the administrative procedures was daunting and unmanageable for the target
groups. Documentation was also often late and incomprehensible. The complexity
of the system severely limited the involvement of some types of organisation. In
Chapter 7 we saw that lack of feedback to the projects did not help sustain user
morale or maintain a positive image for the Commission
•
The programme did not incorporate effective structures for the dissemination
and roll-out of project results or best practice strategies;
A major weakness of Leonardo da Vinci lies not in what it did but in the fact that
it does not know /cannot show what it did, and cannot learn from it.
•
The programme design did not take into account the capacity of the
Commission to implement the programme.
Both programme design and structures for programme administration need to be
designed with Commission capacity problems in mind to avoid bottlenecks.
•
The programme was not designed with adequate monitoring or evaluation
systems and therefore lacked any form of sophisticated evaluation capacity.
Recommendations from previous evaluations have not been implemented, despite
the increasing Commission attention to evaluation practice. Proper evaluation of
what has been done is important for the programme and proper monitoring and
centralisation of data storage is needed for this.
These four major conclusions in the area of programme design lead us to
recommend the following actions:
Final Evaluation of Leonardo da Vinci
101
Deloitte & Touche
Recommendations: programme design
•
A needs analysis should be undertaken to ensure that the programme is
accessible to the target group.
•
Simplification of all procedures is urgently required. Efficient payment
systems must be developed.
•
Administrative practices should be tested early in the programme, changed as
required in the light of experience
•
The actions for mobility projects need to be put into a context where they can
contribute to policy ideas.
•
Dissemination of project results and best practices is essential.
•
Ex-ante targeting needs to be increased.
•
Formative utilisation evaluations should be introduced.
•
Information systems should be defined from the start of the programme to
meet the potential evaluation and exploitation needs, as well as the
administrative processes.
•
A proper monitoring system which is focused on outputs and outcomes and
their use needs to be developed.
8.4.2 Programme delivery
In Chapter 5 we outlined a number of findings related to programme delivery. Our
conclusions are centred on the following key points:
•
The programme was successful in terms of provision of:
♦
an important tutelage function for projects
♦ an important information brokerage function for the projects
This was one of the most successful aspects of the implementation experience and
was vested in the NCUs. Continuity is necessary at the centre and in the NCUs to
ensure that the information base is not lost
•
Programmes and policies evolve and mature. During the life of a project goals and
objectives should be able to evolve and respond to changed circumstances. The
programme did not recognise this reality. (see Chapter 5). (The TAO’s tasks,
however, were refined over time reflecting the different stages of programme and
maturity of experience of Commission and TAO. This is a positive finding.)
There was insufficient flexibility for the programme structure to deal with
changing programme needs at different stages in its life cycle. This over-rigidity
was symptomatic of a lack of organisational learning, knowledge management,
Final Evaluation of Leonardo da Vinci
102
Deloitte & Touche
and structures which were overly internally focused on organisational matters
rather than adapting to changing user and market needs.
•
There was a tendency to over-regulate certain aspects of the programme. This
effectively operated to dampen down innovation opportunities and constrain the
implementation capacity of the programme. The adherence to contracting
procedures that were defined before full knowledge of the situation was possible,
and which did not recognise the reality of the programme participants, caused a
loss of programme responsiveness. The Commission tradition (imposed by the
Parliament) of taking resources away from a project at the end of its life,
regardless of the fact that activities may continue for at least a further 12 months
means that there is little chance of proper recognition or use of the outputs.
•
Decisions about the allocation of resources within the TAO did not always reflect
the existing administration requirements - insufficient resources were allocated to
project selection and proposal management for example. This led to a significant
backlog in payments, and contributed to slow approval procedures.
•
Inter-organisational boundaries between the TAO and the Commission and
between administering bodies and projects were not clear as they were not
adequately and clearly defined.
•
Vicious circles were set up - budget reductions interfered with the successful
completion of the project work programmes and led to lengthy negotiation
processes. Setting unrealistic project deadlines as a result of late starts following
extended contracting procedures can lead to project failure. The project selection
procedure was sub-optimal: project quality was sacrificed in favour of geographic
spread and an attempt to fund as many projects as possible. This applied to both
pilot and mobility projects following selection procedure II.
ƒ
The programme objectives were lost sight of and were substituted by an overemphasis on procedures and the administrative process. Too great a focus was
placed on procedures and not enough on the overarching goals (ends). The
administrative issues were prioritised and uppermost. The issue of quality in
mobility projects needs to be addressed in Leonardo da Vinci 2. This does not
imply that there was necessarily a problem with quality for these projects but that
the mechanism for quality control was absent.
•
The opportunity for strategic European planning was lost in the Leonardo
Committee, and there was no other effective policy channel for review of results
and capitalisation thereof. Some information is detailed in the Compendium, but
this is essentially just a list and it does not describe outputs of projects. This does
not provide an adequate base for strategic European planning in the vocational
educational and training system. The large size of the committee inevitably made
strategic direction difficult.
•
The financial arrangements for the TAO, whereby profit was effectively excluded,
were extremely questionable and risky. This was an important factor impacting on
the operability of the TAO.
•
Participants were effectively discouraged from seeking to participate in Leonardo
again. There was no real interest in their environment on the part of Brussels, and
the programme suffered from a very inappropriate management system and no
Final Evaluation of Leonardo da Vinci
103
Deloitte & Touche
strategy for the crisis when it came. This meant that the system was stacked
against the supposed target groups.
Conclusions related to the area of programme delivery are also detailed further in the
matrix below:
Recommendations: programme delivery
•
Build on the positive aspects, be willing to revise and modify goals. Recognise
the natural evolution of programme life-course.
•
Succession planning for the post-Leonardo continuation of projects should be an
integral part of project planning and approval.
•
A crisis management strategy should be in place.
•
The roles of the different bodies should ideally be defined through the setting up
of guidelines and agreed conventions.
•
A system for quality management and procedures needs to be developed.
•
In recognition of the partnership character of the projects, selection should not
necessarily be based on the nationality of the lead partner alone.
•
For Leonardo to be a true European programme the Member States must operate
to agreed European guidelines in the selection procedures.
• Incentive (profit) is necessary for limiting risk, and enhancing motivation and
quality in outsourced management activities. This should be considered when
awarding future management contracts.
8.5
Outputs and Impact
In this section we detail our conclusions and recommendations related to outputs and
impact. In the paragraphs that follow we draw on our discussions in Chapter 6 and 7
of this document. Outputs and impact are two aspects of the policy cycle that are
intrinsically related and so the two sets of findings are discussed together.
•
The policy framework and political consensus, surrounding the role of the EU in
training as an element of economic and social policy, had developed by the time
of Leonardo da Vinci 1 to the extent that it could have played a coherent and
integrated role in testing and sharing ideas and exposing people to other
approaches. The testing and sharing of ideas was not achieved due to lack of
capitalisation of project results, while the exposure to other approaches (which has
a much smaller penetration) occurred for those who participated directly.
•
Leonardo da Vinci 1 lost some of the specificity which the previous multiple
programme approach had secured, without securing greater impact or
comprehensiveness (because of its muddled design13 and failure to capitalise on
project results).
13
See our comments on programme design in section 8.4)
Final Evaluation of Leonardo da Vinci
104
Deloitte & Touche
•
The lack of insight into products is unacceptable. A working methodology for
capitalising on projects’ outputs is crucial and central to a programme such as
Leonardo.
•
The delays experienced during the programme had a negative effect on:
♦ project design (difficult to implement in the time left available),
♦ dissemination (the problems with the TAO consumed time allocated for
dissemination),
♦ ability of target groups to be included (projects had to be self-financing or start
late),
♦ innovation (projects lost their innovative edge because of delayed starts)
♦ participants’ willingness to participate in future Leonardo programmes.
•
However, project participants largely felt satisfied. They created informal
networks, initiated learning about partnership operation and learned about
participating in European programmes. Evidence of organisational learning was
identified in the areas of:
♦ work related skills,
♦ managing complex administrative procedures,
♦ inter-organisational relations (negotiation, bargaining, co-operation,
teamwork, networking skills).
Exchange and placement participants also reported learning in the areas of:
♦ work related skills,
♦ knowledge of other cultures,
♦ increased independence,
employability.
maturity,
self-confidence,
and
enhanced
•
The statistics strand was very useful: the resources provided allowed real progress
to be achieved.
•
The surveys and analyses strand suffered from isolation from the rest of the
programme.
•
Despite everything the programme embodied an innate flexibility allowing space
for the projects remain at the forefront of technological advances, to adapt to
changing market needs, and to allow the projects to become a platform for
innovation. This was more by luck than design however.
•
The transnationality of the programme contributed mostly to the Europeanisation
of civil society and the growth of European consciousness. More importantly for
the vocational and educational training system, a wide network of transnational
contacts was built up allowing the exchange of skills, cultures and learning.
•
In eastern and central Europe additional benefits accrued as the result of the
unique circumstances of these countries.
♦ There was increased harmonisation with the European vocational education
and training system,
Final Evaluation of Leonardo da Vinci
105
Deloitte & Touche
♦ Changes in working cultures and project management systems were
implemented
♦ The practice of partnership and inter-SME co-operation was encouraged by
Leonardo
♦ Projects recorded an increment in institutional capacity
•
Leonardo was responsible for initiation of a number of projects which could not
have come into existence without the Leonardo da Vinci programme.
As we have seen in Chapter 7, impact is nevertheless very difficult to measure
because of the absence of necessary data about products and achievements. Data
enabling assessment of impact needed to be collected at the outset of the programme.
The pilot projects often alleged that Leonardo had had a significant impact on the
national systems of the different Member States, but this is very difficult to establish
in the absence of this kind of data. What is clear is that impact has been conditioned
by the organisational dynamics and by the structural programme characteristics which
helped shape the implementation environment and subsequent attainable impact for
the programme.
We outline the main recommendations arising from this analysis below:
Recommendations: outputs and impact
•
data on project services and products should be collected and best practices
disseminated
•
in addition mechanisms for ensuring quality of the products should be put in
place.. Project results need to be capitalised upon and the programme design
should be transparent to users. A user focus for the programme design is needed.
•
A working methodology capitalising on the projects’ outputs would entail an
approach designed to ensure that important best practices were captured,
innovative trends pointed up and strategies for follow through put in place. This is
especially important for projects of significant potential and value.
•
Organisational learning, and participant learning was a positive impact of
Leonardo. Future programmes will need to capitalise on this positive outcome
through an effective dissemination and knowledge management strategy.
Workshops to share information, effective web sites, and the dissemination of and
reward for best practices must be seen as integral and critical for this process
•
Both output and impact strands have a key input to make into the data collection
and dissemination strategy for Leonardo
•
The eastern and central European experiences must be built on where they were
positive. This would entail the organisation of workshops and pro-active
dissemination of the learning experiences.
•
Appropriate data collection is of critical importance for the subsequent
measurement of impact. Baseline data needs to be collected and performance
measures established.
•
Member States and the Commission should establish a formal mechanism for
assessing results and processing them thereafter.
Final Evaluation of Leonardo da Vinci
106
Deloitte & Touche
8.6
A summary
In the matrix below we summarise the strengths, weaknesses, opportunities and
threats for the future which relate to the Leonardo programme we have evaluated.
Strengths
Weaknesses
Involvement of key players
Programme not user friendly or usercentred
Trans-national dimension
Open to a large number of themes and
issues
Delays
No use made of outputs
Innovative approaches to process
Focus on means not ends
Organisational learning by projects
Over regulation of TAO and
administrative aspects
Some believe Leonardo has influenced
national systems
European citizenship enhanced
Maturity, independence, skills and
employability of young participants
enhanced
Projects left to their devices. Although
this led to feelings of isolation it also de
facto created space for innovation at
project level
Many of the projects would not have
happened without Leonardo
Insufficient quality control mechanisms
for project outputs and products
Organisational roles not clearly defined
Insufficient recognition that goals evolve
and change through the lifecycle of a
project
No crisis plan
Strategic direction not achieved
Lack of quality controls
Dissemination strategy not in place
Contributed to continuing socialisation of
EU training actors
Opportunities for future
Threats for future
European complementarity could bring
great rewards if secured
Poor image accorded to the Commission
Forging of sustainable partnerships: keep
building the links between practitioners
Projects not willing to be involved with
Leonardo again
Capacity to manage the programme
threatened
Partnerships heavy to manage for projects
8.7
Overall
We set out below some recommendations arising from the conclusions of this
Chapter. In general terms we find that future Leonardo programmes would benefit
from a much greater user-focus. The projects and participants should be uppermost in
the focus of the programme design. We therefore advocate that a comprehensive
needs analysis is done during the phase of programme elaboration so that the
programme can respond directly to the needs of the target groups. This kind of needs
Final Evaluation of Leonardo da Vinci
107
Deloitte & Touche
analysis is best done by listening to the voices of the participants. We would propose
the employment of focus groups, questionnaires and user panels to aid in the design
and programme elaboration.
In the area of complementarity , output analysis and dissemination there is a need for
considerable change. CEDEFOP, the ETF, and new internet-based initiatives could
play a useful role in drawing all of the relevant material together (not just from
Leonardo 2).
We also propose the convening of high level workshops to disseminate key findings
and to rollout best practice to other related EU programmes. The complimentary links
between Leonardo and other programmes should be officially co-ordinated.
Complementarity entails recognition that EU actions in this domain area are
systematically related and need to be integrated at the European level and not just at
the micro level by individual participants and projects.
A high-level think tank, with the imprimatur of the Member States, could rise above
some of the management and allocation functions that the Leonardo da Vinci 1
Committee carried out, and set an integrated course for the whole range of training
activities that the EU is now engaged in. This would encompass the selection of major
themes for policy development, research, demonstration projects, event s and so on.
The continuum of programme styles could be harnessed to give real power to this
work and to allow each area of Community intervention to contribute in an integrated
way.
With regard to evaluation, we strongly urge the establishment of proper monitoring
and data storage systems. In addition- given the need to tailor the programme to the
needs of the target groups, and to ensure that evaluation findings are utilised, relevant
and helpful - we would advocate the establishment of formative evaluation structures.
This would entail the decentralisation of the evaluation process to more broadly
include the relevant stakeholders. The Expert Group would provide a good forum for
this as it comprises a fairly representative cross-section of stakeholders. Consideration
should be given to broadening the group to include other important stakeholders if it
were to take on this role.. Such evaluation approaches invest ownership of the
evaluation in the stakeholders and tend to induce capacity enhancement as well as ongoing programme improvements throughout the life cycle of the project. This
evaluation approach is centred on action, rather than audit of outcomes, and is a
proven mechanism for ensuring enhanced quality and better outcomes for
programmes.
8.8
Specific Objectives of the evaluation
In Sub-Section 2.2.2 we set out a number of specific objectives for this evaluation – to
evaluate the programme and provide advice and counselling to the European
Commission on five key issues:
•
Outputs and impacts of Leonardo
•
Effectiveness of the programme
•
Efficiency of the programme
•
Ways to improve efficiency and effectiveness in future programmes
Final Evaluation of Leonardo da Vinci
108
Deloitte & Touche
•
Ways to develop complementarity and synergy with other vocational education
and training instruments and policies.
8.8.1 Outputs and impacts
Leonardo undoubtedly impacted upon the lives of individual participants and was
instrumental in many instances in allowing the start up of projects which otherwise
could not have come into being. It undoubtedly contributed to the broad European
project helping to shape the emergence of European civil society and European
consciousness.
The outputs and impact of Leonardo were difficult to assess because of the absence of
a clear data collection strategy; gaps in information were evident in several areas
(products, equal opportunities etc). We have therefore stressed the need for evaluation
to be conceptualised as a process which starts with the elaboration of the programme
and is on going throughout the programme’s life cycle. This entails a more formative
approach to evaluation and an acceptance of programme flexibility and adaptation
during the course of its delivery,
We have also stressed the need for rigorous collection of appropriate information
which should be determined in line with the questions which future evaluations seek
to answer. For tracking impact, for example, it would be useful to establish baseline
indicators as well as performance measures. This should be carried out in conjunction
with qualitative evaluation designed to illuminate and shape processes
(implementation, design, partnerships, and experiential data, for example).
Appropriate dissemination and quality assessment of products is critical.
8.8.2 Effectiveness of the Programme
Effectiveness is closely related to impact. Judgements of effectiveness must, by
definition, be related to the goals of the programme. Questions of effectiveness
normally concern a programme’s effectiveness in the achievement of defined goals
and objectives.
We can observe that the programme had definite micro-level benefits and in this
respect was effective in achieving some of its objectives, most particularly in the area
of achievement of broadening the European citizenship and, to a degree, enhancing
employability.
The degree to which the Programme was effective in achievement of its other goals is
more difficult to determine for a number of reasons. Firstly, the lack of clarity,
precision, and measurability of the programme’s multiple goals is a real obstacle to
measurements of effectiveness. Goals should be SMART – Specific, Measurable,
Achievable, Realistic and Time-achievable. Progress with the rationalisation of
objectives for Leonardo has begun with the reduction of objectives from nineteen to
three. We have noted, however, that there is still room for greater clarity and
transparency in this area.
Assessments of the degree of innovation are also difficult because of a lack of
common definition for this term. Lack of clarity about the meaning of ‘innovation’ is
not undesirable, however, as the degree to which a project can be innovative is
heavily contextually determined. Some guidelines for the identification of what is
Final Evaluation of Leonardo da Vinci
109
Deloitte & Touche
innovative in specific contexts would however be useful if Leonardo is to be judged
successful in stimulating innovation in the future.
A key objective of Leonardo was to develop a European vocational and education
training policy. We have noted during the course of this evaluation that the absence of
data (baseline indicators in particular), the time frame and the multi-causal nature of
change in the vocational education and training system make it impossible to assess
the degree of impact of the programme in this area. Approximately 50% of the
projects we sampled felt that Leonardo had impacted on the national vocational and
education training system, as their own projects’ products had filled gaps in
vocational education and training provision. However, identification of the
effectiveness of Leonardo in the Europeanisation of vocational education and training
systems can only be achieved in conjunction with the prior adoption of a clear
strategic focus for the programme in this area. Leonardo was not able to capitalise on
this opportunity. Impact of Leonardo in this area is consequently not focused and
opportunities for synergy and strategic planning are therefore lost.
8.8.3 Efficiency of the Programme
There is little doubt that the efficiency of Leonardo has been seriously adversely
affected by the administrative difficulties which characterised the programme during
the period of this evaluation. We have recorded the adverse impact of this
circumstance on the programme. Long delays have been the most serious obstacle to
efficient programme operation, and in Chapter 5 we have discussed in some detail the
nature of these implementation difficulties. In summary, a combination of structural
features (administrative procedures) and organisational dynamics have jointly
established an adverse implementation environment for the programme. This has had
a significant impact on the programme’s efficiency which cannot be lightly ignored.
8.8.4 Ways to improve efficiency and effectiveness in future programmes
In the previous sections of this Chapter we have set out a number of recommendations
which will help improve both the effectiveness and efficiency of the programme. In
summary, we have advocated that reforms should be made in the design, evaluation,
delivery, and dissemination phases of the programme.
The adoption of a user–focus is central to the production of a user-friendly workable
programme and meaningful and useable evaluations. A high level think tank should
be established to guide the strategic direction of Leonardo within the overall European
training debate (which need not be limited to EU for a), and move towards a European
dimension for vocational and education training. During the delivery phase, care
must be taken to ensure that the objectives are not marginalised in favour of rigorous
application of administrative procedures; it is important to keep sight of the goals of
the programme at all times. Dissemination of the programme’s results must be seen as
critical and we have suggested that CEDEFOP, could play a useful role in this
function.
Adoption of the recommendations set out in this Chapter will help to ensure that the
intrinsic efficiency problems experienced are minimised in the future, and the
programme becomes more effective as it is better able to capitalise on its successes
and better able to achieve its goals in a more systematic and targeted fashion.
Final Evaluation of Leonardo da Vinci
110
Deloitte & Touche
8.8.5 Ways to develop complementarity and synergy with other Vocational
Education and Training instruments and policies
We have discussed the current situation of Leonardo in the context of other, related,
EU programmes in Chapter 4. Finding ways to avoid dead-weight is a widespread
problem across many European programmes. Ensuring that programmes are
complementary to each other is a more manageable and productive way of
approaching this issue. Inherent to this approach is the practice of knowledge
management and the notion of the learning organisation. Thus best practices and
important lessons from programmes should be available to other programmes and
policy-makers to optimise and build upon the learning patrimony of the European
Commission.
In the domain of vocational and educational training, this approach calls for Leonardo
to be situated within the broader context of other vocational education and training
programmes. We have alluded throughout the report for the need for programmes to
have porous boundaries. This means that best practices and valuable policy and
practice lessons are able to flow in and out of programmes to the mutual benefit of
related programmes, also operating in this domain.
Ensuring complementarity and synergy with other programmes necessitates the
creation of opportunities for inter-programme dialogue. We have suggested
workshops, web fora and so on, and effective dissemination strategies. We have
provided detailed recommendations for the achievement of these principles in Subsection 8.3 of this Chapter. The tracking of related initiatives and measures in the
vocational and educational training arena is also necessary.
Final Evaluation of Leonardo da Vinci
111