INSTITUTIONAL ARRANGEMENTS AND THE
INNOVATION PROCESS
Discussion paper
Prepared by Brent Wheeler
This version: June 2013
2
CONTENTS
INSTITUTIONAL ARRANGEMENTS AND THE INNOVATION PROCESS .............................................. 3
Central Place of Science and Technology in the Policy Mix ......................................................................... 3
Research and Development ......................................................................................................................... 3
The Innovation Process ................................................................................................................................ 4
Successful Innovation Processes are Adaptive......................................................................................... 4
Successful Innovative Processes Involve Trial and Error .......................................................................... 4
Successful Innovation Processes Incorporate Variation and Selection .................................................... 5
Successful Innovation Often Involves Unconventional Working Models ................................................ 5
Successful Innovation Often Involves Unconventional Individuals and Teams ....................................... 6
Why Innovation is Tough for Government Owners and Stakeholders ........................................................ 6
Trying New Things Fully Expecting Failure ............................................................................................... 6
Making Failure Survivable ........................................................................................................................ 7
Making Sure Failure is Identified and Fed Back........................................................................................ 7
Unconventional Staff and Working Modes .............................................................................................. 8
Summary .................................................................................................................................................. 8
Implications for Institutional Design ............................................................................................................ 8
Conclusions: A Difficult Policy Challenge ......................................................................................................... 9
3
INSTITUTIONAL ARRANGEMENTS AND THE INNOVATION PROCESS
1. This paper discusses the nature of innovation processes and the way various institutional
arrangements tend to either reinforce the odds of success as opposed to deliberately or, more
usually inadvertently, lower those odds through creating obstacles.
2. The origin of the paper lies in a series of projects undertaken by the Morley Wheeler Partnership
for both public and private sector clients focussed on commercialisation of “raw” ideas, the
“development” component of R&D activity, and the institutional context in which these processes
occur.
CENTRAL PLACE OF SCIENCE AND TECHNOLOGY IN THE POLICY MIX
3. Numerous western governments have recently placed much store on the ability of R&D to lift their
economies to new highs, restore moribund growth rates and solve significant fiscal problems. The
New Zealand government provides a typical example. The 2011 budget placed emphasis on the
role of science and innovation in the following terms:
A key platform of Government policy is to “lift the long run performance of the economy”.
A number of policy initiatives are being implemented in order to achieve this. Most
pertinently in respect of this report, “science” and “innovation” are specifically identified as
two of six drivers that are pivotal to achieving the long run outcome.
4. The 2013 Budget for NZ sees innovation grants averaging $141.5m for the next four years being
made available1 for such purposes.
5. Thus like many western economies New Zealand is looking to advances in science and technology
to provide the platform for maintenance of current lifestyles and growth in overall wellbeing.
Moreover, like other governments the N.Z. government has and is planning to continue to invest in
the area.
RESEARCH AND DEVELOPMENT
6. The activities loosely termed Research and Development (R&D) can be described as involving
research or like activity leading to innovation (research) and related activity which renders
innovations capable of delivering commercially viable functionality.
7. While the slant of each activity is distinctive (research considered separately from development),
the distinction is blurred, differs from project to project and situation to situation and defies ready
definition.
8. What is common to both activities and underlies each is innovation.
innovation, most R&D effort is likely to prove fruitless.
1
NZ Herald May 16 2013
Without successful
4
THE INNOVATION PROCESS
9. We argue that the key means for delivering on both public and private value propositions
concerning R&D activity is embedding the very nature of the innovative process in the institutional
arrangements which are advocated.
10. The innovation process may be characterised as follows:
SUCCESSFUL INNOVATION PROCESSES ARE ADAPTIVE
11. The fundamental driving process in innovation is that of adaption. Innovation processes proceed
through varying trial and error stages adapting as they go to produce, ultimately, successful
outcomes. In that sense we can therefore expect – and indeed do observe – numerous mistakes,
errors, omissions and failures.
12. Failure is a significant part of successful innovation. It is failure which allows success to shine
through and success is the result of adapting such that the outcome sought, is delivered.
13. Significantly then successful innovation involves what would normally be considered “an
uncomfortably high” number of failures. In this respect, innovation processes differ from many
other processes (for example assembly line production or distribution processes).
14. This point is critical because it differentiates “standard” activity (for example continuous
production runs and like processing activity) strongly from innovation. Failure in innovation is
actively sought so that success can be assured through adaptive discovery. In production
processes significant efforts are made to avoid failure because it signals unsuccessful outcomes.
SUCCESSFUL INNOVATIVE PROCESSES INVOLVE TRIAL AND ERROR
15. An inherent part of the adaptive nature of successful innovation is that the process itself is one of
trial and error. The primary reason that trial and error is the dominant process is the complexity of
many of the situations in which innovation is sought.
16. The trial and error process is not necessarily a great deal different to that found in the standard
scientific method in that it might be characterised as the proposing of hypotheses and their
subsequent testing for rejection or acceptance.
17. What is important is that the number of failures (rejections) vastly exceeds that of successes
(acceptances). The history of innovation along with research into that history shows that the trial
and error process is tremendously powerful and it is that process which is capable of producing
high value innovation.
18. Considered in terms of public policy (and specifically fiscal) Governments providing grants can
expect and should welcome the direct loss of the majority of the funding they commit to the area,
hoping that the gains will offset the loss. This is the stance they need to be comfortable with and
justify against other policy objectives, if they are to be credible in the light of what is known of the
innovation process.
5
SUCCESSFUL INNOVATION PROCESSES INCORPORATE VARIATION AND SELECTION
19. An inherent part of the trial and error process (and indeed the adaptation process) is that it
involves variation and selection. Variation involves the continuous initiation of different ways of
“doing things” time and again in each case selecting those which succeed while rejecting those
which fail.
20. The net result then is – in line with conventional evolutionary thinking – a process in which there
are numerous trials amongst numerous variations resulting in numerous errors and the
consecutive selection of successful outcomes as processes are evolved and adapted to producing
the outcome sought.
21. This approach is known as an “adaptive experimental” approach and may be seen – usually in
hindsight – to account for the vast number of successful innovations. The process may be
contrasted with much more “controlled” processes where more mechanistic and deliberate
frameworks are used in an attempt to devise new means of achieving goals and objectives.
22. Successful innovation over history has been shown to involve significant random, accidental
characteristics with innovations being uncovered through a wide variety of unusual, often
unexpected and at times unwitting discoveries.
23. Well known examples include the discovery of penicillin in a contaminated petri dish (accidental
discovery) and mathematician Cardono’s imaginary numbers – useless at the time of his musings
but essential in the development of radio centuries later2.
SUCCESSFUL INNOVATION OFTEN INVOLVES UNCONVENTIONAL WORKING MODELS
24. A significant number of successful innovations have emerged from entirely unconventional
working models of research and development. In several cases (for example Capecchi’s Nobel
winning work on mouse genes3) success has come in spite of conventional process such as
accepted guidelines or funding directives being ignored.
25. In many cases, rationally designed funding programmes are overly risk averse relative to the
processes that produce revolutionary breakthroughs. Programmes for example typically back
projects and prospects rather than people. Approaches which back people however (for example
the Howard Hughes Medical Institute4) have been shown to be highly successful.
26. Throughout history, prizes have frequently been used as means for motivating successful
innovation. Recent decades have seen a growth in the use of such forms of motivation5 with
various prize giving models being developed.
27. To promote successful innovation institutional arrangements need to have the flexibility to
contemplate and likely nourish alternative forms of working model and in particular processes
which step outside the norms of “the day.”
2
See Herford, T. Adapt: Why Success Always Starts with Failure, Little Brown, U.K., 2011.
See Nobel Price website www.nobelprize.org .
4
Gautam, N. 2009, 21st December, Wall St Journal.
5
See Wired 17, December 2009, Hanson, R. “Patterns of Patronage: Why Grants Won Over Prizes in Science”,
Working Paper Univ of California, Berkeley.
3
6
SUCCESSFUL INNOVATION OFTEN INVOLVES UNCONVENTIONAL INDIVIDUALS AND TEAMS
28. It is frequently the case that successful innovators do not fit the mould of conventional or standard
staff. The literature is replete with examples of individuals who have records of great success in
innovation but who would not be regarded as complying with conventional standards,
qualifications or experience.
29. The ability to deal with successive failure, generate continuous out of the ordinary variations,
develop entirely original approaches to problems and apply this set of unusual characteristics and
their associated behaviours successfully in the field of innovation is, almost by definition, “nonaverage” and at least somewhat idiosyncratic.
30. A number of highly successful innovation teams have been staffed with unconventional individuals
operating in unconventional ways. Lockheed’s “Skunk Works” engineering team is a well-known
example6.
31. Accordingly, institutional arrangements, while not necessarily needing to take active steps to seek
such individuals, must be designed to be flexible enough to accommodate and benefit from them.
WHY INNOVATION IS TOUGH FOR GOVERNMENT OWNERS AND STAKEHOLDERS
32. Innovation then is rather different to the deliberate and controlled actions commonly associated
with successful business and commerce. It involves serendipitous activity and discovery through
various random trial and error processes which address variation and selection.
33. In particular the need to embrace failure as an inherent part of success is important. This type of
approach is extremely difficult for individuals to internalise and work with. It is even more difficult
for corporate forms (such as large companies or trade associations) and almost impossible for
governments.
34. The incentives associated with collectives such as companies and the political nature of
governments and the arrangements they commonly employ simply militate against the
characteristics of successful innovation7.
35. In considering value propositions frequently associated with growth through R&D it is important to
understand why this might be.
TRYING NEW THINGS FULLY EXPECTING FAILURE
36. Governments and the agencies they spawn are not well adjusted or suited to failure. In fact quite
the opposite expectation prevails. Thus, any form of agency which is constantly exhibiting failure
and has a very high ratio of failures to successes is likely to be deemed unacceptable by
government and its agencies and doomed to eventual closure.
37. The reasons for this are, while not necessarily relevant in the present context, easy enough to
understand. They include the fact that taxpayers money is being used, that failure tends not to
6
Rich, B. and Janos, L Skunk Works Little Brown, U.K. 1994.
The issues as they affect corporates are virtually identical to those of the Jensen and Meckling thesis (Journal of
Financial Economics, October, 1976, V. 3, No. 4, pp. 305-360.).
7
7
attract “votes” and that popularity with the public at large is not broadly associated with the
strings of failures which successful innovation and innovation processes demand.
38. The conclusion is then that conventional institutional arrangements used by governments are
unlikely to be well adapted to the kinds of processes which produce successful innovation. In
designing institutional arrangements to deliver the value propositions associated with economic
recovery via R&D deliberate and significant design steps therefore need to be taken to avoid or at
least mitigate heavily this problem.
39. The institutional arrangements need to be designed such as to encourage and fully expect failure.
MAKING FAILURE SURVIVABLE
40. Given that failure is an endemic element of success in innovation, it is of great importance that
failures must be survivable. This means that it must be possible to have a large number of failures
and continue to try new things (expecting even more failure) until success is achieved. Again, such
processes are not readily embraced by governments and typical government arrangements.
41. The result of failure in government agencies is typically the closure of programmes, the denial of
funding, the exit of key staff, “auditing” by authorities and curtailment of activity in general. In
other words, precisely the reverse set of behaviours as might be expected to promote successful
innovation.
42. Moreover, the nature of government and popular successful government, is that “grandiose”
projects tend to be undertaken and such projects are the least easy to make survive. The prime
example in New Zealand would be the “Think Big” programme which involved very large projects
where failure made survivability difficult.
43. The grandiose tends to be favoured over the prosaic and mundane simply because popular acclaim
and high profile success does not normally accompany a series of relatively low profile and
apparently trivial failures. Low profile trivial failures are essential to innovative success however
precisely because they involve the many failures necessary for success and the numerous
variations accompanied by selection which lead to innovative breakthrough.
MAKING SURE FAILURE IS IDENTIFIED AND FED BACK
44. The third element critical to the success of innovative processes is the ability to have sharp and
immediate feedback on failure. Once again government arrangements are not well placed to deal
with feedback. Typically when failure occurs in government agencies and under government
arrangements feedback is slow, denial is common, ability to reverse out of unsuccessful ventures is
limited and the ability to send clear signals about failure (so that new initiatives can be started) is
equally limited.
45. Both factors that make failure anathema to conventional government and the grandiosity
associated with government in supply of projects also operate to make the generation of relevant
feedback difficult.
46. Institutional arrangements must be such that they are able to generate the type of rapid feedback
about failure which is capable of feeding the initiation of further projects (fully contemplating
further failure) readily.
8
UNCONVENTIONAL STAFF AND WORKING MODES
47. Typically, governments are not well suited to operating in ways which consistently depart from
convention, operate outside norms, rules, precedent, accepted practice and well established
methods of proceeding. This is not a matter of criticism. It is precisely these characteristics that
give governments comparative advantage in certain other areas (for the most part consistent
application of law and order for example.)
48. It is however to recognise that protocols, processes and operating models which characterise
governments are frequently the reverse of those which favour successful innovation. Certain of
these processes – for example the running of lotteries – do not necessarily sit well with other
objectives of government.
49. Similarly in respect of staff. Governments are not renowned for the employment of the kind of
mavericks so often associated with success in innovation. Equally, the types of contractual
relationship which might be required for dealing with such personnel are not necessarily those
commonly found in governments.
SUMMARY
50. The three characteristics required for successful innovation then are:
a. Trying “new things” fully expecting failure;
b. Making failure survivable; and,
c. Making sure failure is identified and fed back.
51. In addition the entities established within such arrangements need to be:
a. capable of deploying unconventional work processes and methods of achieving desired
outcomes; as well as being; and,
b. staffed by unconventional individuals and operating teams which may depart significantly
in make-up and work process from the norms applying at any given time.
52. It is precisely these arrangements which governments tend to be poor at establishing and
operating.
53. It should be noted that difficulties such as those faced by governments arise for other forms of
collective organisation including standard private sector companies and other collective forms.
54. Trade associations are a case in point. These various “associations” in various sectors typically face
the same difficulties as governments – they do not like failure, they tend towards grandiosity in
projects undertaken, they tend not to survive and endure strings of failure, and their response to
feedback often involves strenuous denial.
IMPLICATIONS FOR INSTITUTIONAL DESIGN
55. Given this analysis it is relatively simple to see why the business forms typical of the private start
up and venture capital sectors have to date enjoyed a great deal more success than attempts at
“government engineered” innovation.
9
56. In designing innovation units or divisions within governments (and like entities having the
characteristics of collectives), success requires that attention be paid to at least the following:
a. A stand alone, branded and autonomous innovation agency with distinct, recognisable
culture and business process offers more chance of success than extant government
departments or ministries.
i. Autonomy is required in order to sustain an institution run to principles which
differ significantly from those of its parent(s); and,
ii. Autonomy is also required to allow independence in developing operating
procedures, hiring staff and like matters.
b. Forms of accountability and monitoring regimes which support such accountability need to
be developed. These need to reflect the essentials of the innovation process, i.e.:
i. The generation of variations and selections and operation of trial and error
processes across a wide variety of areas within chosen markets; and,
ii. The generation of survivable failures, i.e. the scale and scope of projects
undertaken, need to be such that the expected failure rates and their costs in
dollar and other (for example time) terms can be sustained.
c. Establishing governance structures and making appointments which reflect the need for
autonomy and help sustain the distinctive innovation culture. This includes:
i. Having selection criteria which exclude appointees who have vested interests in
parent organisation principles as opposed to the principles of the innovation
enterprise; and,
ii. Having selection criteria which actively seek directors and managers who
understand, are sympathetic to and have a passion for the principles of innovation
rather than the principles appropriate to conventional production and distribution.
CONCLUSIONS: A DIFFICULT POLICY CHALLENGE
57. Innovation as a spur to growth is attractive to policy makers at present in part because of the
positive benefits it appears to offer and in part because of the persistent failure of many orthodox
interventions such as protectionism, infant industry subsidies and various regulatory fillips.
58. Expectations of growth through more frequent and intense innovation has sound conceptual
foundations and empirical support as explained in the finest of detail by authors such as
McCloskey8. Explanations of the failings of mercantilist interventions are well known.
59. It is understandable then that policy makers are drawn to the promotion of innovation.
60. This discussion has noted that innovation has fundamentally different characteristics to those
found in production and distribution activity. Further success – defined as risk adjusted value
creation on a sustained basis – has been described as emerging from processes which seem
counter intuitive in their logic, random in outcome and to involve maverick behaviour.
8
McCloskey, D. Bourgeois Dignity, Univ of Chicago Press, Chicago, 2010.
10
61. Consequently the conventional institutional arrangements and incentive structures of entities such
as governments and government agencies are often ill suited to the requirements for success in
innovation. Indeed outputs typically lauded under such arrangements are likely to be detrimental
to the possibility of successful innovation and thus the growth it might deliver.
62. The implication is that policy makers need to heed the requirements for success in innovation
when choosing the institutional regimes under which they intervene. The requirements might call
for a total re think of institutional design, the contracting out of innovation practice to those who
do operate in structures which are well attuned to success or simply removing obstacles to private
sector innovation wherever possible so as to maximise the chances of success and minimising the
risk of crowding out opportunity.