CALIFORNIA STATE UNIVERSITY, NORTHRIDGE
TERHINATION DECISION FOR
ENGINEERING RESEARCH PROJECTS
i
I
I!
A graduate project submitted in partial satisfaction of the
requirements
fo~
the degree of Master of Science in
Engineering-
II
by
Eugene Frederick Heiman
Januar·y, 1982
I
II
L--·~-
·~-,,. _ _,_n•-•~----~--••-•-• ----·
-
i
I
I
f
:
I
i
I
II
l
i
I
The Project of Eugene Frederick Heiman is approved:
I
I
I
I
.I
Haris_r._._V_a_i~sh-(Advis_o_r~)--------~---
I
;
Rein Turr. (Advisor)
California State University, Northridge
ii
ACKNmiLEDGMENTS
I wish to express my sincere appreciation to Dr. Arnold Roe, ·Professor
of Engineering, School of Engineering and Computer Science, California
State University of Northridge, for his guidance and direction
. throughout this project effort.
I am also indebted to Mr. Joey Fairchild, Division Engineer of the
Engineering
~aboratories,
Lockheed-California Company, for his
enthusiasm and cooperation by providing resources to ·allow me to
complete my graduate studies.
A special note of gratitude is given to Lillian Kobe, Carole
Montgomery and Kathleen Price whose typing and editing efforts assisted
me in the preparation of this document.
A final special appreciation is given to my wife, Trudy Heiman, for her
constant and everloving help in prodding, assistance and morale support
in helping me to achieve this goal of the submittal of this project.
iii
TAB~E
OF CONTENTS
Page
TITLE PAGE
1
APPROVAL PAGE
i1
ACKNOHLEDGMENTS
iii
LIST OF TABLES
vi
LIST OF FIGURES
vii
ABSTRACT
viii
CHAPTER
1.
2.
3.
INTRODUCTION
1
Objective
6
Study
6
Definition of Ter-ms
7
REVIEW OF LITERATURE
8
Introduction
8
Research
9
Research Selection anq Evaluation
11
Research Manager
15
Termination
17
QUESTIONNAIRE DATA COLLECTION AND ANALYSIS
25
Introduction
25
Questionnaire
25
The Population of Interest
25
Assumption of this Sttidy
26
iv
Page
CHAPTER
26
2'!
32
34
34
34
35
35
45
47
47
47
48
50
53
53
55
59
60
61
63
v
LIST OF TABLES
Table
Page
1.
TYPE OF PROJECT VS SUCCESS/TERMINATED RESULTS
27
2.
PROJECT. PERFORMANCE LOCATION VS SUCCESS/TERMINATED
28
RESULTS
3.
SOURCE OF PROJECT FUNDS VS SUCCESS/TERMINATED RESULTS
29
4.
PERHANENT R & D STAFF VS SUCCESS/TERMINATED RESULTS
30
5.
PROJECT IDEA ORIGINATOR VS SUCCESS/TERMINA.TED RESULTS
3_1
6.
PROJECT TERt-1INATION REASONS
31
1.
TERMINATION CRITERIA
36
vi
LIST OF FIGURES
Figure
1.
Page
RESEARCH AND DEVELOPMENT PROJECT EVALUATION
TASK SHEET.
vii
52
ABSTRACT
TERMINATION DECISION FOR ENGINEERING RESEARCH PROJECTS
by
Eugene F. Heiman
Master of Science in Engineering
The objective of this study-was to determine if firms had criteria for
determining
~1en
a
project should be terminated and to outline the
available criteria.
A questionnait·e was distributed to coll'=ct data about engineering
research and development projects and the success/termination results.
This data was analyzed to determine relationships of the research and
development project 1 s environment, management and financing.
Relationships Here found to exist between success/termination results
and the location of where the project was executed, the funding
source, personnel assigned to the project. life span of project and the
originator of the idea for the
p~oject.
vi.ii
Interviews were held with ten research and development
ma~agers,
located in the Los Angeles area, discussing the termination and
selection criteria used on engineering research and development
projects.
It was determined that formal criteria for termination of
projects does not exist.
Research and Development was classified as three types: Contract
Research and Development, Research and Development for Military
Products, and Research and Development for Commercial Products. The
general termination criteria of Contract Research and Development and
Research and Development for Military Products were related to the
contractual status of the program, i.e. the contract concluded or the
military deciding not to piace the proposed idea/product under
contract.
The termination_ crite!'"'ia of Research and Development for
Commercial Products is a series of warning signs.
These signs are
Sl.ltrnlarized as manag.:ment. economic, technical and personnel criteria.
ix
CHAPTER 1
INTRODUCTION
Over five hundred papers and al'tiCles have been published outlining
methods and cri-teria for initiating research projects.
Information is
published defining how to begin projects and methods to control and
continue projects; however, very little discussion has been presented
defining criteria used to determine when a project should be completed,·
· reorganized, combined, or hal ted.
~
I
I The S!Jbject of evaluation and selection of research and development
1
J
projects has been discussed in detail, among others by Burte (7),
I
I Clarke
I
Il ( 16),.
(10), Dean (12), Yershov and Shchedrina (35), Flinn and Turban
MUJ~phy
.
( 24) and Weschler and Brown ( 34).
The selection and
evaluation criteria have been implemented by two methods and were
generally used in tandem.
These methods are intuitive judgment,
including educated guesses and hopeful aspirations, and the use of
numerical project evaluation tools utilizing subjective probability
, analysis.
i
Primary criteria used in the selection are economic; because, after
all, the firm is in business to make a profit.
The termination of a
research and development project is an agonizing decision.
Those
working on the problem are invariably sure that success is just around
the corner.
Criteria for selecting and terminating projects should be
identifiable to the research and development administrators.
1
2
The project selection criteria can be briefly summarized as follows:
o
Consistency with corporate objectives.
o
Technical soundness and newness (Reasonable probability of
success.).
o
·o
o
.o
Capabili.ty of research people available for assignment.
Availability of facilities.
Reputation of originator .
Financial - cost and return.
During the past .twenty years, algorithms have been proposed as Research
i
and Development project selection techniques.
The utilization factor
of these methods appears to be low because the models
r~quire
detail
I quantitative input and/ or the methods are based on techniques too
j elaborate for the Research and Development managers' routine.
I {11)
I and
I
II
Cooper
I
i
suggested an algorithm and value scale to aid in reaching Research
Development decisions.
The criteria are used
Research and Development programs.
The final
tailored to the individual laboratory.
to
rank/evaluate
constr~ction
The development would be the
result of interactive and iterative experience.
A specific example
follows:
The proposed algorithm:
V
=2
must be
log (Impact x Feasibility) + Research
~erit
3
-···------~--~--~---····
·--··
----~----~--~-.
--··----··-·····----~
·-------·--·-····---
The terms are defined below:
Impact
=
(Relevance) x (Authority) x (Coupling),
Relevance
Value
Parameter
1000
Recognized Organization Goal
100
Supports Recognized Objective
10
1
General Function Research
Limited, Specific Interest
Authority
Value
100
10
Parameter
Specific Request
Laboratory/Department Objective
5
Assigned Responsibility
1
General Laboratory Responsibility
Coupling
Value
10
Parameter
Direct User Participation
5
Substitution in Existing Market
1
Uncertainty, indifference
0.1
Conflicts, Displacement
·-·-····~---
---,
I
4
Feasibility
= (Technological
Risk) x (Competence) x (Management)
Technology Risk
Value
Parameter
10
Demonstrate Practice
4
Evidence Technology Exists
1
Uncertainty but resolvable
0.1
Major Advances Required
Competence
Value
10
Parameter
Necessary Skills
4
Skills generally available
1
Requires learning new skills
o. 1
Demands significant effort to
acquire skills
Management
Value
10
Parameter
Acknowledged Progr2m
4
Necessary Skills Achievable
2
Requires Substantial Effort
1
Must "import" champion
5
and Merit
= (Research
Opportunity).+ (Foster Research Strength)
Research Opportunity
Parameter
Value
2.0
Major Opportunity
1. 0
Broad Impact
Compliment other Research Programs-
+1 to -1
Specific Technology Fix
-1
I
Foster Research Strength
I-
I
Value
I
1
I
0.5
·I
Parameter
Enhances Research Talents
Maintains Sldlls at State-of-art
I
I
Some opportunity for growth
0
Depletes, Diverts, Obsoletes talents
-1
I This
1
ex-ample with the priority order permits the Research and
I Development
II
manager to determine and compare specific strengths and
i weaknesses of the projects.
It allows the manager to evaluate the
i
I projects
in terms of the long-term effects in the laboratory.
Some
!
I risk must be taken; after all, if we wanted no risk, no research would
I
be performed and no new products would be developed.
The values
assigned to the parameters must be adjusted to_accommodate each
evaluator's set of values.
The parameter values should be adjusted to
refle-ct the accuracy of the parameters to be evaluated.
When a project
is being selected, the probability of success would have a wider
6
• variance than that of a projeet nearing its completion.
As a result,
the selection function should have different utility values and some
! different
parameters than the evaluation function.
.
; This study was to determine criteria of termination of Engineering
.
; Research and Development projects.
i Objective
i.
This study was performed to determine if firms do have criteria for
I
determining when a project should be terminated and to outline the
! available
criteria.
I
I
I .
1 Study
i This
1
study was performed in four phases:
i
1.
I
i
I
A revie\-r of literature was conducted to determine what efforts
have been made to define criteria for termination of engineer-
!
ing research and development projects.
i
i
2.
I.
!
A questionnaire
wa;· prepared
and sent to selected firms in
order to identify criteria that effect success/termination of
specific proje0ts.
3.
Ten Senior Engineering Research and Development Hanc;gers were
interviewed to determine the management techniques used in
their operations for selection and termination cr-iteria of
engineering research and development projects.
7
Response from phases two and . three were studied and conelusions and recoJMlendations were drawn.
A project evaluation
model was developed and is presented.
Definition of Terms:
A SUCCESSFUL project is defined as a project that met the project
goals and has been classified as completed.
The results of.this
project may or may not have been used or applied to the particular
company products.
A TERMINATED project is a project that was reorganized, combined,
abandoned, or unsuccessful.
A terminated project did not meet the.
original goals and the results were not applicable or useable -to
the company's products.
A terminated project may have been halted
because of lack of adequate funding.
i
'
I
I
I
CHAPTER 2
REVIEW OF LITERATURE
Introduction
A search of the literature was conducted in public and university
libraries located in Los Angeles.
The National Aeronautics and Space
Administration, Scientific and Technical Information Facility and the
Defense Technical Information Center of the Defense Logistics Agency
were utilized using the key words "Research Decision t1aking."
Secondary search key words utilized were "Decision Making" and
"Research Management."
Searches produced an excess of 326 citations.
Comrnerci al Computerized Abstract and Literature Search Systems were
used and produced an additional 300 abstracts and citations.
The search revealed an extremely small amount of literature that
relates to the subject - termination of research projects.
It may be
that the word "terminate" creates a negative image and research
administrators and managers do not like to be associated with failure
and therefore have not documented their reasons or criteria for
stopping research programs.
Research programs may be completed,
reorganized, or combined but they are hardly ever halted.
criteria for terminating
The idea or
a particular research program and/or moving
i
the project into a product development or production phase may be under·
ia
subject title that this researcher has been unable to ascertain.
8
9
• Research
Research is by nature dynamic, its very purpose is to create change.
i The Random House Unabridge Edition dictionary (29) defines research:
i
i
i "diligent
and systematic inquiry or investigation into a subJect in
order to discover or revise facts, theories, applications. et6."
Other:
!
studies (20) (28) (32) have defined research
objectives.
furthe~
according to its
For potential comparison, the following definitions are
provided:
BASIC RESEARCH is that type of research which is directed towards
increase of knowledge in science.
It does not have specific
product/process objectives, although such investigations maybe in
the field of present or potential interest of a compan·y.
APPLIED RESEARCH investigates the discovery of new scientific
knowledge that has specific objectives with respect to products or:
processes.
Applied research differs from basic research mainly in!
terms of objectives of the company.
DEVELOPHENT RESEARCH projects include technical activities of a
nonroutine nature concerned with translating research findings
into products or processes.
Quinn (28) wrote that research results are difficult to predict.
Projections of intended results are often insufficiently accurate t6
serve as standards of performance, and actual results are frequently
10
...•. 1
different from those planned.
Often, the only results obtained
is an increased knowledge of the subject area, the formulation of new
ideas ·and/or an improvement in research techniques,. all admittedly of
an intangible nature.
Holzmann (19) stated that the inherent character of research- its
uncertainty, intangibility, uniqueness. of accomplishments, longtime
delays in feedback results - has frustrated attempts to construct
managerial
practic~s
such as those used for managing other resources.
Research and development suffers from lack of standards of performance,
lack of understanding of its process, and lack of an educational basis
for its managers.
Research is an investment that must be studiously considered and
evaluated in the same light as any other corporate investment.
No
business can be successful without some program that provides a line of
new and improved products.
activities are necessary.
To be viable, some research and development
Since any research involves costs, there
ought to be a cost-effective factor, even in the question of basic
research where the proper objective is knowledge.
Is this scientist
generating a "reasonable" amount and quality of knowledge?
There is no
more justification for promoting an unprofitable research program than
r
i
i
there is for continuing an unprofitable product line·.
11
: Research Selection and Evaluation
j
'Research projects do not have sharp beginnings and endings; they seldom
i progress
from exploratory stages directly to development.
Instead they
]
; pass through the feasibility research stage in which the:lr technical
: and economic viability are investigated.
As a general rule, develop-
i
: ment projects should be undertaken when ul t.imate technical and
r
economical su.ccess is fairly well guaranteed.
The research and development manager must realize that product planning!
and budgeting are a preparation for the future, and modern product
planning involves long-range projections into a highly uncertain
future.
There are tools available with which to make an initial
estimate of the probability that a certain action will be successful
..
and then define i t with a series or additional probabilistic
calculations.
There is value in using simple statistical designs in
research situations involving variable screening and optimization.
Once experiments are well planned, as in a statistical design, the.
analysis of the data is very often straight forward and leads to
informative results.
Howard (20) described his method of logical
decisions, "\-lhen we have established y_our preferences, the values you
place on outcomes, your attitude towards time, your attitude towards
risk and have established models necessary for the decision and have
assessed probabilities as required on the uncertain variables, then we
I need
I
I
nothing but logic to arrive at a decision.''
A good decision is
j not very simply defined as the decision that is logically implied by
i
I
l.---- ............ -···~- -- '-
·---~
"-.
12
the choices, information, and preferences.
the loop.
Man is still very much in
He is deciding which decisions are to be made and when.
And.
then, facetiously, there is no ambiguity frorn that point on -- there is
only one logical decision.
One technique called "Decision Tree Analysis for Industrial Research"
(l6)
has been proposed as a
selection.
m~thod
for managers to perform project
This method looks at a decision tree of capital investments'
i
l required
I! the
In this method,
manager attempts to project the events that will occur between
Ij inception
I and
to run the research project to completion.
and completion of a project.
Probabilities of success, cost
return are estimated for each of the steps.
By this means, one is
I
I able to predict a probable profitability of the project.
i
I proceeds,
I
As a project
this decision tree may be used as a control tool to determin~
whether the projects can
b~
allowed to continue or to be terminated.
The project managers, before the project is started, can perform
sensitivity analysis to evaluate the total system and to determine
whether the desired rate (l,f·return is achieved, or how sensitive it is.
to variations within the system.
Barclay, et al (1) wrote "it should be emphasized that in no sense
doe~
the decision analysis replace decision makers with the arithmetic or
does it change the role of human judgement in decision making, rather
it provides an orderly and more easily understood structure that helps
to aggregate the wisdom of experts on the many topics that may be
I needed
to make a decision.
L,_._ ·--··:--·-····----.. .,.--- ·--·------·-··.
-. - -·---- --···· --------
It supports the skilled decision maker by ;
----~----~------------- ·-- ... -~-- ·-··-- ----· .•..- -··.
------ ------- *•-· --
····------------- .. , ---- ··-·· -.---
--~
13
providing him with logically sound techniques to support, supplement,
and insure internal consistency of his judgements."
The first goal of the research manager is to be able to select projects
that afford the greatest promise of technical success. However,
•
teohnic~l success does not assure a significant return 6n investment.
It can .still be a commercial failure.
A bridge between research and
development and marketing is vital to enable all to work toward common
objectives.
Gerstenfeld, et al ( 18) wrote that more than half of the
eleven 1msuccessful projects
studied were really due to marketing
problems rather than technical difficulties.
The danger of
technological-push (where the idea originates within the technical
group) innovations is that. the innovators often misread their market.
Project selection criteria that have been suggested by Clarke
(10)
include:
1.
Consistency with corporate objectives.
2.
Technical soundness and newness.
3.
Capability of research people· for assignment.
4.
The availability of special facilities.
5.
Reputation of the originator.
6.
Resourcefulness of the project proposal and originator.
These criteria are applicable to the firm and it's selection process.
When determining selection criteria at a national level, broader
guidelines must be specified.
The criteria may become more political.
14
···-~
The Russian authors, Yershov and Shchedrina (35), wrote that in their
country there were four selection criteria for Research and
Development:
1.
Reliability Criteria for the Selection.
This criterfa is to
justify and to evaluate the hypothesis to eliminate ill-founded
assumptions.
2.
Technical-Economical Selection Criteria.
These criteria are to
allow comparable evaluation of technical and economic impacts
to evaluate if this research will produce the expected effect
·and if the expected return can be obtained from the innovation.
3.
Systems Criteria.
This criteria will evaluate the prospects of
this specific research on the basis of degree of effect it may
have on other scientists and fri determining its role in the
wider science-technology industry system.
4.
Policy Criteria.
This wide class of criteria is based on how
the research will affect internal and foreign policy in the
general social structure of their society.
Aft~r
the project is selected, evaluation of the project must be
performed during it's life.
Holzmann (19) states that there are five
significant situations which require periodic revaluation:
l.
The loss or the acquisition of personnel with special
knowledge.
2.
Development of new information, technical or marketing.
3.
Changes in company goals.
15
4.
Availability of new .equipment.
5.
New technical developments.
Block (3) suggests that when evaluating research and development
resource allocations, there are several requirements to be considered:
1.
A work unit should be identified as a smallest identifiable
level of effort within the laboratory.
2.
The method of an evaluation should not be made detrimental to
human relations·between members of the organizations.
3.
The evaluation technique should be flexible.
4.
Each work unit should be evaluated in view of the understood
priorities of the parent organization.
Research Manager
Generally the research manager 1 s position is in middle management which
requires that he establish-a viable communication channel between the
scientist and the management.
According to Holzmann (19), the research
manager's responsibilities include:
1. Telling the
people~of
the laboratory what are the needs of the
organization and telling management where new markets look
promising.
2. Defining needs in promising markets in ways that enable people
to direct technical capabilities toward corporate objectives,
and relating to managers those developments that have begun to
show promise.
His influence is powerful in the world of research and the world of
mar·keting.
Brooks (4) states that one problem in the decision pattern,
16
is that an individual strives to maximize indivtdual goals which may be
detrimental to the organization or mission goals.
The individual goal
is related to the power and prestige of an individual, whereas the
:bureaucratic goal is related to both the specified objectives -of the
;organization and the individual's perceptions of the organizational
i
i
;goal.
!
Burte (7) noted that termination decisions have actually depended upon
the qualitative judgement of the research manager.
Possibly management
science should pay more attention to the stimulation and development of
!people suited for such decision making. It calls for a vertical
I
Jinterdisciplinary man who can combine the input from scientific
!possibilities with that from economic ot technologicAl needs.
i
l
i
!Chief engineers interviewed by Dean
(12)
asserted that a typical
I
\project engineer does not know when, or how, to administer proper
I
I
jcutoff of a project and particularly because of:
I
I
I
1•
A desire to be technically perfect.
!I
2.
Personal technical interest of line engineers.
i
3.
Willingness to optimistically regard technical difficulties in
I
I
order to guard a line engineer's reputation.
4.
Desire for recognition and sense of accomplishment afforded by
technical victory.
5.
Low predictability.
'While on the other hand, chief engineers agreed that there were some
I
(factors that support continued experimentation, namely:
'
""--·-----·~--~
····-------
---~
..
-- --------
""
'··-
------·-· ------
--- ---------·.
-~--
------- . - ·--···-----
-------..
------~-
-------- ------,-·- -- . ----··----- ---------- ·---- -----·
---
17
1. E&igineer's frustration reduced through satisfaction of
curiosity.
2. The possibility of discovery is allowed.
3. General personnel relations are aided.
4. Experience is provided.
An interesting idea, relative to people who are to judge and analyze
r~search
and development activities was presented at UCLA (35).
The
person passing judgement must have a demonstrated capability in the
field, preferably by having obtained some distinction in actual
research or development of their own which in turn was judged by their
· peP.rs.
It is best to evaluate research and development activities by a
person who has performed those similar activities.
This person
mak~s·
the decision as to whether the project is to continue, to receive
additional resources or if resources are to be reduced or deleted from
the project.
Therefore, the problem of how to evaluate research and
development activities become one of evaluating the evaluators.
We may
need to select honest and __ intelligent men who have obtained distinction
in the field so that we can use them as evaluators.
This method of
using peer evaluation may help eliminate some of the bias that may be
prevalent versus of having an evaluator who is evaluating his ovm pet
project or idea.
Termination
Top management must make intensive efforts to identify precisely the
·nature and value of- specific pro·gra11s or projects and to match these
18
··- '--
--··~
..
~-------~-~---------~--~------
·---
--·
---
------~--·---------------·-·
-.,.......-~----·-··- ----~-------·
..
------.----~-------------------------·~-----------
------,
I
.
i
identifying what system or project might be dropped, consideration:
should be given as to whether the program has completed a stage in the ; .
development cycle.
Additionally, if one or more systems are to be
dropped with the savings transferred to other systems, it may be
beneficial to
t~e entir~
company, or in the case of the government,
defense posture can be gained.
So, therefore, the interdependence of
the value of the total system's cost should be considered in the "drop
decision.
It is difficult to set down specific and reasonable incontestable
guidelines of wide spread applicability concerning research termination.
An attempt to define the guidelines will require answers
to the;
following:
o
Top management - Does the project conform to corporate goals;
will they finance it; will it achieve the minimum
profitability demanded and will management exploit it in an
existing or new market?
o
Other departments- Has the project been integrated with
marketing· and manufacturing and do they support it?
o
Research department - Is the project timely, innovative and
balanced with respect to other projects as to level of effort,
age and type?
!
o
P~rsonnel - Are they enthusiastic and is there a leader?
I
I
I
I Despite the attempts
I
o(
many companies to establish decision points
l~~~-~~d-,:~~~~. -t~~~--~~-1~-~-~t--~~---~~-~h~~-~---~~-~~~~·~·~·--~u-~-~-~~~c-~~~~~-:- they find•
p '
19
r·-~-~---
..
--~·····
......... - ............. ... ......... . ...............
~--------·--------
............._....
-~
.. · · -· ......................... ------ ···-"................................... .
l
carefully with their company goals.
terminated?
On
When should a program be
successful research and development projects some or
all of the following criteria were met:
1.
Developments were timely in that the technical success was
transformed into economic developments only when the market
became fertile •
2.
The projects were compatible with corporate aims.
3.
The projects must relate to the next logical step.
4.
Successful projects were-set up to develop a process or
product to fill a well-understood and easily defined need.
Gerstenfeld ( 17) wrote that research ought to be terminated· if it is
I
.
!not relevant.
Research must interface with engineering, production,
!marketing,. and legal.
Often in engineering, there is a resistance to
!research knowledge because-it wasn't developed within the engineering
!organization itself. Research people are often unwilling to recognize
I
!the constraints that specifications and schedules imposed on engineers.'
I
;carney (8) gave the following examples of reasons for product
I
!termination in companies:
1.
Reaching the objective.
2.
Costs become too high.
3.
Solutions require personnel more specialized than are
available.
4.
Increasing hazards.
5.
Finding an
alt~rnative
solution.
company interest.
20
themselves having to make investment decisions on
in a general climate of uncertainty.
insufficien~
evidence
However, a company should stop
developing a product as soon as it is shown to be technically not
feasible and/or economically unsound.
Too many technical managers
assume that the decision to initiate a development program implies a
.
development status to completion.
Burte (7) noted that some_ company's policy is that the dominant
criterion for continuing a project was to use the expected net return
on the estimated remaining investment required to complete the project.
It was indicated that decisions are easy to make only in those
instances where the key factor is either that the project is
demonstrated successfully or that the support is abruptly withdrawn ,.for
reasons outside of the control of the research manager.
Some of the
more difficult types of decisions is when a gradual or partial
withdrawal of resources occurred, such as the availability-of funds
became less and less until it reaches a point where it is felt that a
viable project could no longer be maintained and therefore was
--~
cancelled.
An.other reason to terminate is because of the loss of the
experienced project leaders.
A decision might be made to terminate
even though work could have advantageously been accomplished.
Another
termination action was initiated when an idea arising from a scientific
possibility that had led to a technically successful project was
terminated because of the inability to relate this project's
profitability to the parent organization's business.
21
The discipline of time is important.
When the discipline of time is
not imposed on research efforts, research is not terminated when it
should be terminated,. opportunities are lost and the resources are
wasted. Twenty-two projects, of Gerstenfeld's (17) study, were
evaluated by utilizing averages of manpower assigned and the elapsed
time in the number of years to complete each of the projects.
It was
noted that generally signais of success or failure appear, however,
these early warning signs may tend to encourage an organization to
either gamble with a large number of personnel assignments for the
signals of success, or to limit the numbers of people assigned to the
job if signals of failure start to appear.
Sometimes, when faced with
negative information, the firm may have a great difficulty in stopping
a project but will allow it to limp along.
Gerstenfeld noted that
unsuccessful projects generally had longer life, whereas, projects that
were successfully completed, typically ran in a lesser period of time.
In his sample of twenty-two, the mean of both success and failures were
four years, however the standard deviation was slightly less for
successful projects than it was for failed projects.
It may be that
t(le negative information often becomes distorted or sufficiently
blocked so that the organization management does not receive all of the
information or a complete story and therefore does not terminate the
project because of this lack of information.
While an organization may
be willing to decrease the manpower assigned to each project, it may be
unwilling to make the more difficult decision to end the project.
However, with the benefit of hindsight, the organization may now
22
realize the danger signals were utterly apparent, but that the
organization chose not to act until additional investments of technical
energy had been exerted.
Murphy (24) indicates that the exclusion approach is a way to select
•
the projects that are to be terminated.
It_ requires a different type
of selling activity by the project engineers or managers than would
normally be used to explain why the project or process should be
continued.
1.
This exclusion approach may be used on a project that:
The promise of technical success is so small as to render the
project improbable.
2.
Would take so much more time or more money than is available.
3.
The strategy need for the product or process is no longer
high.
4.
The product cannot· be manufactured economically.
5.
Production process is so costly that the market gain is no
longer attractive.
___.-/
There may be a single criterion ·among the many possible that is so
important that the decision of continuance or termination must hinge on
that criterion.
In this case, the exclusion approach should
concentrate on that single factor.
In discussing project cutbacks or cancellations by the Pentagon, an
article in Fortune ( 32)" notes that "there are exploratory looks at a
sizeable number of possibilities but advanced work is done only on
23
those that seem promising.
part of the
oper~tion
Cutbacks and cancellations are a necessary
and not a sign that things are going wrong."
Fortune quotes Eugene.Fubini as saying "I want to try as many ways as
possible to do something, then I will select."
This approach described
a kind of "funneling" within a funnel which is wide at the research
aperture and narrow at the engineering spout. With this funneling
approach, project terminations are a result of exclusion, not a sign
that anything is wrong.
Many organizations make little attempt to quantitatively evaluate
individual research projects, but try instead to measure the output of
the total research function.
Their approach is to view research
success in light of overall company progress without arbitrary
establishment of payout or amortization periods in allocation of
research and overhead costs for a particular prcject.
Carney (8) documents the final budgeting in the Department of Defense.
An example is how the Army accomplishes their budget cuts. Difficult
trade-off decisions must be made.
Generally, only the highest cost or
critic ally important i terns get careful investigations, while other
projects in the program may experience something of approaching an
across-the-board percentage reduction.
Myers (25) noted that small research and development projects tend to
remain unstructured and. open-ended, and are typically managed with a
minimum of review from higher authorities.
He noted that once
24
implemented, the projects usually continue at some effort _as long as
funds are available and the specific performance milestones are not
generally identified.· Funding becomes the criterion for program
continuation, revision, or termination.
Re-evaluation of the
requirements and justification of the project is seldom critically
•
analyzed except when funds are scarce.
Holzmann ( 19) asks, "When a research project fails the corporate
criteria for success, what should be done then?"
By-products are
defined as a secondary or additional product and sometimes unexpected
results can be profitable.
It is important that management implement a
strong policy and program for licensing research and development
products;
These "failuresri should be disposed of externally as
profitably as possible.
place!
M?nY products were invented in the wrong
There are many examples that indicate new product successes
result from innovative application of what might have been considered
negative results.
Penicillin would seem to be of no value to most
observers because it killed everything.
be a total
disca~d
wasted motion.
However, a product that should
must be recognized as such and eliminated without
CHAPTER 3
QUESTIONNAIRE DATA COLLECTION AND ANALYSIS
Introduction
A questionnaire was developed to determine if
rel~tionships
exist
between the management decisions and the environment of the development
projects and the success/termination completion status of the project.
This chapter contains the
analysis of the significant responses to the
questionnaire that.was mailed to thirty-two firms.
QuestionnClire
Project success or termination may be dependent on certain requirements
and environments.
These include the type of environment where the
'project was performed, type of personnel assigned to the project, or
who originates the idea for the project.
A questionnaire was devised
to determine if a correlation of data exists.
It was arranged so that
answers to the questions could be easily tabulated and entered into a
computer for statistical analysis.
The questionnaire and the
associated data sheet are contained in the Appendices.
The Population of Interest
This study was directed at engineering research and development firms
that were mainly located in the Southern California area.
These firms
typically deal in high technology that is utilized in the commercial
and military fields.
Some of these firms have been involved with the
University and the managers expressed an interest in the University and
25
26
1 ts research activities.
Other firms were selected because the
laboratory managers of research and development activities were known
to be cooperative with other research projects.
Several well-known
national companies, as leaders in research and development activities,
were included as targets for the questionnaire.
Responses were
received from twelve of the thirty-two firms with nine firms returning
project data sheets.
The
ques~ionnaire
provided space for responses to
more than one project and.a sample of thirty-two projects was obtained.
Assumption of this Study
It was assumed that each set of answers for a project from a given
company was independent of answers provided for other projects reported
by the same firm.
Statistical Analysis
The data obtained via this questionnaire are handled as nominal level
data.
The existence of nominal level data calls for the use of non-
parametric statistics to
groups.
_t~st
any meaningful relation between two
The use of non-parametric statistics was deemed satisfactory
to test the significance of the results.
A review was conducted of the statistical packages available on a
computer system that was available for use during this study.
The
SPSS: Statistical Package for Social Sciences (27) system was selected.
27
Questionnaire Data Analysis
An analysis of the significant data of the questionnaire follows:
PROJECT TYPE - Si_xteen of the projects were classified as commercial
projects and sixteen projects were. classified as military projects.
projects were classified as non-profit projects.
•
No
Table 1 summarizes
this data and the success/termination results.
TABLE 1
TYPE OF PROJECT VS
SUCCESS/TERMINAT~D
RESULTS
Number of Projects
Type of
Success
Project
---
Terminated
Commercial
11
5
Military
13
3
A non-parametric statistical cross tabulation analysis was performed on
the data of Table 1.
A Chi-Square test gives
x2
= 0.667(1) which
indicates no significant difference in the success/termination ratio
for commercial vs military projects.
PROJECT LOCATION - Twenty:..two of the projects were performed in
separate research and development facilities.
Eight projects were
The Chi-Square statistic may contain a little distortion in the significance level because one cell is less than 5 but greater than 1 ~
28
performed in an engineering environment and two projects
':~ere
performed
in a combination of engineering and production areas.. No projects were
performed in a production area.
This data along with the
success/terminated results is summarized in Table 2.
TABLE 2
PROJECT PERFORMANCE LOCATION VS SUCCESS/TERMINATED RESULTS
Number of Projects
Location
Success
Separate R & D facility
Terminated
16
6
Engineering area
8
0
Engineering & ProducHon area
0
2
A non-parametric statistical cross tabulation analysis was performed on
the data of Table 2.
2
results in X
at P <.02.
The chi-square test of statistical significance
= 8.73 (2)
.
w1th 2 degrees of freedom which is significant
Therefore we can conclude that the project performance
locations vs the success/termination ratio are not independent.
PROJECT FUNDING - The source of funds for each project was identified
and is summarized in Table 3.
The success/termination results are
included.
2 3 cells have expected frequencies of less than 5 and therefore the
chi-square statistic may be distorted.
29
TABLE 3
SOURCE OF PROJECT FUNDS VS SUCCESS/TERMINATION RESULTS
Number of Projects
Source of Funds
Success
Terminated
100% Internal
9
6
External
13
0
.
100%
The chi-square test gives
is significant at P
x2 = 6.62( 3 )
= 0.01.
with one degree of freedom which
Therefore we can conclude that the
project's success/terminated results is not independent of the source
of funds.
PROJECT PERSONNEL - The staff of personnel assigned to research and
development projects was comprised of a 100 percent permanent research
and development staff for 17 projects and. less than a 100 percent
permanent research and development staff on 15 projects.
This data is
summarized in Table 4 with the success/terminated results.
A chi-square tests gives
is significant at P <.01.
x2 = 7.07( 4 )
with one degree of freedom which
It is concluded that when a staff of 100
percent permanent Research and Development personnel is used on a
project the results of success/terminated are not independent
3 1 cell has an expected frequency of less than 5 and therefore the
chi-square statistic maybe distorted •.
4
The chi-square statistic may contain a little distortion in the
significance level because one cell is as small as 1.
30
TABLE 4
PERMANENT RESEARCH AND DEVELOPMENT STAFF VS SUCCESS/TERMINATED RESULTS
Number of Projects
Permanent R & D Staff
Success
100%
Less than 100%
16
1
8
7
PROJECT LIFE -The project life spans were analyzed.
·ranged from 0.4 years to twelve years.
was 3.61 years.
Terminated
The project life
The mean life of all projects
The mean of the successful projects was 2.89 years and
the mean of the terminated projects was 5.75 years. At-test was
performed with t
·at P <.02.
= 2.52
and 30 degrees of freedom which is significant
It can be concluded that the project life is not
independent of the success/terminated results.
PROJECT ORIGINATOR - The originator of the projects were tabulated and
summarized in Table 5 with the success/terminated results.
The chi-square test gl·ves x2
is significant at P <.01.
= 13.89(5)
with 4 degrees of freedom which
It can be concluded that the originator of a
project and the success/terminated results are not independent.
5 7 cells have expected frequencies of less than 5 and therefore the
chi-square statistics may be distorted.
31
· TABLE 5
PROJECT IDEA ORIGINATOR VS SUCCESS/TERMINATED RESULTS
Number of Projects
.
. Originator
Success
Terminate
Top Management
2
1
Middle t4anagement
1
0
Supervisors
0
1
Engineers/Scieritists
5
6
16
0
Customer (user)
CRITERIA/REASONS OF TERHINATION - Eight projects in the survey were
terminated.
The reasons are tabulated in Table 6.
TABLE 6
PROJECT TERMINATION REASONS
Reason
. ---
.
No. of Projects
Change of Management
2
Lack of funds
1
Potential payoff too low
1
Slow progress
1
Technical problems
1
Low DoD ratings
2
-
32
From the eight projects terminated, the reason for the termination
follows the same rationale given during the interviews (Chapter 4).
It
is note worthy that two of the projects were cancelled because of low
Department of Defense rating.
This DoD rating does affect the
company's income tax in determining how much of the research and
development cost can be written off as an expense item.
By eliminating
these low projects, one can assume that the company will raise their
scores so that more of the research may be written off against their
revenues.
General Questionnaire Comments ·
The following comments are included with this study to assist other
persons performing similar- study activities utilizing questionnaires.
If the specific name of
t~e
projects had been as listed on the
questionnaire data sheet, I believe that this information would have
in performing the analysis.
Some of the answers to the questionnaire
may have been in error because the responder was not adequately
instructed as to the nature of each question and the possible answers.
A project may have been classified as commercial because of the way the
Federal Government and its various bureaus and agencies fund research
programs.
A NASA project while being non-military probably was
classified as commercial, however the product would never be sold in
the local hardware store.
commercial or military?
An example - is the Space Shuttle Oribiter
33
Most literature indicates that the response to questionnaires is
typically in the 10 - 15 percent range.
32 questionnaires were returned.
data sheets.
On this questionnaire, 12 of
The 9 of the 12 companies returned
A possible reason for this higher rate of return is that
many of the questionnaires were addressed to specific individuals
within the various firms.
notes.
Several of the responses included personal
This leads me to believe that the personally addressed letter
will result in a more positive
response~
. CHAPTER 4
INTERVIEI'l SUMMARY OF ENGINEERING RESEAHCH AND DEVELOP1·lENT MANAGERS
Introduction
This chapter contains the summary of intervieHs w"ith ten Senior
Research End Development managers.
These managers were interviewed to
obtain an insight of their firms' selection and termination criteria
and how the termination criteria may relate to project stze and project
cost.
_popul~;ior!__.Of
Interest.
The target firms were selected from the Professional Advisory Council
to the School of Engineer·ing and Computer Science and from the
Industrial Associates of the School of Engineering and Computer Science
from the California State University, Northridge for the year of 1980
through 1981.
These tHo lists v:ere selected because I believed that
these representatives would be willing to assist the University and its
students.
Tne lists were reduced by eliminating advisors that were not
aeti.ve in the engineering field and advisors fr·om companies that had
been selected to receive a questionnaire.
located in the San Fernando Valley.
34
The selected companies wet·e
35
These firms produce a large cross section of products.
Products
include:
Aircraft
Computer Systems and Displays
Computers
Engineering Consultants
Hydraulic Equipment
Missile, Spacecraft, and Aircraft Guidance Systems
Computer Programs
Tlv: sales of the firms range from $5 million per year to several
billion dollars per year.
Each counsel member or advisor was contacted by telephone.
1he reason
for the call was explained and an intervieH was scheduled Hith the
Research and Development Hanager.
an intervieH.
Personnel from nine firms agreed to
One firm suggested that I tall< to tHo managers.
interviews Here conducted.
IntervieHs
Four major questions were presented during each interview.
questions and answet·s are presented below.
The
Ten
36
QUESTION 1:
Outline your firm's
11
termination cri teria 11 that were used
when a project was terminated.
The answers to question
are summarized in Table 1.
TABLE
7
TERMINATION CRITERIA
----~-~-~~-···----·-----------
'I
No. of
I
Firms
Criteria and Reasons
-· --- -·- --==-
Responding
·-·=-=--==:===
A. Return on Investment (ROI) Has not adequate.
3
B. Other projects became higher in priority and the
3
lower priority projects were terminated.
C. Pr·oject dl.d not satisfy performance
requit~ements.
2
D. Requirement (need) for project dl.sappeared.
2
E. Lack of Funds
1
F. Conflic't of responsibility for the project caused
1
it to be terminated.
G. Project schedule was not met.
1
H. Project was terminated \vhen i.t became part of another
project.
I. t1anagement 1 s judgemental decision
lJ.
Project over ran budget.
-----------------------
1
1
I
-a
37
Economic considerations are high on the list of termination criteria
and reasons.
The return on investment greatly influences the analysis
of the project during its life.
To assist in this analysis, a project
schedule is setup for the product including milestones describing when
the product Hill be ready for production, how many dollars they intend
to spend on the project, and very importantly, what would be their
gross margin on the end i tern.
After the schedule has been set E1nd the
project has been approved and initiated, periodic project revievJS would
be held.
These revievlS may be held as frequently as once per month.
Continuous monitoring occurs to determine if the performance meets the
plan.
Of importance is the cost of the project and if it has met the
projected cash flows.
reviews.
Technical judgements are performed during these
Tne project is scrutinized to detennine if it is still as
good an idea as when it was initiated.
The market place must be
continually reviewed, especially when a project is overbudget and its
original schedule is no longer satisified, to make a determination if
the market is still available.
Projects have been tenninated that vlere on schedule and budget because
the market and competition caught up with the project..
The firm
determined that they would not receive an adequate return on their
investment so the project was terminated.
One of the most difficult
times to terminate a project is when the project has moved beyond the
research and development stage and has progressed into prototype and
early production.
to be satisified.
Again, the retur·n on investment Has the major i tern
38
Two managers stated that some projects should always fail.
If you have
no failures. it may mean that you are not pushing the state-of-the-art
enough by the research.
If you are pushing the state-of-the-art,
occasionally you >>ill fail because you cannot 8lways move the art.
In
true advanced development, odds may be as much as fifty or one hundred
to one against you solving a technical problem to a total successful·
conclusion.
You will alHays gain some information and thet'efore you
will have a partial success.
Projects should be partially sUccessful.
One must ali-Jays look at it Hith an optimistic point of view.
. an important part to saving face.
This is
No one likes to adm}. t that they are
a failure or to having participated in a failure.
From an engineering management view point. technical requirements and
test plans are probably the most critical milestones in revieHing
research and development projects.
Generally. schedule and cost growth
can be overcome; however, technical problems are more difficult to
satisfy.
One big problem in Contract Research and Development (CRAD)
is that the original goals and project requirements are generally
satisfied but numerous changes occur during the life of the project
created by the principal investigators and the contractlng
organization.
These changes can be said to have done a very good job
of mudding the contractual water.
It is then more difficult to tell
whether you have a valid reason for terminating a project because it
does not satisfy the technical requirements as no\-;r defined
vel~sus
the
39
technical requirements defined when the contract was entered.
These
technical problems and challenge of growth must be closely watched by
management.
Some research projects are abandoned because funds are allocated to
other projects that management has decided to pursue.
An example is
the research on metals that has been basically stopped and research is
noH concentrated on composite materials and powder metals.
NASA has
init:i.ated many contracts relating to composites and powder metals that
may be used to reduce the structure weight of aircraft and spacecraft.
The military has required that some of the aircraft structural members
to be converted to composite materials.
structm~al
NASA has initiated research on
members for commercial aircraft.
This change in rcseal'ch
initiated the termination of the research on metal projects.
The
metals projects v!Ould probably have been funded if more money and the
number of researchers were available, but these research projects were
given a
J.m~er
priority and composi te/poHder metals utilize the Hmi ted
resources av atlable, the research activity \-las hal ted.
Some projects have been terminated after the job Has completely
developed.
An example is one particular product of a set of leg braces
that were made of composite materials.
These leg braces were much
lighter in weight and appeared to function better than the metal leg
braces.
decision.
HoHever, product liability entered into this particular
The manufacturer was willing to pr·ovide these leg braces at
a very competitive price; hoHever, the distributor was asked to assume
responsibility for any public liability in case of accidents Hith the
braces.
The distributor was not willing to take this risk and neither
was the manufacturer.
This project was terminated after the trial runs
and prototypes had been used and fully developed.
The manufacturer
used this composite material to develop the shafts of golf clubs.
This
has turned out to be a very pt•ofi table market with this particular
material.
The manufacturer supplies the shafts and al1mvs the golf
,club manufacturer to assembly the head and hand grips as required for
the particular needs.
QUESTION 2:
\-That selection criteria are used by your firm.
The selection criteria are s•1mmari zed belotrJ:
o
Customer requests an idea or product improvement.
o
A problem is recognized and an answer (solution) is sought.
o
Firm is looking for diversification in a new product/service.
o
Marketing has found a new product line.
o
Pur suing neH concepts and ideas.
During the inte!"VielrJS, it became apparent that commercial resear·ch
projects are selected differently than military research projects.
\~hen
a commercial research project is selected. a business plan will be
prepared including total cash flows, analysis of competition, and
consumer buying power for the particular product.
Some firms have used
computer models to assist them in the business plan, but have
subsequentially stopped using those models. (People have a tendency to
. believe the accuracy of the data from the computer model forgetting
that the input to the model was imprecise and therefore the output is
also impreclse.)
economic.
The main thrust of commercial research is the
The company survives on profits and if the ROI is not what
the company exp<:wts, the project is not selected.
An example of Hhat
oceurred to one firm follows:
The small finn, whose main product lines had been associated with
ths governffient and military attempted to diversify by performing
research and enter into a commercial field.
Tbe ne;..r product line
has required a significant amount of funds from the firm and has
not proven successful after two and half years.
The m:1rket
( ilTigation equipment) has partially ev8.porated since ti·te :·eturn
of norma1 rainfall in the California area.
At the time the
venture was started, during the drought of 1976/1977. the product
looked very promising.
This particular fit·m did not hire or
fmplement a marketing team and a marketing survey was not
performed.
A lack of a detailed business plan has now become
evident as a shortfall to the particular project.
This firm will
not terminate this project because of the monies they have
involved and the hope that they will eventually he able to sell
the product, if not in the United States, at least to export the
idea and product.
Firms dealing with military products approach research differently than
firms with commercial products.
The products researched are ideas that
are normally carried to a preliminary prototype phase of development.
42
These projects are then presented through proposals to branches of the
military or other branches of the U.S. Government.
The proposals are
intended to lead to contracts from the government allowing the
completi.on of the resear·ch and development..
These contracts are called
CRAD (Contract Research and Development).
·A company's internally sponsored research and development programs are
cnlled IRAD (Independent Research and Development).
The research and
development projects may be initiated as heing targeted at reseach
suggested by the Department of Defense (DoD). Another method for
selecting the project is to recognize the problem before the customer
r·ealizes he has a problem.
Hopefully, the prospective customer is
presented with a solution before he recognizes the problem.
Some firms
indicate that about thirty percent of the ideas generated for CRAD
p1·ojects are by the company and about seventy percent of the CRAD ideas
by the customer.
On exploratory or pure research projects,
approximately seventy percent of the ideas come from the company.
One
of the biggest problems in getting the research and development to a
successful conclusion is to align it to the customer's requirements and
needs and then to obtain a contract Hith the customer.
When the
.company vectors its ideas towards the customer's idea, it is more
probable to obtain a GRAD contract.
Generally, small firms do not attempt independent research directed
towards military products.
The reason i.s that they typically lack the
capital to invest in products that may take several years to develop.
43
The government and its agencies recognize the handicap of small
companies and are generally more willing to provide GRAD contracts to
assist these firms in developing research and development ideas.
QUESTION 3:
How does your termination criteria vary with different
.sizes of projects?
The answers to question 3 are summarized below:
o
If a large amount of resources have been consumed, the return
goals may be lowered.
Return goalz include:
1. Return on Investment.
2. Project/Product performance l'equirements.
3. Percent of market obtainable.
o
If completion dollars is small compared to dollars invested complete job (face saving).
The economic impact is the major· consideration when considering the
termination of a project.
If during the research and development
phase, a project manager determines that the competition had caught up
Hi th the new product or the market has changed, the status of the
project would be reviewed and determined whether more money should be
put into the project or to terminate the project.
During this review,
management may determine that the gross margin should be loHered on the
project.
The project may be far into the development and it would not
be beneficial to terminate the project and lose all of the money that
had been invested.
The project may be allowed to run to completion and
produce a product but at a lower margin to obtain a return on their
investment.
This continuation of the project may be the only method at
the time to maintain the technical work force.
If the project ·Here
terminated, the work force may be released if no other pr·ojects are
available.
The loss of the technical talent may be more devastating to
the firm than the loss of resourses caused by continuing the project
that has been determined to be a non winner.
l\.n example of a project that is overbudget and behind schedule is the
Space Shuttle Orbiter.
It is several billion dollars overbudget and
three yeat·s behind schedule.
Ho\-rever, one would not terminate it
because of the national pride. New technologies have been derived from
this program Hhich can make the project a success.
QUESTION 4:
How does the amount of money spent on a project influence
a termination decision?
The summary of the responses to question 4 follows:
o
No influence - Look to future - Not Past.
o
Has the payoff dropped?
o
Compare ratio of Spent vs To-go - No fixed ratio.
o
\fnat is the penalty clause?
o
LoHer project goals if large amount of money has been spent.
o
What is the gross margin?
o
If minimum spent - will terminate faster.
45
v!hen reviewing the cost to complete versus the resources consumed,
management will especially look at the projected overrun as the final
goal to obtain an adequate 1·eturn on their investment.
The amount of
money spent on a project does influence the termination decision.
If a
minimum amount of money has been spent on a job, it may be more easily
terminated than if a large pet·centage of the original budget has been
expended.
One firm indicated that the 2.mount of money spent on a project does not
influence the decision to terminate the project.
to the future, not to the past."
Their motto is: "look
This may be one of the reasons why
this particular corporation's division has been very successful.
HoHever, another division of the same
cor~poration
stated that the
amount of dollars they spend on a project does influence the decision
of ter·mination.
If a large amount of the resources planned have been
expended. the return on investment goals may be lowered if only a small
pereent.age of the resources will be required to complete the project.
_;I:nterview _Sur~mar~
Termination criteria of engineering research and development commercial
projects is largely based on economic considerations.
The return on
investment, the business plan and the product's market are the main
checkpoints.
The termination criteria are not specifically documented
or contained in procedures.
The criteria of termination may be more
de sc r i pti v e 1 y described as warn ing__sign_als________Jfueu___these-si-gP.-&1--3--a~e~------
46
received, a more indepth review must be performed in reviewing the
project.
These signals are summarized in the conclusion chapter of
this study.
CHAPTER 5
SUHHARY, CONCLUSIONS AND RECOt1MENDATIONS FOR FUTURE STUDY
Introduction
.This chapter presents a summary of the study, the conclusions drawn
from the analysis of the data and concludes with t·ecommendations for
future research.
The data from the questionnaire indicates that there are significant
findings between the success/termination results and:
o
The source of funding of the project.
More research and
development projects were successful Hhen they were funded by
contracts than when they were funded by tl1e firm's
o
o~,om
money.
More research and development projects were successful if
they were conducted by a staff comprised of all permanent
research and development personnel than those projects that
vwre not staffed by 100 percent permanent research and
dev~lopment
o
personnel.
The life span of successful projects tends to be shorter than
the life of terminated projects.
o
More research and development projects od.ginated by the
customer Here successful than the research and development
projects that were not originated by the customer.
47
The summary of the data from the intcrvietls indicates that the main
termination cr:l teria is based on economics.
analysis of the return on investment.
The economics include the
other projects maybe terminated
because there is a lack of funds because the project's priority was not
high enough to be above the funding cutoff.
Some projects Here
terminated because of a low DoD score and Hhich in turn alloHs the
company a Jower percentage of the research costs that were allmved to
be written off against
revenues.
Formal or informal criteria for the
termination of projects did not exist at any of the firms where
interviews were conducted.
Projects that are not selected for
continuance are terminated by default.
SeJ.12ction of engineering research a:1d development projects for
commer·d. al Pl'ojects were based on business plans t retur-n on
and the consumer market of the project.
products
HE.Te
tnve::.~tment.
The selection of military type
based on selling the product to the government.
The
military type project was generally pursued through the conceptual
and l.hen sold as opposed to the eommercial project which w2s
based on the funding the conc2pt through production.
the basis for the selections.
No fir·m
int~nds
Economics Here
to lose money.
Conclust.ons
This study has led to the conclusion that there are three basic types
of research and development projects.
These are:
1.
Contr·act n:'search and development.
2.
Research and development for military pr-oducts.
49
3.
Research
r~nd
development for commercial products.
The termination c:·i teria for these types of projects are different and
are summarized below.
o
Con tr·act research and development
Research and development performed under contract has the
highest ratio of success to termination.
If this type of
project was terminated, it was at the convenience of the
Ot"gani zati.on placing the contract.
The reasons given
included change of technical direction. low priority or lack
of funds.
o
Resecwch and development for military products
Prcjects that are initiated for military products ore set for
a definite goal and are company funded.
The projects are
int tiated to advance ideas through the feasibility stage.
The ideas and projects are then proposed to the miJ i tary and
further development will occur if a contract. is placed to
de';elop the idea or product.
He search and development
projects of this type are term:i.nated if a contract is not
obtained.
The criteria of term:i.nation for this type of
project is based primarily on the lack of success of
obtaining a contr::wt to further the v1ork.
o
Hesearch and. development for commercial products
The sole purpose of the existence of the research 3nd
development organization in the commercial product firm is to
increase the f:i.rm 1 s gr·owth and earnings.
The organization
50
improves and initiates products that Hill insure the firm 1 s
viable existence.
The research and development administrator
must be aware of the direction of the company as defined by
top management.
The adminis'l:rator must know and develop the
research and development organization to tr·ack management's
dir·ections.
In order to keep on target, he must continually
revieH the research and development program and select and
inl.tiate programs to achieve the company's goals and to
terminate programs that do not support the goals.
P r~;i3_(?~t:__li_~-~~~_c:_ti ~!!_Mode~
A;.> a t·esult of this study, the following research and development
project evaluation model was
developed~
This model is a tool to be
used periodically by fi t'ms Hhen evaluating resec:wch and development
projects that are in existence.
Each project should be evaluated by
several depad,ments within the organization including general
mnnagement. research and development managers, engineering, marketing,
and production.
These multiple scores may be averaged and given a
priority rating according to their scores from the model.
At this time
an in'v:;ger pt·ogramming technique could be implemented to determine the
optimum set of projects to be funded within the const1'2ints defined by
the company.
Pr·ojects not funded arc terminated by exclusion.
The nineteen factors utilized in this evaluation model are the salient
evaluation features noted during the interviews, from the significant
results of the questionnaire and from the review of literature.
The
factors \'i'ere grouped into four major categories:
Technical and Personnel.
as ordinal data.
Hanagernent, Econom:i.c,
The weight of each factor has been assigned
The score of each factor was assigned as a 0, 1 or 2.
The weight and score values may be adjusted after testing of the
algorithm has been performed to coincide with the priori ties of the
evaluator.
The Research and Development Project Evaluation Hodel is presented as a
task sheet.
See Figure 1.
Value
'Y~l1ere:
The algorithm on the model is:
(i))(\>Jeight (i))
1.
Value = the total weighted score of each project.
2.
Score (i)
= the
appraisal of each factor by the evaluator.
Each factor (factors are the nineteen evaluating criteria
which have been classified ]n four categories) is scored as
~
3.
--2, 1, or 0.
Weight (i)
= the
value assigned of the ranked importance to
ttJe firm for each factor.
This evaluation method of research and development p1·ojects alloHs the
factors Hith the most significant influence on the fin1' s \-Jell being to
have the most impact on the decision.
As a project's life becomes
longer, the credibility of factors becomes stronger· as they are based
more on past results and less on the projected future.
RESEAH.Cf. AUD DEVELOPHENT PROJECT EVALUA'l'IO:\ TASK sm:T
Froject Nrune
Date
No.
------
Score
(eire~~ !1'.os t correct)
----ye;; :.~iybc t:O-
1.
MAiiAGEl·iENT
1.1. Pt·oject related ta company goals.
1.2. Hanagement supportive of project.
1.3. Project in balance within R & D
I>rogram.
1.1!. De other affected departments
support pro,ject?
1. 5. Has market segment remained
same m: increased?
1. 6. Has the company's priority of
project re:uain('d sar:e or
increased?
J2.
E;:!OhOt,UC
I
2.1. Does projected ROI satisfy com-
'
pany goals and gui<lslines?
2.2. Company can provide scope of
fin-s.nce.
2. 3. Have levels of f1md.ing been
~de'J.ue.te?
2.4.
H<>i;ht
-------
16
2
2
1
0
0
13
2
1
0
1
2
1
0
3
2
1
0
8
2
1
0
4
2
1
0
19
2
1
0
17
2
1
0
14
0
1
2
11
2
1
0
9
2
1
0
18
2
2
l
l
0
0
6
lO
2·
l
0
15
2
0
l
0
2
12
1
0
1
2
5
2
1
0
1
1
Score X
Height
Rave cospetitive factors
i:tt~~rea.sed?
2.5. Are the projected production
cost.s reasonable?
3.
4.
TECHEICAI,
3.l. Is the probability of success
reasonable?
3.2. Cen the technolos:y be produced
cheaper than if J?U!'chased?
3.3. Is the project's goal practical?
3.4. Is the :project's schedule
-reasonable?
PERSON N'l<;I.
4 .1.
Co1-rect pe1·sonnel are assigned to
project.
lt. 2. Has project gro'.lp become stale?
4.3. FAYe key technical people left
proje.::t?
4.4. Are: project personr.el e:-;thu.siastic about the project's chance of
su.ccess,.
-----·-··----------------------'------.1------'-SCORE
2
I
--~----
__j
--------------------------·
FIGURE 1
RESEARCH AND DEVELOPHENT PROJECT
EVALUATION TASK SHEET
.5".)
Project Selection/Termination
The final decision to proceed with a project rests on the availability
of funds for support of the project.
Projects must be evaluated and
ranked to determine which projects are the most critical to the
organization~
A tool, such as the previously described Project
Evaluation Hodel and/or the intuitive evaluation of the research and
development director, ar·e to be used in establishing the ranking of
each old and new project.
Ne\<1 proposed
pl~ojects
and previously funded
projects must be evaluated with the same criteria in establishing their
ranl.dng.
These rankings must be merged into a common data set.
The
research and development funds are then allocated to the highest ranked
projects.
This sludy represents a thrust into the world of termination criteria
of engineering research and development projects.
recommended in the
follm~ing
Additional study is
area:
Pei'form e>n evaluation of this research and development evaluation
tool by enlisting several firms to use the tool during an
evaluation cycle.
The usage Hould validate the assumpU.ons of the
tool.
In summary, this study attempted to determine if firms do have criteria
for det8rrnining \-Then a project should be terminated.
been formalized on this subject.
i
·I
i
Very little has
The contact with the managers may
54
have started them thinking on this subject.
While this study only
scratched the surface. it is hoped that it Hill be the catalyst leading
to detail studies and the validation of this procedure for the
evaluation and criteria for determining Nhen a project should be
terminated.
SELECTED BIBLIOGRAPHY
55
BIBLIOGRAPHY
Barclay~
1.
S.; Bl'OHn, R. V.; Kelly, C. W.,III; Peterson, C.R.;
Phillips, L. D.; and Selvidge, J< Handbook for Deds_ion Analy~_?.:~.
Technical Repot·t TR 77-6-30. HcLean, Virgi.nia: Decisions and
Designs, Inc., September, 1977.
2.
Block, J. R. and Hadlow, G. E. "The Authority Relationships of
Contracting Officers in a Project/Program Hanagement Environmentu;
Air Force Inst. of Technology, Wright-Patterson AFB, Ohio, School
of Systems & Logistics, Report #SLSR-10-75A. January 1975.
3.
Block, Robert G. "R & D Resources Allocation -A Quantitative
Ai.d 11 ; -~search-D~velopment, Vol 23, No 8, Pg 20-24, August 1972.
4.
Brooks, T. L. upolicy Capturing of Management Personnel through
Project--SE:lection Decision Haking in an Air Force Research &
Development u; Air Force Inst. of Technology, Wright-Patterson
AFB. Ohio, School of Engineering, Repol"t ttAFIT/GStVStV79S-1.
September 1979
5.
Buede. Dennis H. 11 Decision Analysis: Engineering Science or
Clinical Art?"; Decisions and Designs, Inc., McLean, Va. November
1979
6.
Burgess, Halph E. 11 Criteria for the Evaluation of Industrial
Scientific Research", The Management of Scientjfic Tnlent, edited
by Jerome H. Blood. t·1anagement Report No. 76, .A.mer"ican-r·la.nagement
Association. New York, 1963.
7.
Bm~te,
8.
Carney, Thomas P. 11 An Optimization Hodel for Investi gc;Un5
Alternative Research & Development Programs of the U.S. ArmyH;
Naval Post-Graduate School, t1onterey, Ca. September 1971.
9.
Chakrabarti, A. K. and Rubenstein, A. H. "Interorganizational
Transfer of Technology: A Study of Adoption of NASA Innovati.onsn;
IEEE Transaction on Eng. Management, Vol. EM-23,No. 1, February
Harris M. "Project Initiation & Termination: Moments of
Deci sion--Surr.mary of a Symposi.um"; Journal_'?.£_ Heta}~c;_ t·larch 1970.
1976-;-p·p~
10.
11.
20-:j--4:-----·
Clarke, Thomas E. 11 Decision-Haking in Technologically Based
Organizations: A Literature Survey of Present Practice", ].EEE
Transacttons on Engineering HanC1.~~men~. Vol. EM-21, No. 1,
FeSI:t1ary--T97 4.
Coop(~r,
Martin J. "Evaluation System for Project Selection",
Volume XXI/No. 4, July 1978.
y.esea~ch_~~ncgement,
56
12.
Dean, Burton V. "Evaluating, Selecting and Controlling Research
and Development Projects 11 ; American t'lanag::_~<:;nt Assoc.·-~ AMA
Research St~dy 89, 1968.
"13.
Dobbins Jr., E. B. "Research and Development Project
Prioritization - An Annotated Bibliography"; Army l1issile Command,
Redstone Arsenal, Al. Technology Integration Office, Report
#DRSMI/RN-80-1. April 1980.
14.
15.
English, J. Morley; Cost Effectiveness -The Ecomonic Evaluation
of E_nginee_!'_~d Systems, John Wileyand--Sons, Inc-~-.-,-968-------
Fish, Howard C. and Wilson, J. W. "A Survey of Industrial Research
& Development Budgeting, Effort Selection & Evaluation 11 ;
Naval
Post-Graduate School, HonteJ"ey, Ca., Report iiNPS-54CF76092.
September 1976.
16.
Flinn, R. A. and Turban, E.; "Decision Tree Analysis for
Industcial Research", Research ,~lanagement, Vol. XIII, No. 1,
Je:muary 19~(0.
17.
Gersten feld, Arthur: 11 A Study of Successful Projects, Unsuccessful
Projects, and Pr·ojects h1 Process in Hest Ger·many" IEEE
Transactions on Eng. Jla!!_ageme!lt, Vol. EM-23, No. 3 P.ugust 1976.
,,
1o.
Gerstenfeld; Turk, C.; Farrow, R,; and Spoteer, R.; 11 t1arketing and
R & D", Research Management, Volume XII, No. 6, pp. 409-412,
November·~-1969. _ _ _ ,_____ _
19.
Holzmann, Richard T.; 11 To Stop o1· Not-· The Big Research Decision 11
Chemtech Vol. 2 - Feb. 1972~ pp. 81-89.
20.
Holtn:;rd, R. A.; rrDecision Analysis as a Professional Decision~
Making Tool", Readings in Decision Analysis. Stanford Research
InstHute, Menlo
Decision Analysis Group, 19714.
Park-:--californi a;
21.
Longmore, H. J.; "Improving Research and Development Decisionfor the User" Army Har College, Carlisle Barrack, Pa. Hay
1974
~·'laking
22.
t1cGeehan, T. J.; "Decision Analysis Technique for Program
Evaluation (Goal Programming) 11 ; Averbach Assoc., Inc.,
Phil adelphi a, Pa., Report //AUER-·2325/2326-TR-ll. April 1977.
23.
Ivioore, Laurence J.; Taylor III, Bernard W.; "Hultiteam, Hultiproject Researd. and Development Planning vli th GERT 11 ; Nanagement
~cience, Vol. 24, No. 1+ December, ·1977.
-------
24.
Hurphy. J. F.; "An Exclusion Approach to Selecting and Terminating
Research Projects"; Research l'lanagement, Vol. X No.2 1967.
53
25.
Hyers, J. E.; "Decision Hilestone Criteria: The Small USAF
Development and Acquisition Program 11 ; Air War College, Maxwell
AFB, Ala. Report /15363. April 1974.
26.
Nie, Norman H; Hull, C. Hadloi; Jenkins, Jean G.; Steinbrenner,
Karin; Bent, Dale H.; ~PSS: -~t~tisti_cal Package for the Social
Sciences_, Second Edition, lvlcGraw-Hill Book Company, 1975.
27.
Quilter, H. K.; "Problems of Research and Development Cost
Predictions in an Avj_onics Supply Co" Pr:~ceed~nB_~_of Methods of
~ost Predic~on~ymposium, London, England, t-larch, '1977.
28.
Quinn, J. L.; 11 Research and Development Appraisal and Evaluation"
Air Force Inst. of Technology, \-Iright-Patterson AFB, Ohio, School
of Systems and Logistics, Report #SLTR-5-70. November 1970.
29.
Random House Dic_tionary of the English Language, Random House,
1970.
30.
Reimherr, G.; "Decision t-laking in Hanagement - A Bibliography with
Abstt'actsH; National Tech. Info. Service, Springfteld, Va. June
1980.
32.
Seligman, D.; "t1cNamara' s Hanagernent Revolution,"; Fo!:_tune July,
1965.
33.
Trozzo, Charles L. , nnesc ription & Critique of Quantitative
t'1ethods for the Allocation of Exploratory Development Resources";
Institute for Defense Analyses, Arlington, Va., Science and
Technology Div., Report #P-731. Hay 1972.
34.
Weschler and BroHn ( eds.) "Evaluating Research and Development";
Annotated Proceedings of a Conference of Reseat·ch Administrato:·s
held at UCLA, May 10, 1952.
35.
Yershov, Y. V. and Shcheddna, T. I. "The Selection of Nost
Promising Alternatives for Research & Development Projectsn;
Materialy po naukovedeniyu, Issue 10, 1969. Kiev, pp. 14-26; Army
Foreign Science Technology Center, Charlottesville, Va., Report
#FSTC-HT 23 0819-73.
APPENDICES
59
APPENDIX A
Letter of Transmittal
19646 Romar Street
Northridge, Ca 91324
February 24, 1981
<TITLE>
Sir:
I <1m a grac!u3te studA~1t at California State University-Northridge applying for
the d,:gre:>. of' rl.'3::>t,;r· of :~;:ience ir: Engineerir.g. V.y tbesis <ldvisor· is Di·. /wnold
Roe.
The thesis topic is "Rationille of. Enginect·ing R•;,senrch and Development
Decisi,on :'laking". I ~,·ant you to help me by completing the enclosed questionnaire
fol' at. le~:st two projects.
Over 500 paper·s cmd ;:wticles bave been published cuU.ining nethod:.: and criteria
for initiati~g research prcjects. Information is putltshed defining how to begin
projects. and meth.)ds t.a control and continue projects; hov 1eve1·, ve1·y lit.tle
discussion has been presented defining criteria used to ~etermine when a project
was co~:plete<l, to be reo;ganized, conbined or tennu1ated.
The evaluotion cf tLe
respor.ses to this qtEestinr.nc.i.re will be used to gair. ins.i.gl\t on manager·s' dzt.d.s1.c,ns '.m whether or ne;t to terminate engineering res-:arch pr·ojects.
Your input to this survey is ardently sought and answers to the quc~s!-.icnn;~.ire <.~re
essent.i al to my study. Please send me your completed form as soon a~, you can.
Bany tht<:1 :.::s.
Yours truly,
E. F. Heiman
Enclosure:
Questionnaire
SAE
P.S.
Feel free to keep or toss the enclosed bj_bliography on the topic on my
'thesis.
(,
60
APPENDIX B
Questionnaire
QUESTIONNAIRE ABOUT ENGI:\EERING RESEAr'CC!! A..\ID DEVELOPMENT PRCJECI'S
(Please place the letter of your response or wri.te in the requested inforwation on Data Sheet
for each Project.)
1.
Enter Project name or Produet:
(optional}
2.
This project was classified as:
.A. Commercial (Consumer)
B. Defense (Hilitary)
C. Non-Profit (Service)
3,
The Project was performed in a:
A. Separate Research and Development Facility
B. Engineering Area
C. Production Area
D.
4.
O~her
The I'rvject was initi:ot:ed for the following reasons:
(Select ONE primary and all applicable secondary reasons)
A.
B.
C.
D.
E.
1".
G.
H.
5.
Search for new kao1,oledge
New Project (Venture)
Hodernizing old idea
Enn::>.ncro an ide-:\ or product
Reduce labor- cost
Redur:e material cost
Safet:y
Reduce pollution
This projeet was directed
I.
J.
K.
L.
M.
N.
0,
P.
Conserve energy
Self-Defense (~ati~nal)
Offensive weapons
Direction from a higher level of authority
Profit motive
l~"l obvious market
Cost savin9.,s other than labor or Material
Other, specify
tm~ard:
A.
Pure and basic research.
L'e finition: Pure research is a fundamental i.nqui ry. Basic research .investigates
broad questions and s.oeks anslvc-:rs that may be applicable.
B.
Applied Research
Dcfin"i.tion; Applied research studies a specific question or problem area.
C.
Devcloprrlent.
Definit.ion: Develop~1ental resem:ch is the last stage when incorporating more
basic and applied rese<nc:h findings into operational reality.
6.
'\oihat percentage of the funding for this project was internal?
7.
Hhat
8.
What was the
~~as
the total direct cost of the project (in
i:_f!rcentEz~
(Non·-contract funded)
tho~ts::cnd£)?
of each type of personnel ass:i.gr.ed to this project?
Type A - Hired and trained for the specific project.
Type B - Assigned as a result of past performance in other areas.
Type C
Assigned frm:> permanent Research and Development staff.
Type D - Hired to acquire specific technological capability.
Type E -Other, specify
61
2 9.
'lo,'hich organizc\tions P<ll:ticipated i.n the Project selection decision, Proj">ct. revie;ts,
nnd i:'roject ecr..;>letion decision?
(Select Ol,;E pr-:.'.mary and .sll applicable secondary participants for each aetivity)
A.
B.
Consult:tnts
Nanagement
E.
F.
D.
Hanufo.cturing
1·brketic. . g
G.
c.
H.
~roj ect:
10.
Row \r,any Yf"ars did this
11.
i-lho origJna Led the idee. for this project:
A~
B.
C.
D.
12.
Non R
& D Eng:l.neering
R & D Engineering
R & D Hanagement
Other, specify
exist?
Top !.fanagr~m~nt
H:lddle ~iana::;ement
Superv:!.sors
E.
Customer (user)
F.
G.
Don't know
Other
Engineers/~:c·l2.ntists
Was this !-'reject a SUCG:::SSFUL project or a TERHINATED prc-.jec.t?
Enu:r S or 'f
A SUCCES~oFlTL proj<:ct is ODe that m;:ct the project goals and r,.:rE> clessifi_cd as
comp letcd. This p1·oj ect may or m<~y not have been used or applied.
A T.t:f~NINATED p ~oj E:Ct is clef ined as or.c that ,..raB reorganj zed, cor.;b in ad, ab3ndtJn~~d,
or unsucces~;[ul.. Tl:-.e terminat~d ~roject dili not meet the ori.£:.nal goals end the!
result.'> -..,e\·c not applicable (usable) to your company products. This project !':ay
have been halted because of a l<::ck of adequate funding.
13,
Hhy was the project tenni:1ated?
A.
B.
-C.
D.
E.
l'.
14.
(If successful - enter "sue")
Changed taanagement
Company's goals cl'.aoged
~Lack
of funds
L<:l(:;< of personnel
ObsoletE: idea
Potential paycff is too low
G.
H.
I.
J.
K.
L.
l'roject leader quit
Slow progress
Sold project
Technical probler.8
I.ovl DoD rating
Other, specify
If you have any thoughts on ''criteria ti'nni.natioa" and/or differences bet~'een your
"selection ·c:clteria" and "termination crHeria," please put your comments ou ri-,e
da.ta sheet.
APPENDIX C
Data Sheet
AN. ENGINEERING RESEARCH AND
QUESTIO~:.:AIRE
14.
Terminati.on criteria:
DEVELOP~XNT
PROJECT REVIE\1
DATA SliEET
(For additional space, use reverse sid'"-)
63