InnoSkills – Innovation Skills for SME´s
5. Evaluation of innovative solutions
5.1 Decision Making Process
----------------------------------------------------------------------------------------------------------------------------------------------------------------
5
Evaluation of innovative solutions
5.1 Decision making process
Keywords:
Decision-making process and analysis, Decision making under risk and uncertainty, Multicriteria decision making, Goals, Alternatives and criteria.
This module introduces you to the company’s decision-making process.
You will learn the basics of:
• the process and steps involved in making a decision
• the criteria for evaluating the alternatives
• the models and tools used for the selection of the optimal alternative
As a result, you should be prepared to organise a decision making process and to select
the best possible solution. The estimated time to go through this module is about 50
minutes; additional time will be required to study the examples in the appendix.
Introduction
Both in everyday life and in business environment we often have to choose between
alternative actions.
• Search for innovative ideas and solutions can result in many different ideas and
suggestions. How to deal with them at this point?
• From several available alternatives, which one to select for realization, considering
the incomplete information and approach to risk?
• If you have resources for several innovative projects, how to select the appropriate
product portfolio?
We have to make clear and transparent decisions. In the following, we will present some
techniques which can help you.
This Module is based mainly on literature, especially on Baker D. et al., Guidebook to
Decision-Making Methods, 2001 (see Bibliography for the complete information).
5.1.1 What is a decision-making process?1
Good decisions can best be reached when everyone involved uses a clearly defined and
acknowledged decision-making process. A clear decision process depends on asking and
1
This part has been taken from Baker D. et al., Guidebook to Decision-Making Methods, page 1.
© InnoSkills
1/18
2008-06-12
InnoSkills – Innovation Skills for SME´s
5. Evaluation of innovative solutions
5.1 Decision Making Process
----------------------------------------------------------------------------------------------------------------------------------------------------------------
answering enough questions to ensure that there are no significant omissions and all the
relevant factors are taken into account.
This module provides:
• Description of an eight step decision-making process.
• Introduction to specific decision methods.
• Examples of the specific decision methods in action (Appendix 5.1-A.xls).
5.1.2 Why use a decision-making process?2
For most of familiar everyday problems, decisions based on intuition can produce
acceptable results because they involve few objectives and only one or two decisionmakers. In business environment the problems are usually more complex. Most decisions
involve multiple objectives, several decision-makers, and are subject to external review. A
well-organized decision-making process employing credible evaluation methods will
provide:
• Structure to approach complex problems
• Basis for decisions
• Uniformity in the decision making process
• Objectivity
• Documented assumptions, criteria, and values used to make decisions
• Decisions that are repeatable, reviewable, revisable, and easy to understand
Using such approach can help avoid misunderstandings that can lead to disputes about
the validity of the analyses, and eventually slow down the whole process. Its use will set a
baseline for continuous improvement in decision making in your company.
5.1.3 Where and when should a decision-making method be used?3
The decision-making methods described in this text are readily applicable to a wide range
of decisions, from simple ones, as picking a restaurant for a special meal, to those that are
made complex by interdepartmental or inter-company interfaces. Use of this decisionmaking process and supporting methods is recommended any time decisions:
• require many reviews at different management levels,
• involve more than one project,
• affect new or redirected funding,
• require approval for new facilities or upgrades to existing facilities,
2
3
This part has been taken from Baker D. et al., Guidebook to Decision-Making Methods, page 1
Ibid
© InnoSkills
2/18
2008-06-12
InnoSkills – Innovation Skills for SME´s
5. Evaluation of innovative solutions
5.1 Decision Making Process
----------------------------------------------------------------------------------------------------------------------------------------------------------------
• have alternatives with high technical risk,
• have alternatives that appear equally feasible,
• require a decision to revise or cease a project,
• have impact mainly in the future,
• involve multiple or competing drivers,
• define data that will support future decisions.
In a few words, this module should be followed any time a clear, efficient and
understandable decision is desired.
5.1.4 How to use a decision-making process?
We will start by defining the decison-making process, then we will describe some specific
classes of decision problems together with description of available tools.
5.1.4.1 Decision making process4
Main concern in making a decision is to establish who are the decision-maker(s) and
stakeholders in the process. Identifying the decision-maker(s) early in the process reduces
disagreement about problem definition, requirements, goals, and criteria.
When appropriate, stakeholders should also be consulted. By acquiring their input during
the early steps of the decision process, we can get useful feedback before a decision is
made.
Figure 1 shows the steps in the decision-making process. The process flows from top
to bottom, but may return to a previous step from any point in the process when new
information is discovered. It is up to the decision team’s job to make sure that all steps of
the process are adequately performed.
In more complex decision, you should ask skilled and experienced analysts / facilitators to
help assist with all stages of the decision process. Expert facilitation can help assure that
all the steps are properly performed and documented. Their experience and expertise will
help provide clearness to the decision making process and help avoid misunderstandings
that often lead to questions about the validity of the analyses, which can slow down the
progress.
4
This part has been taken from Baker D. et al., Guidebook to Decision-Making Methods, pages 2 - 5
© InnoSkills
3/18
2008-06-12
InnoSkills – Innovation Skills for SME´s
5. Evaluation of innovative solutions
5.1 Decision Making Process
----------------------------------------------------------------------------------------------------------------------------------------------------------------
Figure 1: General Decision–Making Process5
Step 1: Define the problem
Problem definition is the crucial first step in making a good decision. This process must, at
least, identify root causes, limiting assumptions, system and organizational boundaries
and interfaces, and any stakeholder issues. The goal is to express the issue in a clear,
one-sentence problem statement that describes both the initial conditions and the
desired conditions. This step should result in a written problem statement about how the
problem is going to be solved before proceeding to the next step.
The key to developing an adequate problem statement is to ask enough questions about
the problem to ensure that the final report will clearly answer the questions of reviewers
and stakeholders (see Figure 2 below). When stakeholders are involved, it may be
appropriate to have them review the problem statement in its initial and desired state to
provide an external check before requirements and goals are defined.
Figure 2: Problem definition6
5
6
Baker D. et al., Guidebook to Decision-Making Methods, page 2
Ibid, p.3
© InnoSkills
4/18
2008-06-12
InnoSkills – Innovation Skills for SME´s
5. Evaluation of innovative solutions
5.1 Decision Making Process
----------------------------------------------------------------------------------------------------------------------------------------------------------------
Step 2: Determine requirements
Requirements are conditions that any acceptable solution to the problem must meet.
Requirements spell out what the solution to the problem must do. For example, a
requirement might state that a process must produce at least ten units per day. Any
alternatives that produce only nine units per day will be discarded. Requirements can be
also external, given e.g. by legislation and standards (health and safety, environmental
issues, etc.)
Requirements that don’t discriminate between alternatives need not to be used at this
time.
The cooperation of decision-makers with experts in operations, maintenance, environment,
safety, health and other technical disciplines typically provide a definition of requirements
that a viable alternative must meet.
Step 3: Establish goals
Goals are broad statements of intent and desirable values. Examples might be: reduce
worker fatigue, lower costs, lower public risk, etc. Goals go beyond the minimum essential
must have’s (i.e. requirements) to wants and desires. Goals should be stated positively
(i.e. what something should do, not what it shouldn’t do). Because goals are used for
selection of optimum alternatives, they are developed prior to alternative procedures.
Some goals may be conflicting, but this is neither unusual, nor cause for concern. During
goal definition it is not necessary to eliminate conflicts among goals or to define the
relative importance of the goals. The process of establishing goals may suggest new or
revised requirements or requirements that should be converted to goals. In any case,
understanding the requirements and goals is important for defining alternatives.
Step 4: Generate alternatives
Alternatives offer different range of actions to transform the initial state into the desired
one. The decision team evaluates the requirements and goals and suggests alternatives
that will meet the requirements and satisfy as many goals as possible.
Generally, the alternatives differ in their ability to meet the requirements and goals. Those
alternatives that do not meet the requirements must be deleted from further consideration.
If an alternative does not meet the requirements, three actions are available:
1. The alternative is discarded
2. The requirement is changed or eliminated
3. The requirement is restated as a goal
The description of each alternative must clearly show how it solves the defined problem
and how it differs from the other alternatives. A written description and a diagram of the
specific functions performed to solve the problem will be useful.
© InnoSkills
5/18
2008-06-12
InnoSkills – Innovation Skills for SME´s
5. Evaluation of innovative solutions
5.1 Decision Making Process
----------------------------------------------------------------------------------------------------------------------------------------------------------------
Step 5: Define evaluation criteria based on goals
Usually one alternative will not be the best for all goals, therefore alternatives must be
compared each other. The best alternative will be the one that achieves most goals.
Decision criteria which discriminate among alternatives must be based on the goals. It is
necessary to define discriminating criteria as objective measures of the goals to measure
how well each alternative achieves the project goals.
Each criterion should measure something important, and not depend on another criterion.
Criteria must discriminate among alternatives in a meaningful way (e.g., if the colour of all
alternatives is the same or the user is indifferent to the colour selection, then colour should
not be a criterion).
Criteria should be:
• able to discriminate among the alternatives,
• complete – include all goals,
• operational – meaningful to the decision maker’s understanding of the implications of the
alternatives,
• non-redundant – avoid double counting,
• few in number – to keep the problem dimensions manageable.
Using a few real discriminators will result in a more understandable decision analysis
product. However, every goal must generate at least one criterion. If a goal does not
suggest a criterion, it should be abandoned.
Several methods can be used to facilitate criteria selection.
Brainstorming: Brainstorming may be used to develop goals and associated criteria.
Brainstorming is discussed in module 4.2.
Reverse Direction Method: Team members consider available alternatives, identify
differences among them, and develop criteria that reflect these differences.
Previously Defined Criteria: End users, stakeholders, or the decision-maker(s) may
provide criteria.
Input from the decision-maker(s) is essential to the development of useful criteria.
Moreover, the decision-maker’s approval is crucial before the criteria are used to evaluate
the alternatives.
Step 6: Select a decision-making tool
Later on in this module we will introduce and describe the following widely employed tools:
• Decision making under risk and uncertainty
• Pros and Cons Analysis
© InnoSkills
6/18
2008-06-12
InnoSkills – Innovation Skills for SME´s
5. Evaluation of innovative solutions
5.1 Decision Making Process
----------------------------------------------------------------------------------------------------------------------------------------------------------------
• Kepner-Tregoe Decision Analysis (K-T)
• Analytic Hierarchy Process (AHP)
There are some other methods, which are more complicated and difficult to apply. The
method selection needs to be based on the complexity of the problem and the experience
of the team. Generally, the simpler the method, the better. Be as simple as possible (but
no more)7. More complex analyses can be added later if needed.
Step 7: Evaluate alternatives against criteria
Alternatives can be evaluated with quantitative methods, qualitative methods, or any
combination. Criteria can be weighted and used to rank the alternatives. Both sensitivity
and uncertainty analyses can be used to improve the quality of the selection process.
Experienced analysts can provide the necessary detailed understanding of the mechanics
behind the chosen decision-making methodology.
Step 8: Validate solution(s) against problem statement
After the evaluation process has selected a preferred alternative, the solution should be
checked to ensure that it truly solves the identified problem. Compare the original problem
statement to the goals and requirements. A final solution should fulfil the desired state,
meet requirements, and best achieve the goals within the values of the decision makers.
Please stop and think: What decision have you recently faced in your company?
Try to compare your decision-making process in accordance with Figure 1. Did
you find any differences between your usual way of decision-making and this
formalized process?
5.1.4.2 Decision analysis8
Decision analysis is a systematic approach in order to solve decision problems optimally
under conditions of uncertainty. It does not describe how or why an individual makes a
decision; it prescribes a decision procedure for the individual that is consistent with his or
her preferences and attitudes toward risk. You might be asking yourself: „Why do I need to
approach the decision making process with a scientific method, when I know most
decisions (business and otherwise) are made on intuitive level?“ The answer is threefold:
7
8
A. Einstein: „Make things as simple as possible, but not simpler.“, http://en.wikiquote.org/wiki/Albert_Einstein
Part of this section (p. 8-11) adapted from: McClave J.T., Benson P.G.: Statistics for Business and Economics, p. 1041-1052
© InnoSkills
7/18
2008-06-12
InnoSkills – Innovation Skills for SME´s
5. Evaluation of innovative solutions
5.1 Decision Making Process
----------------------------------------------------------------------------------------------------------------------------------------------------------------
1. Yes, it is true that the vast majority of decisions made in business does not require, and
are made without, formal analysis. But for that one crucial decision upon which
“everything depends”, it is very helpful to have a systematic, logical decisions
procedure to follow.
2. Most of us have little experience in intuitive processing of the probabilistic and sample
information that may confront us in a complex decision-making problem. Consequently,
it is frequently more profitable to rely on the mechanical approaches of decision
analysis than on the less reliable information-processing capabilities of our intuition.
3. To use decision analysis, we are forced to consider carefully and logically all possible
courses of action and the outcomes that could result from each. By doing so, we may
see a side of the problem not seen before, or we may even discover that we have been
addressing the wrong problem. Thus, the information obtained from decision analysis
may widely compensate for the effort expended in the analysis.
One of the alternatives we face in making a decision is whether the decision should be
made now – using information we already have on the problem (we will refer to this as
prior information) or postponed until we have gathered additional (sample) information.
Although all decision problems involve the selection of a course of action chosen from
among two or more alternatives, we can classify them into one of three categories:
1. Decision making under certainty
2. Decision making under uncertainty and risk
3. Decision making under conflict
Decision making under certainty entails the selection of a course of action when we
know the result each alternative action will lead. If the number of alternatives being
considered is small, such decisions may be easy to make. However, if the number of
alternatives is large, the optimal decision may be difficult – if not impossible – to obtain. It
may take too much time and/or be too costly to evaluate all the alternatives and select the
one with the most favourable results. Many problems of this type can be solved using the
operational analysis methods (linear programming, queuing theory …) and will not be
addressed in this text. In another class of problems we must evaluate the alternatives
according to multiple criteria (often controversial). Some methods for solving multi-criteria
decision making problems are introduced later in this module.
It is important to note that “certainty” is often only theoretical concept and is very rare in
practical problems, It is therefore important to analyze the sensitivity of results to changes
in inputs and to evaluate potential risks.
© InnoSkills
8/18
2008-06-12
InnoSkills – Innovation Skills for SME´s
5. Evaluation of innovative solutions
5.1 Decision Making Process
----------------------------------------------------------------------------------------------------------------------------------------------------------------
Decision making under uncertainty and risk entails the selection of a course of action
when we do not know with certainty the results that each alternative action will lead.
Furthermore, we assume that the outcome of whatever course of action we select is
affected only by chance and not by opponent or competitor.
Decision making under conflict is similar to decision making under uncertainty and risk,
in which we do not know with certainty the result of each alternative action. However, the
reason for this uncertainty is different: we are “playing against” one or more opponents or
competitors. The outcome of our chosen course of action depends on decisions made by
our competitors. Decision problems of this type fall under the discipline known as game
theory and are not discussed in this text.
Decision making under uncertainty and risk
Four basic elements common to decision problems involving uncertainty are:
• Actions: the set of two or more alternatives the decision maker has chosen to consider.
The decision-maker´s problem is to choose one action from this set.
Example: to start new product development or not
• States of nature: the set of two or more chance events upon which the outcome of the
decision-maker´s chosen action depends.
The states of nature should not overlap, so that they are independent and the probability
of the sum of events is equal to the sum of probabilities. They should be complete in the
sense they cover the whole spectrum of possibilities; completeness ensures that the sum
of probabilities of states of nature equals to 1.
Example: the intervals of expected market share for the new product based on market
research (0-5%, 5-10%, 10-20%, more than 20%)
• Outcomes: The set of consequences resulting from all possible action / state of nature
combinations.
In general, outcomes reflect the actual reward to the decision maker in terms of payoff.
Alternatively, outcomes can be expressed in terms of opportunity loss, i.e. the difference
between the payoff a decision-maker receives for a chosen action and the maximum that
he/she could have received for choosing the action yielding the highest payoff for the
state of nature that occurred.
Example: The expected payoff if we select to develop the new product and the market
share will be 5-10%.
• Objective variable: the quantity used to measure and express the outcomes of a decision
problem.
© InnoSkills
9/18
2008-06-12
InnoSkills – Innovation Skills for SME´s
5. Evaluation of innovative solutions
5.1 Decision Making Process
----------------------------------------------------------------------------------------------------------------------------------------------------------------
Decision making under risk
Whenever possible, we attempt to determine (or estimate) the probabilities of states of
nature. In this case, we are talking about decision making under risk.
The problem can be formulated in terms of payoff table: the rows of the table represent
actions, the columns represent the states of nature and the elements of the matrix are the
outcomes for the combination of action / state of nature. Let us illustrate the model by
example:
Example 1: Beer producer intends to expand and build a new brewery. The problem is to
determine how large a brewery is to be constructed. It has been decided that the size
should be based on the planned gross profits for the fifth year of operation. The marketing
department forecast is that the company cannot obtain more than a 15% market share
during the 5-th year of operation and the probabilities of market shares are 0,4 for market
share 0-5%, 0,5 for market share 5-10% and 0,1 for market share 10-15%). The analysis
resulted in the following payoff matrix (payoffs in millions of €):
ACTION
(brewery size)
pi : probability
a1 : small
a2 : medium
a3 : large
STATE OF NATURE
Market share during fifth year of operation
S1 : 0 – 5% S2 : 5 – 10% S3 : 10 – 15%
0,4
0,5
0,1
3
3,5
4,5
2,5
7
8
2
6
10
EXPECTED PAYOFF
EP(ai)
3,4
5,3
4,8
The most commonly used criterion is the expected payoff: the decision-maker selects the
action that produces the maximum expected payoff. In terms of probability theory, the
random variable x is payoff and the associated probability is the probability of occurrence
of the state of nature. Then the expected (mean) value of x is
E(x) = Σx p(x)
The values of expected payoffs are calculated in the last column of the payoff table and
according to the expected payoff criterion the decision-maker selects action a2 , i.e.
recommends to build the medium-sized brewery.
Please stop and think:, Do you attemp to estimate the probabilities of market
success, before you decide on starting a new product development? Do you
evaluate the reliability of your estimates? Do you think some additional
information can improve the reliability of your predictions?
© InnoSkills
10/18
2008-06-12
InnoSkills – Innovation Skills for SME´s
5. Evaluation of innovative solutions
5.1 Decision Making Process
----------------------------------------------------------------------------------------------------------------------------------------------------------------
Decision making under uncertainty
If the decision maker does not want or is not able to determine probabilities of the states of
nature, he/she faces decision making under uncertainty. There are several criteria that do
not require the assignment of probabilities in order to reach a decision. Two common
nonprobabilistic criteria are the maximax and maximin decision rules.
To use the maximax rule, determine the maximum payoff associated with each action and
choose the action that corresponds to the maximum of these maximum payoffs. This
criterion ignores all information except the maximum value; it is optimistic, as it is based on
the assumption that the most favourable state of nature will occur. In our brewery example,
maximax criterion would select action a3 – large brewery.
To use the maximin rule, determine the minimum payoff associated with each action and
choose the action that corresponds to the maximum of these minimum payoffs. This
criterion ignores all information except the minimum payoff for each action, therefore it is
pessimistic. In our brewery example, maximin criterion would select action a1 – small
brewery.
Utility, risk attitude
The above mentioned criteria sometimes fail to provide decisions compatible with the
decision-maker attitude toward risk. An objective variable, reflecting this attitude, is called
the utility function and the values assigned to the outcomes are refereed to as utilities.
Utility function assigns to payoff values the values of utilities, usually from interval <0,1>.
As the detailed explanation of this often non-intuitive approach is beyond the possibilities
of this introductory text, we refer the interested reader to corresponding literature.9
Please stop and think: What is your approach to risk? How it influences your
decisions? Try to compare your risk attitude with that of your colleagues, but
also with that of other entrepreneurs. How much do you think the risk attitude is
influenced by eduaction, culture and simile factors?
Decision trees
Decision trees are very useful tools for the decision-making under risk. In principle, any
payoff matrix can be converted to the decision tree; however the decision trees may be
used for much broader range of problems, especially to multi-phase projects, which are
typical for R&D and innovation projects that are usually performed within the framework of
multi-stage process (typical representative being the stage-gate process of R. Cooper10)
with some probablities of success. Again, the detailed explanation of this method goes
beyond the possibilities of this introductory text and we refer the interested reader to
9
Baker D. et al., Guidebook to Decision-Making Methods
10
G. Cooper: Winning at new products
© InnoSkills
11/18
2008-06-12
InnoSkills – Innovation Skills for SME´s
5. Evaluation of innovative solutions
5.1 Decision Making Process
----------------------------------------------------------------------------------------------------------------------------------------------------------------
corresponding literature.11
Decision making using sample (a posteriori) information
Sometimes it may be possible and useful to postpone the decision until we have gathered
further information that can be used to revise prior probabilities. Using this approach, it is
possible to estimate the maximum expected worth of the sample information before
actually obtaining the sample, i.e. we can determine the maximum amount we should be
willing to pay for such information. As in the above paragraphs, we have to refer the
interested user to specific literature.12
5.1.4.3 Multi-criteria decision making13
In decision-making under risk and uncertainty we evaluate the alternatives according to
one criterion only. In many practical applications we must evaluate the alternatives against
more criteria, sometimes controversial, often of differing importance expressed by their
weights.
The evaluation methods introduced here are adaptable to many situations, as determined
by the complexity of the problem, needs of the customer, experience of the decision
team/analysts/facilitators, and the time and resources available. No decision making
method is appropriate for all decisions.
The examples provided in Appendix 5.1-A.xls are intended to facilitate understanding and
use of these methods.
Pros and Cons analysis
Pros and Cons Analysis is a qualitative comparison method in which positive aspects
(pros) and negative aspects (cons) are identified for each alternative. Lists of the pros and
cons, based on the input of subject matter experts, are compared each other for all
alternatives. The alternative with the strongest pros and weakest cons is preferred. The
documentation regarding the decision should include an exposition, which justifies why the
preferred alternative pros are more important and cons are less significant than those of
the other alternatives.
Pros and Cons Analysis is suitable for simple decisions with few alternatives (2 to 4) and
few discriminating criteria (1 to 5) of approximately equal value. It requires no
mathematical skill and can be implemented rapidly.
11
12
13
C. W. Kirkwood, Decision Tree Primer
J.T.McClave, P.G.Benson: Statistics for Business and Economics, chap. 19
This part has been taken from Baker D. et al., Guidebook to Decision-Making Methods, pages 6 - 8
© InnoSkills
12/18
2008-06-12
InnoSkills – Innovation Skills for SME´s
5. Evaluation of innovative solutions
5.1 Decision Making Process
----------------------------------------------------------------------------------------------------------------------------------------------------------------
Kepner-Tregoe (K-T) decision analysis
K-T is a quantitative comparison method in which a team of experts numerically score
criteria and alternatives based on individual judgements/assessments. In K-T method each
evaluation criterion is first scored based on its relative importance to the other criteria (1 =
least; 10 = most). These scores become the criteria weights (see K-T example in Appendix
5.1-A.xls).
When the time comes to evaluate the alternatives, the alternatives are scored individually
against each of the goal criteria based on their relative performance.
A total score is then determined for each alternative by multiplying its score for each
criterion by the criterion weights (relative weighting factor for each criterion) and then
summing across all criteria. The preferred alternative will have the highest total score.
K-T Decision Analysis is suitable for moderately complex decisions involving a few criteria.
The method requires only basic arithmetic. Its main disadvantage is, for istance, that it
may not explain how much better a score of “10” is than a score of “8”. Moreover, total
alternative scores may be close together, making a clear choice difficult.
Analytic Hierarchy Process (AHP) – Saaty´s model
AHP is a quantitative comparison method used to select a preferred alternative by using
pair-wise comparisons of the alternatives, based on their relative performance against the
criteria. The basis of this technique is that humans are more capable of making relative
judgements than absolute judgements. The pair-wise comparisons are made using a
nine-point scale:
1 = Equal importance or preference
3 = Moderate importance or preference of one over another
5 = Strong or essential importance or preference
7 = Very strong or demonstrated importance or preference
9 = Extreme importance or preference
Matrices are developed wherein each criterion/alternative is compared against the others.
If Criterion A is strongly more important compared to Criterion B (i.e. a value of “5”), then
Criterion B has a value of 1/5 compared to Criterion A. Thus, for each comparative score
given, the reciprocal is awarded to the opposite relationship. The normalized weight is
calculated for each criterion using the geometric mean14 of each row in the matrix divided
by the sum of the geometric means of all the criteria. This process is then repeated for the
alternatives comparing them one to another to determine their relative value/importance
for each criterion (i.e. determine the mean weighted value). The calculations are easily set
up in a spreadsheet (see example in Appendix 5.1-A.xls).The order of comparisons can
14
The geometric mean is the nth root of the product of n values. Thus, the geometric mean of the scores: 1, 2, 3, and 10 is the fourth
1/4
root of (1 x 2 x 3 x 10), which is the fourth root of 60, (60) = 2.78. The geometric mean is less affected by extreme values than is the
arithmetic mean.
© InnoSkills
13/18
2008-06-12
InnoSkills – Innovation Skills for SME´s
5. Evaluation of innovative solutions
5.1 Decision Making Process
----------------------------------------------------------------------------------------------------------------------------------------------------------------
help simplify this process. Try to identify and begin with the most important criterion and
proceed through the criteria to the least important. When comparing alternatives try to
identify the one with the greatest benefits for each associated criterion, and begin with it.
To identify the preferred alternative calculate the mean weighted value, i.e. multiply each
normalized alternative score by the corresponding criterion weight, and sum the results for
all of criteria. The preferred alternative will have the highest total score.
AHP, like the other methods, can rank alternatives according to quantitative or qualitative
(subjective) data. Qualitative/subjective criteria are based on the evaluation team’s
feelings or perceptions. A sensitivity analysis can be performed to determine how the
alternative selection would change with different criteria weights. The whole process can
be repeated and revised, until everyone is satisfied by the fact that all the important
features needed to solve the problem, or the selection of the preferred alternative, have
been covered.
AHP is a useful technique when there are multiple criteria, since most people cannot deal
with more than seven decision considerations at a time.
Example 2: Example of multi-criteria decision-making process is presented in Appendix
5.1_A.xls. Sheet 1 defines the problem: to pick a replacement vehicle for the motor pool
fleet.
The requirements are:
1. made in USA (eliminates products not manufactured in the USA);
2. minimum 4, maximum 6 passengers (eliminates vans, minibuses, sports cars);
3. maximum cost $28 000 (eliminates high-end, luxury cars);
4. new car, current model year (eliminates used vehicles).
We agreed on the following goals:
1. maximize passenger comfort;
2. maximize passenger safety;
3. maximize fuel efficiency;
4. maximize reliability;
5. minimize investment cost.
Despite the limitations, many alternatives remain - see the preliminary selection of four
models in sheet "Inputs".
Sheet “Pros_and_Cons” illustrates the procedure of Pros and Cons approach – in
corresponding boxes we list arguments for and against each alternative and in the end
select one with the highest score in the most important criteria. This approach can be
easily extended by assigning numerical values to criteria and use so called “forced-field
analysis”.
Sheet “Kepner-Tregoe” can be used as a guide to the K-T method. In this method, we first
assign weights to criteria (column “criteria weights”) and then evaluate the level of criteria
© InnoSkills
14/18
2008-06-12
InnoSkills – Innovation Skills for SME´s
5. Evaluation of innovative solutions
5.1 Decision Making Process
----------------------------------------------------------------------------------------------------------------------------------------------------------------
satisfaction on the scale 0-100 for each alternative (“alternative score”). Finally we
calculate total scores and select the alternatives with the highest total score. Please,
study the formulas in column “total score”. You can use this sheet as a template for your
problem.
Sheet “AHP (Saaty)” shows how to use the AHP method. This approach is the most
sophisticated of all approaches presented here and, in order to understand it properly, we
suggest to carefully studying all the formulas in this sheet.
5.1.5 Summary
The goal of this module on decision-making is to help decision-makers choose and
document the best alternative in a clear and logical way. The decision-making methods
described in this module are promptly applicable to a wide range of decisions, from simple
ones, as picking a restaurant for a special meal, to those that are made complex by
interdepartmental or intercompany interfaces. Expert facilitation can help assure that all
the steps are properly performed and documented. Feedback from the decision-maker(s)
is vital to the process.
The key to developing an adequate problem statement is to ask enough questions about
the problem to ensure that the final report will clearly answer the questions of reviewers
and stakeholders. Requirements spell out what the solution to the problem must do. Goals
are useful in identifying superior alternatives. The decision team evaluates the
requirements and goals and suggests alternatives that will meet the requirements and
satisfy as many goals as possible. The best alternative will be the one that most nearly
achieves the goals. Criteria must be developed to discriminate among alternatives in a
meaningful way. The decision-maker’s approval is crucial before the criteria are used to
evaluate the alternatives. Alternatives can be evaluated with quantitative methods,
qualitative methods, or any combination.
The evaluation methods introduced here are adaptable to many situations, as determined
by the complexity of the problem, needs of the customer, experience of the decision team
/analysts/ facilitators, and the time and resources available. The decision-making method
selection needs to be based on the complexity of the problem and the experience of the
team. A final solution should fulfil the desired state, meet requirements, and best achieve
the goals within the values of the decision makers.
Once the preferred alternative has been validated, the decisionmaker can present it as a
suggestion to the company management. A final written report must offer a description of
the decision process, assumptions, methods, and conclusions recommending the final
solution.
As the task of this introductory text (based mainly on literature15) does not allow us to deal
in detail with some more sophisticated, yet very useful tools and methods, we provide
15
Baker D. et al., Guidebook to Decision-Making Methods, McClave J.T., Benson P.G.: Statistics for
Business and Economics
© InnoSkills
15/18
2008-06-12
InnoSkills – Innovation Skills for SME´s
5. Evaluation of innovative solutions
5.1 Decision Making Process
----------------------------------------------------------------------------------------------------------------------------------------------------------------
references to additional resources, where the interested reader can find further descriprion
of methods together with examples of their applications.
At the very end of this unit we would like to underline the importance of risk
management16, sensitivity analysis17 and scenario planning.18
In this module, we tried to explain the basics of structured decision-making. We
hope you understand that even the most sophisticated mathematical methods
are influenced by approach to risk, ability to reliably estímate the behaviour of
markets, competitors and other factors, but always depend also on experience.
Especially in the field of innovation, there is a high level of risk. Unfotunatelly, you can not
completely avoid failures if you try something new. However, you should learn from your
successes and failures, as well as from successes and failures of your colleagues,
competitors, etc.
For more detailed information on decision-making, please take a look at the Bibliography
and Web sites sections at the end of this issue. If necessary, ask experienced colleagues
and friends for help, or invite external external consultants.
16
Wikipedia articles on Risk management, Risk assessment, Risk matrix; DoE Risk Management Guide – see Further
reading
17
18
Wikipedia article on Sensitivity analysis – see Further reading
Wikipedia article on Scenario planning – see Further reading
© InnoSkills
16/18
2008-06-12
InnoSkills – Innovation Skills for SME´s
5. Evaluation of innovative solutions
5.1 Decision Making Process
----------------------------------------------------------------------------------------------------------------------------------------------------------------
5.1.6 Bibliography
Baker D. et al., Guidebook to Decision-Making Methods, WSRC-IM-2002-00002, U.S.
DoE, 2001, http://emi-web.inel.gov/Nissmg/Guidebook_2002.pdf
McClave J.T., Benson P.G.: Statistics for Business and Economics, Dellen Publishing
Company, San Francisco, 1988, ISBN 0-02-379020-2
Further reading
Cooper R., Winning at new products, Basic Books, 2001, ISBN 978-0-7382-0463-5
Cooper R., Edgett S.J., Kleinschmidt E.: Portfolio management fo new products, Basic
Books, 2001, ISBN 978-0-7382-0514-4
Forman, E., and Selly, M., Decision by Objectives, http://mdm.gwu.edu/Forman/.
Kirkwood C. W., Decision Tree Primer, 2002,
http://www.public.asu.edu/~kirkwood/DAStuff/decisiontrees/index.html
Michel R. (1995), Product Innovation Strategy Pure and Simple, McGraw-Hill
http://www.economy-point.org/decision-theory/index.html, viewed 15-th July, 2008, Basics
of decision theory, some information on the simple efficiency analysis (NWA) or the more
precise Analytic Hierarchy Process (AHP).
Risk management, http://en.wikipedia.org/wiki/Risk_management , viewed April 24, 2009
Risk assessment, http://en.wikipedia.org/wiki/Risk_Assessment , viewed April 24, 2009
Risk Matrix, http://en.wikipedia.org/wiki/Risk_Matrix, viewed April 24, 2009
Risk Management Guide, DOE G 413.3-7,
http://www.directives.doe.gov/pdfs/doe/doetext/neword/413/g4133-7.pdf
Sensitivity analysis, http://en.wikipedia.org/wiki/Sensitivity_analysis, viewed April 24, 2009
Scenario planning, http://en.wikipedia.org/wiki/Scenario_planning, viewed April 24, 2009
Websites
http://www.investopedia.com/terms/w/weightedaverage.asp, viewed 25-th July, 2008, an
explanation on weighted average can be found at this address
Sarma A.K., Methods/Criteria of project evaluation or measures of project worth of
Investment, http://assamagribusiness.nic.in/agriclinics/Methods%20criteria.pdf , viewed
15-th July, 2008
http://en.wikipedia.org/wiki/Return_on_investment /, viewed: 15-th July, 2008, Financial
terms definitions provided
http://www.managementhelp.org/prsn_prd/prb_bsc.htm, viewed: 15-th July 2008 - basic
guidelines to problem solving and decision making are provided
http://www.freeworldacademy.com/newbizzadviser/fw23.htm, viewed: 15-th July 2008 explanation of the decision making process
© InnoSkills
17/18
2008-06-12
InnoSkills – Innovation Skills for SME´s
5. Evaluation of innovative solutions
5.1 Decision Making Process
----------------------------------------------------------------------------------------------------------------------------------------------------------------
5.1.7 GLOSSARY
Risk analysis - a technique to identify and assess factors that may jeopardize the
success of a project or achieving a goal. This technique also helps to define preventive
measures to reduce the probability of these factors from occurring and identify
countermeasures to successfully deal with these constraints when they develop to avert
possible negative effects on the competitiveness of the company, source:
http://en.wikipedia.org/wiki/Risk_analysis_(Business), viewed June 16, 2009
Criterion - A standard, rule, or test on which a judgment or decision can be based (Factor
for the evaluation of specific feature of innovations in the decision making process),
source: http://www.answers.com/topic/criterion, viewed June 16, 2009
Mean weighted value - An average in which each quantity to be averaged is assigned a
weight. These weightings determine the relative importance of each quantity on the
average, source: http://www.answers.com/topic/weighted-mean, viewed June 16, 2009
Sensitivity analysis is the study of how the variation (uncertainty) in the output of a model
can be apportioned, qualitatively or quantitatively, to different sources of variation in the
input of a model, source: http://en.wikipedia.org/wiki/Sensitivity_analysis
Uncertainty analysis is closely related to sensitivity analysis. It is used to examin the
reliability of model predictions, by taking into consideration various sources of uncertainty.
Operational research is an interdisciplinary branch of applied mathematics and formal
science that uses methods such as mathematical modeling, statistics, and algorithms to
arrive at optimal or near optimal solutions to complex problems. It is typically concerned
with optimizing the maxima (profit, assembly line performance, crop yield, bandwidth, etc)
or minima (loss, risk, etc.) of some objective function. Operations research helps
management achieve its goals using scientific methods, source:
http://en.wikipedia.org/wiki/Operational_Analysis, viewed June 16, 2009
Game theory attempts to mathematically capture behavior in strategic situations, in which
an individual's success in making choices depends on the choices of others, source:
http://en.wikipedia.org/wiki/Game_theory, viewed June 16, 2009
© InnoSkills
18/18
2008-06-12