2 The Analytic Hierarchy Process 2.1 Do You Need a Formal Decision-Making Framework? The complexity of the modern world is a much-acknowledged fact. As the human race develops, complexity increases. Technology has created various artefacts to relieve us of manual, routine and time-consuming tasks. The predictable and deterministic world of the past has been replaced by the uncertain, random and disorderly world of today. Technological advances in multiple fields of human activity have created a planet on which things happen at electronic speed. Rapidly increasing complexity and information overload have schemed together to drastically reduce the time available for making decisions. The decision-maker is stressed, overloaded with unsolicited information, has not enough time to analyse the situation, and yet must make decisions that have high-risk implications or consequences. What does the decision-maker need? Human decision-making in the world characterised above needs a quick-response analysis of the situation that some how captures the decision-maker’s intuition, judgement and experience. This can then be combined with detailed quantitative analysis based on the information glut that is churned out from the plethora of process measurements, balanced scorecards, business intelligence, data accumulation and information generation techniques and systems in place in various organisations. Decision-making, especially strategic decision-making, with high stakes and stochastic future implications, involves multiple actors. In most organisations, these decisions are made collectively, irrespective of whether the organisation is a privately owned business, a public limited company or a government agency. This is true for national, international and multinational organisations as well. Even in small and medium enterprises decision-making is rarely done by a lone individual sitting in isolation. The reality of a group making these high-stakes decisions generates a requirement for creating communication links between the members of the decision-making group with a common understanding of the syntax and semantics of the underlying issues. Decisions made in an ad-hoc, unstructured or semi-structured manner, based on the availability of only a subset of the decisionmaking group at the time of decisions, has a high probability of being not just suboptimal but utterly wrong, with disastrous results. The single-criterion and simple decision-making requirements of the past have today given way to highly complex decision problems involving multitudes of variables, which may be stochastic, fuzzy or at worst unknown. As the time required to 11 12 Strategic Decision-Making and the AHP make decisions has been severely reduced, the onus of decision-making has shifted to the lowest level of the hierarchy of organizations. Thus we can infer that hierarchical organizations need a comprehensive formal framework for decision-making owing to increasing complexity and stochasticity, the involvement of many decision-makers and the shift in decision-making requirements to field-level workers, as explained above. It is not that this has become a sudden requirement; in the past formal decision analysis techniques were developed to tackle these problems. However, these have been found to be too mathematical or theoretical or else capable only of solving older problems totally different from those of today. Structured methods utilising the theoretical and practical advances made in the fields of mathematics, operations research, cybernetics, artificial intelligence etc. have become important aids to decision-making in all sectors. The theoretical underpinnings in such decision aids is the principle of optimisation, which tries to maximise or minimise certain combinations of conflicting variables which represent the metric of interest for the decision-maker under constraints imposed on these variables by the real-life situation. This principle has resulted in an enormous intellectual expansion of quantitative decision-making aids using standard optimisation techniques. Empirical, common-sense or subjective decision-making, supplemented by some simple calculations using arithmetic, geometry and calculus, has evolved into techniques of sophisticated operations research based on the principle of optimisation and has resulted in enhanced decision aids at every level of organisation, thanks to increasing automation in the form of the computerisation of the techniques involved. 2.2 Formal Decision-Making Techniques When the rules of the game are well laid out, when the environment in which one operates is predictable, when the opposition is known, when the actors behave in a deterministic manner, when costs vary within a small, narrow band, and, when linear relations are the norm, one can try to make decisions using the standard optimisation techniques. However, when the benefits of actions are unpredictable, when relationships between variables may be not only non-linear and stochastic but also actually unknown, the principle of optimisation for decision-making will not help much. This is exactly the situation we face in the world of today. Strategic, operational and tactical agility, in quickly absorbing a situation and responding with maximum concentration of effort at the point of need, is the absolute requirement. At the tactical and operational level in various large-scale organisations, standard optimisation techniques for decision-making have in the more orderly world of the past helped to some extent. However, at the strategic level these techniques have been unable to make any greater impact. Decision-making can be considered as the choice, on some basis or criteria, of one alternative among a set of alternatives. A decision may need to be taken on the basis of multiple criteria rather than a single criterion. This requires the assessment of various criteria and the evaluation of alternatives on the basis of each criterion and then the aggregation of these evaluations to achieve the relative ranking of the alternatives with respect to the problem. The problem is further compounded when there are several or more experts whose opinions need to be incorporated in The Analytic Hierarchy Process 13 the decision-making. It is lack of adequate quantitative information which leads to dependence on the intuition, experience and judgement of knowledgeable persons called experts. We can define a generic decision-making problem as consisting of the following activities: ● ● ● ● ● ● Studying the situation. Organising multiple criteria. Assessing multiple criteria. Evaluating alternatives on the basis of the assessed criteria. Ranking the alternatives. Incorporating the judgements of multiple experts. The problem can be abstracted as how to derive weights, rankings or importance for a set of activities according to their impact on the situation and the objective of decisions to be made. This is the process of multiple-criteria decision-making (MCDM). The MCDM problems have been studied under the general classification of operations research (OR) problems, which deal with decision-making in the presence of a number of often conflicting criteria. The field of MCDM is divided into multi-objective decision-making (MODM) and multi-attribute decision-making (MADM). When the decision space is continuous, MODM techniques such as mathematical programming problems with multiple objective functions are used. On the other hand, MADM deals with discrete decision spaces where the decision alternatives are predetermined. Many of the MADM methods have a common notion of alternatives and attributes. Alternatives represent different choices of action available to the decision-maker, the choice of alternatives usually being assumed to be finite. Alternatives need to be studied, analysed and prioritised with respect to the multiple attributes with which the MADM problems are associated. Attributes are also referred to as goals or decision criteria. Different attributes represent different dimensions of looking at the alternatives, and may be in conflict with each other, may not be easily represented on a quantitative scale – and hence may not be directly measurable – and may be stochastic or fuzzy. Further, these attributes may have totally different scales – quantitative or qualitative. Most of the MADM methods require that each attribute is given a weight or relative importance with respect to their impact on the decision problem being solved. MADM and MCDM have often been used to mean the same class of models; here we will use the more commonly used term MCDM to denote MADM problems. The weighted-sum method (WSM), or the decision matrix approach, is perhaps the earliest method employed. This evaluates each alternative with respect to each criterion and then multiples that evaluation by the importance of the criterion. This product is summed over all the criteria for the particular alternative to generate the rank of the alternative. Mathematically, Ri N ∑ aijw j (2.1) j1 where Ri is the rank of the ith alternative, aij is the actual value of the ith alternative in terms of the jth criterion, and wj is the weight or importance of the jth criterion. Let us assume there are two criteria, C1 and C2, and three alternatives,A1,A2 and A3. Let us assume that the weights assigned to the criteria C1 and C2 are W1 20 14 Strategic Decision-Making and the AHP Table 2.1 Decision matrix approach. Alternative A1 A2 A3 Criterion C1 (W1 20) C2 (W2 30) a11 5 a21 7 a31 11 a12 5 a22 3 a32 3 Weighted sum Ri N ∑ aijw j j1 250 230 310 Table 2.2 Forced decision matrix. Alternative A1 A2 A1 1 A2 0 A3 0 0 Total number of comparisons (N) A3 Score (S) Rating S/N 1 1 2 1 0 3 2/3 0.67 1/3 0.33 0/3 0.00 and W2 30, respectively. Each of the alternatives is evaluated with respect to each criterion. The computations are shown in Table 2.1. Subjectivity, bias and prejudice in giving these ratings and weights cannot be eliminated or evaluated in this method. The additive utility assumption on which this method is based creates problems when the units of the multiple criteria differ from one other. A variant of the decision matrix approach is the forced decision matrix (FDM) approach. In this, ratings are given in terms of 0 or 1. This winner-takes-all approach is easier to implement, because if a particular alternative is better on one parameter then the whole weight of that parameter goes to the alternative. The FDM approach is illustrated in Table 2.2. The table shows the evaluation of three alternatives, A1, A2 and A3, with respect to single criteria. In the FDM, pairwise comparisons of alternatives are made. As there are three alternatives we need to make three pairwise comparisons so that each alternative is compared with the others once. The weighted-product method (WPM) is very similar to the weighted-sum method (it also is called dimensionless analysis). Each alternative is compared with others by multiplying a number of ratios, one for each criterion. Mathematically, the comparison of alternatives A1 and A2 will be done as given in Equation (2.2). N R(A1/A2 ) ∏ (a1 j /a2 j ) wj (2.2) j1 where N is the number of criteria, aij is the actual value of the ith alternative in terms of the jth criterion and wj is the weight of the jth criterion. Two other methods, namely ELECTRE (elimination and choice translating reality) and TOPSIS (technique for order preference by similarity to ideal solution), have been described in the literature. The ELECTRE method is similar to the FDM method in principle as it deals with outranking relations by using pairwise comparisons. The basic concept of the TOPSIS method is that the selected alternative The Analytic Hierarchy Process 15 should have the shortest distance from the ideal solution and the furthest distance from the negative-ideal solution in a geometrical sense. Here we will not be discussing these two methods, as they are beyond the scope of the present discussion. The reader is referred to [1, 7] for details on these methods. The analytic hierarchy process (AHP) is a systematic approach developed in the 1970s to give decision-making based on experience, intuition and heuristics the structure of a well-defined methodology derived from sound mathematical principles. It provides a formalised approach where economic justification of the time invested in the decision-making process is provided by the better quality of the solutions to complex problems. 2.3 The Analytic Hierarchy Process – Background The AHP is based on the experience gained by its developer, T.L. Saaty, while directing research projects in the US Arms Control and Disarmament Agency. It was developed as a reaction to the finding that there is a miserable lack of common, easily understood and easy-to-implement methodology to enable the taking of complex decisions. Since then, the simplicity and power of the AHP has led to its widespread use across multiple domains in every part of the world. The AHP has found use in business, government, social studies, R&D, defence and other domains involving decisions in which choice, prioritization or forecasting is needed. Owing to its simplicity and ease of use, the AHP has found ready acceptance by busy managers and decision-makers. It helps structure the decision-maker’s thoughts and can help in organizing the problem in a manner that is simple to follow and analyse. Broad areas in which the AHP has been applied include alternative selection, resource allocation, forecasting, business process re-engineering, quality function deployment, balanced scorecard, benchmarking, public policy decisions, healthcare, and many more. Basically the AHP helps in structuring the complexity, measurement and synthesis of rankings. These features make it suitable for a wide variety of applications. The AHP has proved a theoretically sound and markettested and accepted methodology. Its almost universal adoption as a new paradigm for decision-making coupled with its ease of implementation and understanding constitute its success. More than that, it has proved to be a methodology capable of producing results that agree with perceptions and expectations. 2.4 The AHP – Step by Step The AHP provides a means of decomposing the problem into a hierarchy of subproblems which can more easily be comprehended and subjectively evaluated. The subjective evaluations are converted into numerical values and processed to rank each alternative on a numerical scale. The methodology of the AHP can be explained in following steps: Step 1: The problem is decomposed into a hierarchy of goal, criteria, sub-criteria and alternatives. This is the most creative and important part of decision-making. Structuring the decision problem as a hierarchy is fundamental to the process of 16 Strategic Decision-Making and the AHP GOAL Criterion 1 Criterion 2 …. Sub-criterion 11 Criterion P Sub-criterion 21 Sub-criterion P1 … … … Sub-criterion 1L Sub-criterion 2M Sub-criterion PN Alternative 1 Alternative 2 Alternative Q Alternative 3 … Figure 2.1 Generic hierarchic structure. A Extremely strong Very strong Strong Marginally strong Equal Marginally strong Strong X Very strong B Extremely strong Figure 2.2 Format for pairwise comparisons. the AHP. Hierarchy indicates a relationship between elements of one level with those of the level immediately below. This relationship percolates down to the lowest levels of the hierarchy and in this manner every element is connected to every other one, at least in an indirect manner. A hierarchy is a more orderly form of a network. An inverted tree structure is similar to a hierarchy. Saaty suggests that a useful way to structure the hierarchy is to work down from the goal as far as one can and then work up from the alternatives until the levels of the two processes are linked in such a way as to make comparisons possible. Figure 2.1 shows a generic hierarchic structure.At the root of the hierarchy is the goal or objective of the problem being studied and analysed. The leaf nodes are the alternatives to be compared. In between these two levels are various criteria and sub-criteria. It is important to note that when comparing elements at each level a decision-maker has just to compare with respect to the contribution of the lower-level elements to the upper-level one. This local concentration of the decision-maker on only part of the whole problem is a powerful feature of the AHP. Step 2: Data are collected from experts or decision-makers corresponding to the hierarchic structure, in the pairwise comparison of alternatives on a qualitative scale as described below. Experts can rate the comparison as equal, marginally strong, strong, very strong, and extremely strong. The opinion can be collected in a specially designed format as shown in Figure 2.2. “X” in the column marked “Very strong” indicates that B is very strong compared with A in terms of the criterion on which the comparison is being made. The comparisons are made for each criterion and converted into quantitative numbers as per Table 2.3. Step 3: The pairwise comparisons of various criteria generated at step 2 are organised into a square matrix. The diagonal elements of the matrix are 1. The criterion in the ith row is better than criterion in the jth column if the value of element (i, j) is more than 1; otherwise the criterion in the jth column is better than that in the ith row. The (j, i) element of the matrix is the reciprocal of the (i, j) element. The Analytic Hierarchy Process 17 Table 2.3 Gradation scale for quantitative comparison of alternatives. Option Numerical value(s) Equal Marginally strong Strong Very strong Extremely strong Intermediate values to reflect fuzzy inputs Reflecting dominance of second alternative compared with the first 1 3 5 7 9 2, 4, 6, 8 Reciprocals Step 4: The principal eigenvalue and the corresponding normalised right eigenvector of the comparison matrix give the relative importance of the various criteria being compared. The elements of the normalised eigenvector are termed weights with respect to the criteria or sub-criteria and ratings with respect to the alternatives. Step 5: The consistency of the matrix of order n is evaluated. Comparisons made by this method are subjective and the AHP tolerates inconsistency through the amount of redundancy in the approach. If this consistency index fails to reach a required level then answers to comparisons may be re-examined. The consistency index, CI, is calculated as CI (max n)/(n 1) where max is the maximum eigenvalue of the judgement matrix. This CI can be compared with that of a random matrix, RI. The ratio derived, CI/RI, is termed the consistency ratio, CR. Saaty suggests the value of CR should be less than 0.1. Step 6: The rating of each alternative is multiplied by the weights of the sub-criteria and aggregated to get local ratings with respect to each criterion. The local ratings are then multiplied by the weights of the criteria and aggregated to get global ratings. The AHP produces weight values for each alternative based on the judged importance of one alternative over another with respect to a common criterion. 2.5 AHP – Theory Saaty [19] describes the seven pillars of the AHP as follows: ● ● ● ● ● ● ● Ratio scales, proportionality and normalised ratio scales. Reciprocal paired comparisons. The sensitivity of the principal right eigenvector. Clustering and using pivots to extend the scale. Synthesis to create a one-dimensional ratio scale for representing the overall outcome. Rank preservation and reversal. Integrating group judgements. The use of ratio scales for comparisons helps in unifying the multidimensionality of the problem in a unified dimension from the perspective of the final outcome. Comparison of oranges and apples can be achieved if their properties are reduced to 18 Strategic Decision-Making and the AHP dimensionless quantities such as the ratios of the properties in some specific dimension or measurement. Ratios are invariant under multiplication by a positive quantity. For example, if a steel rod of length A metres is compared with a wooden rod of length B metres, it is easy to ascertain the difference between these two rods, since they are measured in the units.The length of the steel rod is A B metres more or less than that of the wooden rod, depending upon whether A B is a positive or a negative quantity. Now let us assume that the weight of the steel rod is U kilograms and that of the wooden rod V kilograms.Thus we can compare the weights as U V,finding that the steel rod is U V kilograms heavier or lighter than the wooden one. We can see that measurement of two unique properties is quite possible using some specific units. Now look at the problem of comparing two properties. The question asked is how one would compare the two rods in terms of length and weight. The traditional answer will be that the difference in length is A B metres and difference in weight is U V kilograms. Which one should the decision-maker choose? It depends upon the importance that the decision-maker gives to weight or length. Let us assume Imp1 is the importance given to weight, Imp2 that given to length. Can the decision-maker then choose the steel or the wooden rod based on the quantity Imp1 (A B) Imp2 (U V)? The answer is obviously no, as the units do not match. The trick lies in eliminating the units. Instead of difference, if we had taken the ratio of lengths and ratio of weights of the two rods, we could easily have compared the two rods with respect to multiple dimensions. This implies that if we take the ratio of steel rod length and wooden rod length, i.e. A/B, and the ratio between the weights of the two rods, i.e. U/V, we can easily choose the rod which depending upon the quantity Q Imp1 A/B Imp2 U/V. Let us give values to these variables, say A 20 metres, B 80 metres, U 50 kg, V 25 kg, Imp1 20 and Imp2 10; then Q 20 (20/80) 10 50/25 25. What do we do with this Q? Q has to be put in perspective that total importance of both these dimensions, i.e. Imp1 Imp2 20 10 30. Hence when we get Q 25 it has to be compared with 30. The ratio comes out to be less than 1 (i.e. 25/30), hence the decision-maker can choose the wooden rod.Another way of doing this is to normalise the importance of the criteria; if Imp1 20/30 0.67 and Imp2 10/30 0.33, it will help in making the computations simple. How do we get Imp1 and Imp2? We can take the ratios again! It is perhaps easy to measure the lengths and weights of two distinct objects in some predetermined units such as metres and kilograms and then compare them by taking the ratio between the measured quantities. However, when asked to compare two objects or persons with respect to abstract properties such as beauty, honesty, smartness, etc., how does one do it? In this scenario, units for absolute measurement are missing. Not only that: absolute measurement of the two distinct objects being compared is actually not needed. It is the relative measurement that is the essence of comparison. This fact, that only relative measurement is needed, is the fundamental pillar of the AHP. Once we realise that only relative measurement is needed, it means that, at a particular point in time, we need to compare only two objects with respect to the property, criterion, sub-criterion or goal as the case may be. This realisation leads us to paired comparisons. We have now reached the conclusion that relative, paired comparisons are what decision-makers actually do or should do. Since we have taken a ratio of two objects with respect to an attribute, it is easy to translate it into a reciprocal relationship, i.e. if A compares w1/w2 times compared with B then B compares w2/w1 times compared with A. Reciprocal, paired comparisons for relative measurement are the second pillar of the AHP. The measurement scale defined for the AHP is one of 1 – 9 in absolute numbers. The Analytic Hierarchy Process 19 If A is a consistent matrix, small perturbations in A do not lead to perturbations in the principal eigenvector of A. If the order of the matrix, n, is small then small perturbations in A do not create perturbations in the principal eigenvector. The AHP allows for clustering to extend the comparison scale from 1 9 to 1 . Taking an alternative in a cluster with properties measured in the same order and comparing it with higher-order alternatives can perform this function. In this way, very small alternatives can be compared with very large ones. The synthesis of global priorities at each level of the hierarchy is carried out by a multilinear form of elements of priority vectors at the lower levels.The AHP has a well-established theory and guidelines for when to preserve rank and when to allow it to reverse. The AHP also provides a methodology to allow the aggregation of individual judgements for taking group decisions. Theoretically the AHP is based on four axioms given by Saaty; these are: Axiom 1: The decision-maker can provide paired comparisons aij of two alternatives i and j corresponding to a criterion/sub-criterion on a ratio scale which is reciprocal, i.e. aji 1/aij. Axiom 2: The decision-maker never judges one alternative to be infinitely better than another corresponding to a criterion, i.e. aij . Axiom 3: The decision problem can be formulated as a hierarchy. Axiom 4: All criteria/sub-criteria which have some impact on the given problem, and all the relevant alternatives, are represented in the hierarchy in one go. In nutshell, there are three major concepts behind the AHP, as follows: The AHP is analytic – mathematical and logical reasoning for arriving at the decision is the strength of the AHP. It helps in analysing the decision problem on a logical footing and assists in converting decision-makers’ intuition and gut feelings into numbers which can be openly questioned by others and can also be explained to others. The AHP structures the problem as a hierarchy – Hierarchic decomposition comes naturally to human beings. Reducing the complex problem into sub-problems to be tackled one at a time is the fundamental way that human decision-makers have worked. Evidence from psychological studies suggests that human beings can compare 7 2 things at a time. Hence to deal with a large and complex decisionmaking problem it is essential to break it down as a hierarchy. The AHP allows that. The AHP defines a process for decision-making – Formal processes for decisionmaking are the need of the hour. Decisions, especially collective ones, need to evolve.A process is required that will incorporate the decision-maker’s inputs, revisions and learnings and communicate them to others so as to reach a collective decision. The AHP has been created to formalise the process and place it on a scientific footing. The AHP helps in aiding the natural decision-making process. 2.6 The AHP – Applications Since its discovery the AHP has been applied in a variety of decision-making scenarios: ● ● Choice – selection of one alternative from a set of alternatives. Prioritisation/evaluation – determining the relative merit of a set of alternatives. 20 ● ● ● Strategic Decision-Making and the AHP Resource allocation – finding best combination of alternatives subject to a variety of constraints. Benchmarking – of processes or systems with other, known processes or systems. Quality management. Domains that have seen many applications of the AHP include healthcare, defence, project planning, technological forecasting, marketing, new product pricing, economic forecasting, policy evaluation, social sciences, etc. Besides its applications in conflict analysis, military operations research, regional and urban planning, R&D management and space exploration, the AHP has developed as a widely accepted methodology for decision-making. As a technique it has evolved over the years and has been applied in conjunction with other mathematical modeling and analysis techniques. 2.7 Pitfalls, Modifications and Extensions Despite wide applications of the AHP in a variety of domains and at different levels of the decision hierarchy, the AHP has been criticized from several viewpoints. The first problem is that of rank reversal. This was indicated by [12]. In many scenarios, the rankings of alternatives obtained by the AHP may change if a new alternative is added. Belton and Gear introduced one alternative, which was an exact copy of one of the alternatives and then re-evaluated the matrices. This amounted to adding one more column to the matrix with elements similar to those of the original entries in the column corresponding to the earlier alternative. Robins [2, 3, 4] enumerates the following five issues related to the application of the AHP: ● ● ● ● Vendors get improperly penalized. The ratio scale is inaccurate. The process can generate inconsistencies as an artefact of its calculations that have nothing to do with consistency of judgment. Rank reversal. The AHP has seen major controversies. One of them has been reflected in the exchanges of Dyer with Saaty and Vargas [8, 9, 10, 11], in the journal Management Science. Despite the controversies and problems faced by the technique of the AHP, it has survived and thrived. Its ease of use and widespread acceptance has resulted in it being applied to decisions related to war games, to technology forecasting, to the evaluation of attack helicopters, to the assessment of presidential candidates, to decisions about buying a car, to choosing one’s spouse. In the next chapters we will focus on how the AHP can be used to aid strategic-level decisions in business, defence and governance. References 1. Triantaphyllou E, et al. (1998) Multi-criteria decision-making: An operations research approach. In: Webster JG (ed.) Encyclopedia of Electrical and Electronics Engineering.Wiley, New York, 15: 175–186. The Analytic Hierarchy Process 21 2. Robins ES, Five Major Pitfalls in the AHP Process, Technical Report no. 9811pub-esr, http://www.TechnologyEvaluation.com 3. Robins ES, The Analytic Hierarchy Process – Issues, Problems, and Recommendations, Technical Report no. 9811pub-esr, http://www.TechnologyEvaluation.com 4. Robins ES, An Investigation into the Efficacy of the Consistency Ratio with Matrix Order – Limits of the AHP, Report no. ARL97-ER-D01, http://www.TechnologyEvaluation.com 5. Triantaphyllou E, Mann SH (1989) An examination of the effectiveness of multi-dimensional decisionmaking methods: a decision-making paradox. Decision Support Systems 5: 303–312. 6. Forman EH, Gass SI (2001) The analytic hierarchy process – an exposition. Operations Research 49(4): 469–486. 7. Winkler RL (1990) Decision modeling and rational choice: AHP and utility theory. Management Science 36(3). 8. Dyer JS (1990) Remarks on the analytic hierarchy process. Management Science 36(3). 9. Saaty TL (1990) An exposition of the AHP in reply to the paper “Remarks on the analytic hierarchy process”. Management Science 36(3). 10. Harkar PT, Vargas LG (1990) Reply to “Remarks on the analytic hierarchy process” by J S Dyer. Management Science 36(3). 11. Dyer JS (1990) A clarification of “Remarks on the analytic hierarchy process”, Management Science 36(3). 12. Belton V, Gear AE (1983) On a shortcoming of Saaty’s method of analytic hierarchies. Omega 11(3): 227–230. 13. Saaty TL, Vargas LG (1984) The legitimacy of rank reversal. Omega 12(5): 513–516. 14. Belton V, Gear AE (1985) The legitimacy of rank reversal – a comment. Omega 13(3): 143–144. 15. Vargas LG (1985) A rejoinder. Omega 13(4): 249. 16. Zeshui X, Cuiping W (1999) A consistency improving method in the analytic hierarchy process. European Journal of Operational Research 116: 443–449. 17. Ramanathan R (1997) Stochastic decision-making using multiplicative AHP. European Journal of Operational Research 97: 543–549. 18. Frei FX, Harker PT (1999) Measuring aggregate process performance using AHP. European Journal of Operational Research 116: 436–442. 19. Saaty TL, Vargas LG (2001) Models, Methods, Concepts and Applications of the Analytic Hierarchy Process. Kluwer, Dordrecht. 20. Saaty TL (1980) The Analytic Hierarchy Process. McGraw-Hill, New York. 21. Saaty TL, Vargas LG (1991) Prediction, Projection and Forecasting. Kluwer, Boston, MA. 22. Golden BL, Wasil EA, Harker PT (eds) (1989) The Analytic Hierarchy Process. Springer, Berlin. http://www.springer.com/978-1-85233-756-8
© Copyright 2026 Paperzz