Helsinki University of Technology
Systems Analysis Laboratory
Rank-Based Sensitivity Analysis of
Multiattribute Value Models
Antti Punkka and Ahti Salo
Systems Analysis Laboratory
Helsinki University of Technology
P.O. Box 1100, 02015 TKK, Finland
http://www.sal.tkk.fi/
[email protected]
1
Helsinki University of Technology
Systems Analysis Laboratory
Additive Multiattribute Value Model

Provides a complete rank-ordering for the alternatives
– Selection of the best alternative
– Rank-ordering of e.g. universities (Liu and Cheng 2005) or graduate programs
(Keeney et al. 2006)
– Prioritization of project proposals or innovation ideas (e.g. Könnölä et al. 2007)
n
V ( x )   wi vi ( xij )
j
i 1

Methods for global sensitivity analysis on weights and scores
– Focus only on the selection of the best alternative
1. Ex post: Sensitivity of the decision recommendation to parameter variation
2. Ex ante: Computation of viable decision candidates subject to incomplete
information about the parameter values
(e.g., Rios Insua and French 1991, Butler et al. 1997, Mustajoki et al. 2006)
INFORMS Annual Meeting, Washington DC 2008
2
Helsinki University of Technology
Systems Analysis Laboratory
Sensitivity Analysis of Rankings

Consider the full rank-ordering instead of the most preferred
alternative
– How ’sensitive’ is the rank-ordering x1 x 4 x3 x 2 ?
– How to compare two rank-orderings? How to communicate differences?
x1

x4
x3
x2
vs
x1
x2
x4
x3 ?
We compute the attainable rankings for each alternative
subject to global variation in weights and scores
– How sensitive is the ranking of an alternative subject to parameter variation?
– Is the ranking of university X sensitive to the attribute weights applied?
– What is the best / worst attainable ranking of project proposal Y?
INFORMS Annual Meeting, Washington DC 2008
3
Helsinki University of Technology
Systems Analysis Laboratory
Incomplete Information

Model parameter uncertainty before computation
1. Relax complete specification of parameters
» ”Error coefficients” on the statements, e.g. weight ratios
» E.g. Mustajoki et al. (2006)
2. Directly elicit and apply incomplete information
»
»
»
»
Incompletely defined weight ratios: 2 ≤ w3/ w2 ≤ 3
Ordinal information about weights: w1 ≤ w3
Score intervals: 0.4 ≤ v1(x12) ≤ 0.6
E.g., Kirkwood and Sarin (1985),
Salo and Hämäläinen (1992), Liesiö et al. (2007)
(0,0,1)
w3
Sw
w1

Set of feasible weights and scores (S)
INFORMS Annual Meeting, Washington DC 2008
1  w3 / w2  3,
(0,1,0)
w2
(1,0,0)
2  w3 / w1  4
4
Helsinki University of Technology
Systems Analysis Laboratory
Attainable Rankings

Existing output concepts of ex ante sensitivity analysis do not
consider the full rank-ordering of alternative set X
– Value intervals focus on 1 alternative at a time
– Dominance relations are essentially pairwise comparisons
– Potential optimality focuses on the ranking 1

Alternative xk can attain ranking r, if exists feasible parameters
such that the number of alternatives with higher value is r-1
(w, v)  S such that |{x j  X | V ( x j )  V ( x k )}| r  1
INFORMS Annual Meeting, Washington DC 2008
5
Helsinki University of Technology
Systems Analysis Laboratory
Attainable Rankings: Example

2 attributes, 4 alternatives with fixed scores, w1 [0.4, 0.7]
V
x1
x2
x3
x4
w1
w2
ranking 1 is attainable for x3
ranking 4 is attainable for x1
ranking 1 is attainable for x2
ranking 3 is attainable for x3
0.4
0.7
0.6
0.3
INFORMS Annual Meeting, Washington DC 2008
Attainable rankings
6
Helsinki University of Technology
Systems Analysis Laboratory
Computation of Attainable Rankings

Application of incomplete information  set of feasible weights
and scores (S)

If S is convex, all rankings between the best and the worst
attainable rankings are attainable
j
j
k
– Best ranking of xk: min |{x  X | V ( x )  V ( x )}| 1
( w,v )S
– Worst ranking of xk:

max |{x j  X | V ( x j )  V ( x k )}| 1
( w,v )S
MILP model to obtain the best / worst ranking of each xk
– V(x) expressed in non-normalized form (linear in w and v)
– # of binary variables = |X| - 1
INFORMS Annual Meeting, Washington DC 2008
7
Helsinki University of Technology
Systems Analysis Laboratory
Example: Shangai Rank-Ordering of Universities

Shanghai Jiao Tong University ranks the world universities
annually

Example data from 2007
– http://ed.sjtu.edu.cn/ranking2007.htm
– 508 universities

Additive model for rank-ordering of the universities
INFORMS Annual Meeting, Washington DC 2008
8
Helsinki University of Technology
Systems Analysis Laboratory
Attributes
Criterion
Indicator
Quality of Education
Alumni of an institution winning Nobel Prizes
Alumni
and Fields Medals
10 %
Staff of an institution winning Nobel Prizes
and Fields Medals
Award
20 %
Highly cited researchers in 21 broad subject
categories
HiCi
20 %
Articles published in Nature and Science
N&S
20 %
Articles in Science Citation Index-expanded,
Social Science Citation Index
SCI
20 %
Academic performance with respect to the
size of an institution
Size
10 %
Quality of Faculty
Research Output
Size of Institution
Code
Weight
Table adopted from http://ed.sjtu.edu.cn/ranking2007.htm
INFORMS Annual Meeting, Washington DC 2008
9
Helsinki University of Technology
Systems Analysis Laboratory
Data
World
Rank
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
23
25
26
27
28
29
30
30
Institution
Region
Harvard Univ
Stanford Univ
Univ California - Berkeley
Univ Cambridge
Massachusetts Inst Tech (MIT)
California Inst Tech
Columbia Univ
Princeton Univ
Univ Chicago
Univ Oxford
Yale Univ
Cornell Univ
Univ California - Los Angeles
Univ California - San Diego
Univ Pennsylvania
Univ Washington - Seattle
Univ Wisconsin - Madison
Univ California - San Francisco
Johns Hopkins Univ
Tokyo Univ
Univ Michigan - Ann Arbor
Kyoto Univ
Imperial Coll London
Univ Toronto
Univ Coll London
Univ Illinois - Urbana Champaign
Swiss Fed Inst Tech - Zurich
Washington Univ - St. Louis
Northwestern Univ
New York Univ
Rockefeller Univ
Americas
Americas
Americas
Europe
Americas
Americas
Americas
Americas
Americas
Europe
Americas
Americas
Americas
Americas
Americas
Americas
Americas
Americas
Americas
Asia/Pac
Americas
Asia/Pac
Europe
Americas
Europe
Americas
Europe
Americas
Americas
Americas
Americas
Regional
National Score on Score on Score on Score on Score on Score on
Country
Rank
Rank
Alumni Award
HiCi
N&S
SCI
Size
1
USA
1
100
100
100
100
100
73
2
USA
2
42
78.7
86.1
69.6
70.3
65.7
3
USA
3
72.5
77.1
67.9
72.9
69.2
52.6
1
UK
1
93.6
91.5
54
58.2
65.4
65.1
4
USA
4
74.6
80.6
65.9
68.4
61.7
53.4
5
USA
5
55.5
69.1
58.4
67.6
50.3
100
6
USA
6
76
65.7
56.5
54.3
69.6
46.4
7
USA
7
62.3
80.4
59.3
42.9
46.5
58.9
8
USA
8
70.8
80.2
50.8
42.8
54.1
41.3
2
UK
2
60.3
57.9
46.3
52.3
65.4
44.7
9
USA
9
50.9
43.6
57.9
57.2
63.2
48.9
10
USA
10
43.6
51.3
54.5
51.4
65.1
39.9
11
USA
11
25.6
42.8
57.4
49.1
75.9
35.5
12
USA
12
16.6
34
59.3
55.5
64.6
46.6
13
USA
13
33.3
34.4
56.9
40.3
70.8
38.7
14
USA
14
27
31.8
52.4
49
74.1
27.4
15
USA
15
40.3
35.5
52.9
43.1
67.2
28.6
16
USA
16
0
36.8
54
53.7
59.8
46.7
17
USA
17
48.1
27.8
41.3
50.9
67.9
24.7
1
Japan
1
33.8
14.1
41.9
52.7
80.9
34
18
USA
18
40.3
0
60.7
40.8
77.1
30.7
2
Japan
2
37.2
33.4
38.5
35.1
68.6
30.6
3
UK
3
19.5
37.4
40.6
39.7
62.2
39.4
19
Canada
1
26.3
19.3
39.2
37.7
77.6
44.4
4
UK
4
28.8
32.2
38.5
42.9
63.2
33.8
20
USA
19
39
36.6
44.5
36.4
57.6
26.2
Switzerland
5
1
37.7
36.3
35.5
39.9
38.4
50.5
21
USA
20
23.5
26
39.2
43.2
53.4
39.3
22
USA
21
20.4
18.9
46.9
34.2
57
36.9
23
USA
22
35.8
24.5
41.3
34.4
53.9
25.9
23
USA
22
21.2
58.6
27.7
45.6
23.2
37.8
INFORMS Annual Meeting, Washington DC 2008
10
Helsinki University of Technology
Systems Analysis Laboratory
Sensitivity Analysis

How sensitive are the rankings to weight variation?
– What if different weights were applied?
– Relax point estimate weighting
1. Relative intervals around the point estimates
w  {w  S w0 | (1   ) wi*  wi  (1   ) wi*}
– E.g. =20 %, wi*=0.20: 0.16  wi  0.24
6
S  {w  R | wi  0,  wi  1}
0
w
6
i 1
2. Incomplete ordinal information
– Attributes with wi*=0.20 cannot be less important than those with wi*=0.10
– All weights lower-bounded by 0.02
w {w  S w0 | wk  wl k  {2,3, 4,5}, l {1, 6}, wi  0.02}
INFORMS Annual Meeting, Washington DC 2008
11
Helsinki University of Technology
Systems Analysis Laboratory
Results: Rank-Sensitivity of Top Universities
exact weights
Unsensitive rankings
University
20 % interval
30 % interval
”Different weighting would
likely yield a better ranking”
incompl. ordinal
no information
10th
442nd
Ranking
INFORMS Annual Meeting, Washington DC 2008
12
Helsinki University of Technology
Systems Analysis Laboratory
Conclusion

A model to compute attainable rankings
– Sufficiently efficient even with hundreds of alternatives and several attributes

Attainable rankings communicate sensitivity of rank-orderings
– Conceptually easy to understand
– Holistic view of global sensitivity at a glance independently of the # of attributes

Applicable output in Preference Programming framework
– Additional information leads to fewer attainable rankings

Connections to project prioritization
– Initial screening of project proposals for e.g. portfolio-level analysis
– Supports identification of ’clear decisions’ (cf. Liesiö et al. 2007)
» ”Select the ones ’surely’ in top 50”
» ”Discard the ones ’surely’ outside top 50”
INFORMS Annual Meeting, Washington DC 2008
13
Helsinki University of Technology
Systems Analysis Laboratory
References
» Butler, J., Jia, J., Dyer, J. (1997). Simulation Techniques for the Sensitivity Analysis of
Multi-Criteria Decision Models. EJOR 103, 531-546.
» Keeney, R.L., See, K.E., von Winterfeldt, D. (2006). Evaluating Academic Programs: With
Applications to U.S. Graduate Decision Science Programs. Oper. Res. 54, 813-828.
» Kirkwood, G., Sarin R. (1985). Ranking with Partial Information: A Method and an
Application. Oper. Res. 33, 38-48
» Könnölä, T., Brummer, V., Salo A. (2007). Diversity in Foresight: Insights from the Fostering
of Innovation Ideas. Technologial Forecasting & Social Change 74, 608-626.
» Liesiö, J., Mild, P., Salo, A., (2007). Preference Programming for Robust Portfolio Modeling
and Project Selection. EJOR 181, 1488-1505.
» Liu, N.C., Cheng, Y. (2005). The Academic Ranking of World Universities. Higher
Education in Europe 30, 127-136
» Mustajoki, J., Hämäläinen, R.P., Lindstedt, M.R.K. (2006). Using intervals for Global
Sensitivity and Worst Case Analyses in Multiattribute Value Trees. EJOR 174, 278-292.
» Rios Insua, D., French, S. (1991). A Framework for Sensitivity Analysis in Discrete MultiObjective Decision-Making. EJOR 54, 176-190.
» Salo, A., Hämäläinen R.P. (1992). Preference assessment by imprecise ratio statements.
Oper. Res. 40, 1053-1061.
INFORMS Annual Meeting, Washington DC 2008
14