Computers & Operations Research 49 (2014) 28–40
Contents lists available at ScienceDirect
Computers & Operations Research
journal homepage: www.elsevier.com/locate/caor
Learning criteria weights of an optimistic ELECTRE TRI sorting rule
Jun Zheng a, Stéphane Aimé Metchebon Takougang b, Vincent Mousseau a,n, Marc Pirlot c
a
b
c
Laboratoire Génie Industriel, Ecole Centrale Paris, Grande Voie des Vignes 92 295 Châtenay-Malabry, France
Institut Supérieur d'Informatique et de Gestion, ISIG-International, 06 BP 9283 Ouagadougou 06, Burkina Faso
UMONS, Faculté Polytechnique de Mons, 9 Rue de Houdain, 7000 Mons, Belgium
art ic l e i nf o
a b s t r a c t
Available online 28 March 2014
Multiple criteria sorting methods assign alternatives to predefined ordered categories taking multiple
criteria into consideration. The ELECTRE TRI method compares alternatives to several profiles separating
the categories. Based on such comparisons, each alternative is assigned to the lowest (resp. highest)
category for which it is at least as good as the lower profile (resp. is strictly preferred by the higher
profile) of the category, and the corresponding assignment rule is called pessimistic (resp. optimistic).
We propose algorithms for eliciting the criteria weights and majority threshold in a version of the
optimistic ELECTRE TRI rule, which raises additional difficulties w.r.t. the pessimistic rule. We also describe
an algorithm that computes robust alternatives' assignments from assignment examples. These
algorithms proceed by solving mixed integer programs. Several numerical experiments are conducted
to test the proposed algorithms on the following issues: learning ability of the algorithm to reproduce
the DM's preference, robustness analysis and ability to identify conflicting preference information in case
of inconsistencies in the learning set. Experiments show that eliciting the criteria weights in an accurate
way requires quite a number of assignment examples. Furthermore, considering more criteria increases
the information requirement. The present empirical study allows us to draw some lessons in view of
practical applications of ELECTRE TRI using the optimistic rule.
& 2014 Elsevier Ltd. All rights reserved.
Keywords:
Multiple criteria sorting
Preference learning
ELECTRE TRI
Optimistic rule
1. Introduction
Multiple criteria sorting methods have gained wide interest in
the last few decades. Such methods aim at assigning alternatives
to pre-defined ordered categories taking into account several
criteria. Multiple criteria sorting methods differ from standard
classification in two main features: (1) categories are predefined
and ordered, (2) the sorting model integrates preferences of a
Decision Maker (DM). Various methods have been proposed in the
literature (see [35] for a review). Some methods are based on the
use of value functions (e.g. [13,34]); some others involve the use of
outranking relations (e.g. [21,17,33,29,27,1]); others make use of
decision rules (e.g. [12]).
The multiple criteria sorting models require to set many
parameter values, which reflect the DM's value system. A preference elicitation process is therefore needed. In a direct aggregation paradigm, the parameter values are supposed to be provided
by the DM through interactive communication with the analyst.
n
Corresponding author.
E-mail addresses: [email protected] (J. Zheng),
[email protected] (S.A. Metchebon Takougang),
[email protected] (V. Mousseau), [email protected] (M. Pirlot).
http://dx.doi.org/10.1016/j.cor.2014.03.012
0305-0548/& 2014 Elsevier Ltd. All rights reserved.
The multiple criteria assignment model encompassing the DM's
comprehensive preference is first constructed and then applied to
the set of alternatives. Such a paradigm imposes a strong cognitive
effort to the DM who needs to understand the meaning of and set
appropriate values to the preference related parameters. In a
disaggregation paradigm (see [15] for a review, and [4, Chapter 7]),
the sorting model is inferred from a priori preferences expressed by
the DM. The initial proposal of a disaggregation method called UTA
was made by Jacquet-Lagrèze and Siskos [14] in the context of
ranking problems. For additive value based sorting models, Doumpos
and Zopounidis [11] present a disaggregation method to elicit a
UTADIS model as a variant of the UTA model for sorting purposes.
In the context of rule-based models, the disaggregation method
proposed in [12] infers a set of “if…then…” decision rules derived
from rough sets based approximations of decision examples.
In this paper, we are interested in the disaggregation methodology for the ELECTRE TRI method [33,29]. ELECTRE TRI is a well
known sorting method based on an outranking relation. Many
applications have been reported (e.g. [24,30,18]). The ELECTRE TRI
model involves several parameters including profiles that define
the limits between the categories, criteria weights, discrimination
thresholds, etc. ELECTRE TRI requires to select one of the two
assignment rules proposed (optimistic or pessimistic).
J. Zheng et al. / Computers & Operations Research 49 (2014) 28–40
Several authors have presented disaggregation methodologies
to infer a pessimistic ELECTRE TRI rule from assignment examples
provided by the DM [25,7,8,26]. These methodologies propose to
infer the preference parameters that best match the DM's preference information and to compute a robust assignment, i.e. the
range of categories to which an alternative can be assigned,
considering all combinations of the parameters values compatible
with the DM's preference statements. These disaggregation procedures lean on linear programming. Recently, an evolutionary
approach has been presented considering both the optimistic and
the pessimistic rules to infer the parameters of an ELECTRE TRI
model [10]. However, all elicitation procedures (except for [10])
deal with the pessimistic rule only. The literature provides no
mathematical programming formulation for the elicitation of an
optimistic ELECTRE TRI rule, even though in practice, some applications require using the optimistic rule [20,19].
Recent experimental analysis of elicitation procedures, even for
the pessimistic ELECTRE TRI model, has shown (e.g. [16,23]) that the
number of assignment examples needed for specifying the parameters of the model with reasonable accuracy can be rather large
and grows rapidly with the numbers of criteria and categories.
Actually, learning methods for multicriteria sorting models, such as
ELECTRE TRI, can be used in two very different contexts: decision aiding
and machine learning. In the practice of the former, the number of
assignment examples usually is very limited, a few dozens at best.
In such a context, using a rule involving many parameters for
learning purposes is inadequate since it leaves us with a highly
undetermined situation, a large variety of models being compatible
with the assignment examples. In machine learning, in contrast, one
deals with large or huge sets of assignment examples (at least several
hundreds). In such a case, the algorithms designed for eliciting all or
part of the parameters of a classical ELECTRE TRI model are far too slow
and complex. In both cases, while for very different reasons, we
argue that studying the elicitation of a version of the ELECTRE TRI rule
using fewer and less inter-related parameters is a fruitful option. The
fact that such a rule is better understood thanks to the axiomatic
work in [2,3] is also an asset. An exploration of the learning and
expressive capabilities of such a procedure started with [16] using
mixed integer linear programming formulations. In [31], the learning
of all parameters of an ELECTRE TRI – like learning rule (called MRSort,
for Majority Rule Sorting) is done by means of a specific metaheuristic. The method allows to deal with learning sets containing several
hundreds of assignments. Both [16,31] deal with the pessimistic
version of the assignment rule. In this paper, we explore the
elicitation of a simple ELECTRE TRI rule in its optimistic version, without
veto. Working with the optimistic rule raises specific difficulties.
Even the mathematical programming formulation for the determination of the criteria weights, the limit profiles being given, requires the
use of binary variables. Therefore this paper only deals with this
issue. Note that a method for learning the criteria weights, assuming
known profiles, can already help a lot in practice since the DM can
often be directly questioned about profiles. The latter have a
relatively clear interpretation in such models.
When proposing a learning procedure it is crucial to perform
numerical experiments that guarantee its practical usefulness.
A basic requirement (seldom verified) is that it is possible to learn
the “true model” whenever the data in the learning set have been
assigned using this model. A related issue is the size of the
learning set needed to retrieve approximately the true model.
Alternatively, for a given size of the learning set, one can assess the
degree to which the model remains undetermined by recording
the average number of categories an alternative can be assigned to
by the learned models. Another important question is whether it is
possible to learn the model when noise has been added to the
assignments in the learning set. Indeed as DMs cannot be expected
to always provide reliable and consistent information, the learning
29
procedure should be as robust as possible with respect to the presence
of erroneous assignments in the learning set. Finally comes the
question of the expressiveness of the model, i.e. its capability to
reproduce assignments made according to an unknown rule (artificial
or real data sets). In this study we will not try to assess the
expressiveness of the model but we will address the other issues in
depth. The reason for this is that we concentrate here on the elicitation
of the criteria weights in an optimistic ELECTRE TRI rule. The profiles are
considered as given. This makes it difficult to study the expressiveness
of the model since we did not develop tools for identifying appropriate
profiles on the basis of a learning set generated by an arbitrary rule.
Even in the case the assignments have been produced by means of a
general ELECTRE TRI model (involving for instance indifference and
preference thresholds) it is not clear whether using the same profiles
in our simpler optimistic rule would be an optimal strategy. The study
of the expressiveness of the simple ELECTRE TRI sorting rule (either in its
optimistic or pessimistic version) is complex enough to deserve an in
depth study in a separate paper. The introduction of vetoes and their
impact both on the determination of the other parameters as well as
the gain in expressiveness they will provide is also worth studying.
This paper has two main objectives:
Firstly, it establishes mathematical programming formulations
for the elicitation of the criteria weights in an optimistic ELECTRE
TRI rule. Our formulations allow us to deal with “assignment
errors” in the learning set.
Secondly, these formulations are tested in order to assess their
practical usefulness in an interactive elicitation process, in
particular their sensitivity to assignment errors.
The paper is organized as follows. Section 2 briefly introduces the
ELECTRE TRI method. Section 3 presents the mathematical programming formulations for eliciting the weights of the criteria in the
optimistic ELECTRE TRI rule, computing robust assignments and dealing
with infeasible learning sets. In Section 4, extensive numerical
experiments are designed to test the computational behavior and
study the learning capabilities of the proposed algorithms. The last
section groups conclusions and issues for further research.
2. The ELECTRE TRI method
ELECTRE TRI assigns the alternatives in a set A to one out of p predefined ordered categories C 1 ; C 2 ; …; C p . We assume w.l.o.g. that
categories are ordered from the worst to the best, i.e. that C1
represents the lowest quality level and Cp the highest, while
C 2 ; …; C p 1 are intermediate levels ranked in increasing order of
their quality. We denote by C Z h , the set of categories Ch to Cp. One
defines similarly the sets of categories C 4 h ; C r h and C o h . Each
alternative is assessed w.r.t. m criteria. Let ðg 1 ðaÞ; …; g m ðaÞÞ denote
the vector of evaluations of alternative a on the m criteria.
P denotes the set f1; 2; …; pg and M, the set f1; 2; …; mg.
In the ELECTRE TRI method, the assignment of an alternative a to a
category results from the comparison of a with the profiles
defining the “category limits”. To each category Ch are assigned
h1
(the latter
an upper limit profile bh and a lower limit profile b
also being the upper limit of category C h 1 and the former, the
lower limit of category C h þ 1 , provided such categories exist).
A profile is just a vector of reference assessments on all criteria,
h
h
h
h
i.e. b ¼ ðb1 ; …; bj ; …; bm Þ. The lower limit profile b0 of category C1
and the upper limit bp of category Cp are trivial profiles. Profile
0
0
0
0
b ¼ ðb1 ; …; bj ; …; bm Þ has the worst possible performance on all
criteria, so that any alternative is at least as good as b0. Symmep
p
p
p
trically, b ¼ ðb1 ; …; bj ; …; bm Þ has the best possible performance
on all criteria so that no alternative is better than bp.
30
J. Zheng et al. / Computers & Operations Research 49 (2014) 28–40
2.1. Non-compensatory sorting models
The assignment of an alternative to a category is based on rules
in which the vector of evaluations associated with the alternative
is compared to the upper and/or lower limit profile of the category.
In the spirit of the non-compensatory sorting methods characterized by Bouyssou and Marchant [2,3], an alternative a is considered at least as good as profile bh if the coalition of criteria on
which a receives evaluations at least as good as the profile is
“strong enough”. A non-veto condition may be added but we do
not consider such conditions in this study.
Formally, one defines the set of strong enough coalitions of
criteria as a subset F of the power set 2M. The only property that
we need to impose on F is that it is stable, from above, for
inclusion (a coalition containing a strong enough coalition is
strong enough). Typically, the set of strong enough coalitions is
determined by a weighted majority rule as we shall see below. For
comparing a with bh we define the coalitions
h
h
Sða; b Þ ¼ fj A M : g j ðaÞ Z bj g
h
ð2Þ
We define the outranking relation ≿ between an alternative a and a
profile bh, and conversely, as follows:
a≿b
h
if
Sða; b Þ A F
h
ð3Þ
b ≿a
h
if
Sðb ; aÞ A F :
ð4Þ
h
The asymmetric part g of the relation ≿ is interpreted as a strict
preference (better than relation). One has
a ≻ b if a≿b
h
and
h
a ≻ b if Sða; b Þ A F
h
Not½b ≿a
ð5Þ
h
ð6Þ
Sðb ; aÞ2
=F;
and
h
b ≻ a is defined similarly. The indifference relation is the
symmetric part of ≿. Since the relation ≿ usually is not complete,
h
the incomparability relation a ? b is defined as the symmetric
complement of ≿, i.e.,
h
h
a ? b if Not½a≿b h
h
a ? b if Sða; b Þ2
=F
h
Not½b ≿a
and
ð7Þ
h
Sðb ; aÞ2
=F:
and
and
a≿b
h1
ð13Þ
It has been shown [29], and this is easy to check using the
above definitions, that the category assigned to any alternative by
the optimistic rule is guaranteed to be at least as high as that
assigned by the pessimistic rule.
The simple version of the pessimistic rule of ELECTRE TRI described
above is in line with the work of Bouyssou and Marchant [2,3]. Our
definition of the optimistic rule is in the same spirit, yet it has not been
characterized in [2,3]. Note also that the definition of the outranking
relation in (3) and (4) does not involve any non-discordance condition
(see [28,29]). In this work, we focus on the analysis of the elicitation of
the optimistic ELECTRE TRI rule without veto.
2.2. Introducing weights
h
h
h
a assigned to C h if Not½a≿b ð1Þ
Sðb ; aÞ ¼ fj A M : bj Zg j ðaÞg
h
Pessimistic assignment rule: With this rule, alternative a is assigned to
a category strictly below C h þ 1 if a is not at least as good as the lower
h
limit profile bh of C h þ 1 . Formally, we have a A C o ðh þ 1Þ if Not½a≿b .
It is thus assigned to Ch if C h þ 1 is the lowest category such that
h
Not½a≿b , i.e.
ð8Þ
As already mentioned, it is often assumed in practice that the
set of strong enough coalitions F can be represented by a set of
nonnegative weights wj associated with the criteria gj with jA M
and a majority rule. More precisely, we shall assume that there
exist weights wj Z 0 with ∑m
j ¼ 1 wj ¼ 1 and a majority level λ, with
0:5 r λ r 1, such that
F AF
iff ∑ wj Z λ:
ð14Þ
jAF
In words, a coalition F is strong enough if the sum of the weights of
the criteria belonging to F reaches at least some majority threshold λ.
In order to avoid degenerate cases in which an evaluation inferior to
that of a profile on a single criterion can prevent an alternative to be
declared at least as good as a profile (or conversely), we impose that
no criterion weight may exceed 12 . Such a condition has already been
used previously in the literature so as to avoid dictatorship among a
group of DMs (see [9]), and to forbid a single criterion to be decisive
alone (see [25,8]).
h
h
The weight cða; b Þ of a coalition Sða; b Þ is defined as the sum of
h
the weights of the criteria that belong to it: cða; b Þ ¼ ∑j A Sða;bh Þ wj .
h
A similar definition holds for the weight cðb ; aÞ of the coalition
h
Optimistic assignment rule: Using the optimistic rule, alternative a
is assigned to category Ch or higher if the lower limit profile of
category Ch is not strictly preferred to a. Formally,
a assigned to C Z h if Not½b
a assigned to C Z h iff Not½b
h1
h1
h1
≻ a
ð9Þ
h1
≿a or a≿b
:
ð10Þ
Hence the precise category which a is assigned to is Ch if the latter
is the highest one satisfying the previous condition, i.e.
h1
a assigned to C h if Not½b
≻ a
h1
a assigned to C h iff ½Not½b
h
and b ≿a
and
Sðb ; aÞ. Using these weights, the optimistic and pessimistic assignment rules can be rephrased as follows.
Optimistic assignment rule:
h
b ≻a
and
h1
≿a or a≿b
ð11Þ
h
Not½a≿b :
ð12Þ
Paraphrasing this somewhat awkward, yet understandable, condition, we say that a is assigned to the category of which
the lower limit profile is not strictly better than a
and the upper limit profile is strictly better than a.
a assigned to C h if ½cðb
h
and cðb ; aÞ Z λ
and
h1
; aÞ o λ or cða; b
Þ Z λ
h
cða; b Þ oλ:
ð15Þ
Pessimistic assignment rule:
h
a assigned to C h if cða; b Þ o λ
and
h1
cða; b
Þ Z λ:
ð16Þ
2.3. Learning an ELECTRE TRI sorting rule
In the above definitions, instead of considering the general
ELECTRE TRI optimistic and pessimistic rules, which use the ELECTRE III
definition of the concordance–discordance index (see [29]), we
choose to work with the ELECTRE I version of the concordance index,
without using veto, which amounts to compare alternatives and
profiles using a majority rule. Moreover, this paper only deals with
the elicitation of the criteria weights and majority threshold in the
optimistic ELECTRE TRI sorting rule. Why do we focus on such an
apparently limited objective?
J. Zheng et al. / Computers & Operations Research 49 (2014) 28–40
First of all, let us observe that a method for eliciting the criteria
weights, while the profiles are assumed to be known, is often
useful in practice, be it for the pessimistic or the optimistic rule.
For instance, in the application dealt with in Metchebon's doctoral
dissertation [19], the profiles are spontaneously provided by the
expert, so that the inference of the weights arises as the natural
next step.
More generally, it is advisable to approach the elicitation of the
(numerous) parameters of an ELECTRE TRI rule in a stepwise manner.
This allows us to control the interaction of the different categories
of parameters (profiles, weights, veto thresholds). Learning all
parameters at a time has two major drawbacks. It is often an
intractable problem (due to the rapidly growing number of binary
variables involved) and/or its solution is largely undetermined.
Both problems were specifically studied for the pessimistic version
of the simplified version of the ELECTRE TRI method (without veto),
called MR-Sort (see [16]). A conclusion that forcefully emerges
from this experimentation is that the simplified pessimistic rule
based on MR-Sort is already quite expressive in the sense it allows
not only to represent the learning set assignments when they are
noisy but also when the assignments have been generated by
another model (i.e. a rule based on an additive value function and
some thresholds). Due to the additional complexity of the mathematical program required by the optimistic assignment rule, it
was unrealistic, both from a practical and a methodological viewpoint, to try to learn the profiles and the weights at the same time.
Hence the option we have taken in the present study.
Having chosen to only learn the weights, the profiles being
given, our experimentation shows e.g. that the number of assignment examples to be used to learn a model with 9 criteria and
4 categories with an accuracy of 90% requires more than 30
examples when the learning set is noiseless (no assignment
“errors” w.r.t. to an unknown ELECTRE TRI model). Not to mention
the computational effort, eliciting the 3 profiles at the same time
would require the determination of about four times as many
variables. This means that the size of the learning set needed to
obtain a reasonable accuracy would clearly be at least 100
examples. Yet the variety of models providing approximately the
same accuracy would be huge. This type of difficulty is amplified
for noisy learning sets (with assignment errors), as shown in our
experimentations.
3.1. Mixed integer linear programming formulation
We use a regression-like technique to learn the parameters of
such a model. The preference information (in the form of assignment examples) can be represented by linear constraints involving
continuous and binary variables. A set of values for the parameters
of the model can be obtained by solving a mixed integer program
(MIP) whose feasible solutions satisfy these constraints.
Such an approach has been previously followed for eliciting the
parameters of an ELECTRE TRI model with the pessimistic rule [23,8].
However, when considering the optimistic rule, the conditions (15)
for assigning alternative a to category Ch appear as a conjunction of
conditions, one of them being a disjunction:
h1
cðb
; aÞ o λ
h1
or cða; b
ð17Þ
In the above, inequality δ1 þ δ2 r1 ensures that at least one of the
two variables is set to 0, which entails that at least one of the two
clauses in (17) is true.
We aim at eliciting the parameter values of an ELECTRE TRI model
which can restore the DM's assignment examples by using the
optimistic rule. Provided such a model exists, the linear program
below yields the values of these parameters. In the sequel, for each
alternative ae ; eA L in the learning set An, we denote by C he the
category to which ae is known to be assigned.
α
max
ð19Þ
h
cðb e ; ae Þ βe ¼ λ;
s:t:
h
he 1
We consider the optimistic assignment rule (15), assuming that
the limits of the categories are given. Our aim is to elicit the
h
h
criteria weights wj involved in cða; b Þ and cðb ; aÞ and the majority
level λ on the basis of comprehensive preference statements made
by the DM. The preference information consists of a number of
assignment examples, which involve alternatives that the DM is
able to assign to a category in a holistic way, i.e. without having to
analyze their performances. We consider here a static set of
assignment examples, i.e. we assume that all the information is
available at the beginning of the learning process; in other words,
the DM cannot be questioned to provide adequate or critical
information in the course of the learning phase. The set of learning
examples is denoted by An ¼ fa1 ; …; ae ; …; al g, where l is the
number of alternatives in the learning set. We note L ¼ f1; …;
e; …; lg. The assignment to a category of each alternative in An is
supposed to be known. These examples will be used as a training
set to establish an ELECTRE TRI model which can reproduce these
assignment examples. The obtained model can then be applied to
assign other alternatives to categories.
Þ Z λ:
Modeling this condition by a system of linear constraints can be
done in a standard way by introducing two binary variables δ1, δ2
(see, e.g. [32]). Their values indicate, for each of the clauses in the
disjunctive condition, whether it is satisfied or not. More precisely,
h1
h1
δ1 (resp. δ2) equals 0 if cðb
; aÞ o λ (resp. cða; b
Þ Z λ) and
equals 1 otherwise. Condition (17) holds iff the following constraints are satisfied for some value of the positive variable ϵ:
8
h1
>
cðb
; aÞ λ δ1 þ ϵ r0
>
>
>
<
h1
Þ λ þδ2 Z 0
cða; b
ð18Þ
>
;
δ
A
f0;
1g
δ
>
1 2
>
>
:
δ1 þ δ2 r1:
cðae ; b e Þ þ γ e þϵ ¼ λ;
3. Eliciting the weights and majority threshold
of the optimistic assignment rule
31
cðb
ð20Þ
8eAL
ð21Þ
; ae Þ λ δe1 þ ηe þ ϵ ¼ 0;
he 1
cðae ; b
8eAL
Þ λ þ δe2 μe ¼ 0;
δe1 þ δe2 r 1;
8eAL
δe1 ; δe2 A f0; 1g;
8 eA L
8 eA L
8eAL
ð22Þ
ð23Þ
ð24Þ
ð25Þ
α r βe ;
8 eA L
ð26Þ
α r γe;
8eAL
ð27Þ
α r ηe ;
8eAL
ð28Þ
α r μe ;
8 eA L
ð29Þ
∑ wj ¼ 1;
jAM
0:5 r λ r 1
wj Z 0; 8 j A M
ð30Þ
ð31Þ
In this program, the constraints (20)–(25) express condition (15)
for each ae in An. In these constraints βe, γe, ηe and μe are the slack
variables, and ϵ is a small positive value used to ensure that the
appropriate inequalities are strict. The objective function consists
32
J. Zheng et al. / Computers & Operations Research 49 (2014) 28–40
in maximizing the minimal value of the slack variables: constraints
(26)–(29) express that α is not larger than the least slack variable.
The usual constraints on the weights and the cutting level λ
are expressed in (30) and (31). Since we have assumed that the
h
category limits are known, the constraints in which cðae ; b e Þ or
he
cðb ; ae Þ intervene are linear.
If the previous program is feasible and its optimal value αn is
non-negative, then there exists a combination of parameter values
that satisfy all the constraints in the program and, consequently,
there is an ELECTRE TRI model which assigns the alternatives in An as
prescribed. Otherwise, either the DM should reconsider his/her
statements or another type of assignment model should be used.
In case αn is negative, an inconsistency resolution algorithm
(see [22]) can be run to help the analyst identify the inconsistencies in the DM's statements w.r.t. the model considered.
3.2. Robustness analysis
As the preference information in the form of assignment
examples is represented by linear constraints, the set of feasible
values for the weights satisfying the constraints is a (possibly
empty) polytope. Whenever the latter is not empty, there are
generally an infinite number of sets of weights compatible with
the preference information. The objective function (19) in the MIP
(19)–(31) helps us to select a particular set of weights in the
polytope of feasible weight vectors, but this choice is rather
arbitrary. By varying the weights in the feasible weight vectors
set, an alternative a can often be assigned to more than one
category. Finding the set of all categories to which an alternative
can be assigned, given the information available, is the purpose of
robustness analysis [8]. In this context, we are interested in the
following question: “Does there exist a combination of parameters
which would lead alternative a to be assigned to category Ch?”.
Answering this question for all h is tantamount to solving the
problem of computing the robust assignment of a.
This is done by solving the following MIP (32)–(46). Alternative
a is an alternative of interest selected from A, the set of all
considered alternatives. We impose as a constraint that a is
assigned to category Ch and we search for feasible weight vectors
(i.e. for which this constraint is fulfilled as well as all constraints
relative to the assignment of the alternatives in the learning
set An).
The mathematical program to be solved to compute whether a
can possibly be assigned to category Ch runs as follows:
ε
max
ð32Þ
h
s:t:
cðb ; aÞ Z λ
ð33Þ
h
ð34Þ
cða; b Þ r λ ε
h1
cðb
cða; b
; aÞ r λ þ δ1 ε
h1
ð37Þ
δ1 ; δ2 A f0; 1g
ð38Þ
h
8eAL
cðae ; b e Þ r λ ε;
he 1
cðb
∑ wj ¼ 1;
jAM
8eAL
ð44Þ
wj Z 0; 8 j A M
ð45Þ
0:5 r λ r 1
ð46Þ
The objective function (32), to be maximized, is the variable ε
that is used for transforming strict inequalities into the non-strict
ones (it was a constant in the MIP (19)–(31), here it is a variable).
Constraints (33)–(38) express that a is assigned to Ch; constraints
(39)–(44) enforce that the model correctly assigns all the alternatives in the learning set An; (45)–(46) impose the usual restrictions on the weights and the threshold λ. If the assignment of a to
Ch is compatible with the preference information provided by the
DM, the above program is feasible and its optimal value εn is
strictly positive.
Computing the robust assignment of an alternative a consists in
repeatedly using the above MIP to check whether a can be
assigned to category Ch, for each h ¼ 1; 2; …; p. Let Ra denote the
set of categories Ch which a can be assigned to, i.e. the set of
categories Ch for which the above program has a positive optimal
solution εn . jRa j denotes the number of different categories in Ra .
Computing Ra can be performed by using Algorithm 1.
Remark on possible improvements in the search of a robust
assignment: As observed in [8] for the pessimistic ELECTRE TRI rule,
there may exist “gaps” in the set Ra of possible assignment classes
for alternative a. In other words, it may happen that Ra contains
classes Ch and Cl with h ol but not the class Cj with ho j o l. The
same phenomenon appears with the optimistic rule. Consider the
following example of an alternative a assessed on four criteria; the
relative positions of these assessments (gi(a)) w.r.t. two profiles
1
2
b ; b determining three categories are shown in Fig. 1.
We have
1
for i ¼ 1; 2
2
for i ¼ 3; 4:
g i ðaÞ obi
g i ðaÞ 4bi
It is easy to see that such an alternative is never assigned to
category C2 (neither by the optimistic nor by the pessimistic
ELECTRE TRI rule) whatever the weights and threshold can be. Let
us show it for the optimistic rule. Using (15), we see that a is
2
2
assigned to class C2 if and only if cða; b Þ oλ and cðb ; aÞ Zλ and
1
1
½cða; b Þ Zλ or cðb ; aÞ oλ. With an alternative as illustrated in
2
1
2
1
Fig. 1, we have cða; b Þ ¼ cða; b Þ and cðb ; aÞ ¼ cðb ; aÞ, which makes
ð39Þ
8eAL
; ae Þ rλ þδe1 ε;
he 1
cðae ; b
δe1 ; δe2 A f0; 1g;
ð43Þ
ð36Þ
δ1 þδ2 r 1
h
8eAL
ð35Þ
Þ Z λ δ2
cðb e ; ae Þ Z λ;
δe1 þ δe2 r 1;
Þ Zλ δe2 ;
ð40Þ
8 eA L
8eAL
ð41Þ
ð42Þ
Fig. 1. Example of an alternative that can possibly be assigned to categories C1 or C3
but not C2.
J. Zheng et al. / Computers & Operations Research 49 (2014) 28–40
it impossible to satisfy the conditions for assigning the alternative
to C2. In contrast, it is clear that for some values of the weights and
the threshold, such an alternative can be assigned to C3 (in case
the last two criteria form a sufficient coalition) or C1 (in case the
first two criteria form a sufficient coalition). The previous argument holds, in a general case, for any alternative having at least
h1
one assessment above bh and one below b
, but none between
h1
h
b and b
.
The algorithm we propose to compute Ra checks whether
alternative a can be assigned to class Ch for all values of h. Since the
presence of gaps in Ra cannot be excluded, it is not advisable to
blindly apply a dichotomic search procedure to speed up the
computation of Ra . Observe however that, in practice, the number
p of categories is usually quite small (p is seldom greater than 5) so
that solving the 0–1 program in Algorithm 1 p times instead of
log 2 ðpÞ times is not an issue.
Note also that the intuition that Ra should not involve gap is
clearly linked with the prevalence of compensatory models (such
as weighted sums and additive value functions) in Decision
Analysis. If compensation is allowed, it is natural to consider that
an alternative such as illustrated in Fig. 1 could be assigned to C2.
On the contrary, in a non-compensatory approach (such as ELECTRE
TRI), either criteria 3 and 4 form a “majority”, and a is assigned to
C3, or they do not, and then the assessments of a on criteria 1 and
2 will prevail; therefore a will be assigned to C1. This is a striking
illustration of the difference between compensatory and noncompensatory preferences; there are clearly examples of both in
real-life situations.
Algorithm 1. Procedure to compute the robust assignment of an
alternative.
Input:
The set of assignment examples: ae -C he , for all e A L;
The performance vector of the alternatives ae in the learning
set An: ðg j ðae Þ; j A MÞ, for all eA L;
The performance vector of alternative a: ðg j ðaÞ; jA MÞ;
h
The performance vector of each limit profile bh: ðbj Þ; j A M,
for h ¼ 1; 2; …; p 1
Output:
Robust assignment for alternative a: Ra ;
1: Ra ’∅
2: for h¼ 1 to p do
3:
write the constraints corresponding to a-C h in MIP
solve the program
4: if εn 40 then
5:
Ra ’Ra [ fhg
6:
end if
7: end for
3.3. Infeasible learning sets
In the course of a decision aiding process, inconsistency
situations can occur when there is no ELECTRE TRI model which
matches all the DM's preference information. The algorithms
described in Mousseau et al. [22] have been designed to help the
analyst to identify the conflicting pieces of information in the
statements made by the DM. The algorithm identifies and suggests
a subset of assignment examples which can be removed to make
the remaining assignments consistent. However, the algorithm in
Mousseau et al. [22] is dedicated to the resolution of inconsistencies arising when using the pessimistic ELECTRE TRI rule. We adapt
this algorithm to the case of the optimistic rule. Additional binary
variables are introduced to model the constraints stemming from
assignment examples.
33
More precisely, for each ae A L, we introduce a binary variable γe
which is equal to one if alternative ae is correctly assigned by the
sorting model, and equal to zero otherwise. Given a set of
inconsistent assignment examples, the following MIP computes
the maximum number of them, which can be restored by an
optimistic ELECTRE TRI model. The first four constraints impose the
required conditions on the γe variables
∑ γe
max
ð47Þ
eAL
h
cðb e ; ae Þ Z λ Mð1 γ e Þ;
s:t:
h
cðae ; b e Þ rλ ε þ Mð1 γ e Þ;
he 1
cðb
8eAL
8eAL
; ae Þ r λ þ δe1 ε þ Mð1 γ e Þ;
h
cðae ; b e Þ Zλ δe2 þ Mð1 γ e Þ;
δe1 þ δe2 r 1;
∑ wj ¼ 1;
jAM
ð49Þ
8eAL
8eAL
8eAL
δe1 ; δe2 ; γ e A f0; 1g;
wj Z 0;
ð48Þ
ð50Þ
ð51Þ
ð52Þ
8eAL
8jAM
0:5 r λ r 1
ð53Þ
ð54Þ
ð55Þ
The objective function of this MIP guarantees that we obtain a
subset of alternatives of maximal cardinality in the learning set
that can be represented by the sorting model. This model will be
used in the third experiment (Section 4.2.3).
4. Experimental design and results
In this section, we analyze the behavior of the proposed
elicitation algorithm by means of numerical examples. Our aim
is to check the empirical validity of the approach, to validate the
practical usefulness and to provide guidance to the analyst who
would like to use this procedure.
In order to infer in a reliable way a weight vector, the
optimization procedure requires information as input, i.e. on the
set of assignment examples. What is the amount of information
necessary to “calibrate” the model in a satisfactory way? How
large should be An in order to derive the weights in a reliable
manner? This question is essential for practical use to the
inference model in real world decision problems. The analyst
should have some simple guidelines to manage the interaction
with the DM avoiding unnecessary questions, but collecting a
sufficient information.
In practical decision situations, real DMs do not always provide
reliable information. Due to time constraints and cognitive limitations, DMs express contradictory information, their preferences
change over time. The optimization procedure should be able to
highlight the assignment examples that are contradictory or not
representable through the ELECTRE TRI preference model. This
experiment aims at investigating the ability of the tool to “identify” the inconsistencies in the DM's statements in order to help
him/her in revising the preference information. How reliable is the
optimization procedure to identify inconsistencies in the DM's
judgments?
4.1. Experimental scheme
The experiments are designed to address three issues: (1) the
learning ability of the elicitation algorithm; (2) the behavior and
performance of the robustness analysis algorithm; (3) the ability
to deal with inconsistent information. Moreover, some factors
34
J. Zheng et al. / Computers & Operations Research 49 (2014) 28–40
possibly influencing the performance of the algorithms are investigated, including the number of assignment examples, the number of criteria, and the number of categories. Similar experiments
concerning the learning ability and inconsistency identification for
ELECTRE TRI with the pessimistic rule can be found in [16].
We consider the following experimental scheme to answer the
above questions.
We implement a fictitious DM whose preferences are represented by an optimistic ELECTRE TRI model (called the original
model). The model is characterized by the profiles bh
ðh ¼ 1; 2; …; p 1Þ, a set of weights wj ðj ¼ 1; 2; …; mÞ and the
majority level λ. The randomly generated original model assigns
randomly generated alternatives to categories, and then these
alternatives together with their assignments are used to infer an
ELECTRE TRI model (called the inferred model). We are interested in
how “close” the original model and the inferred model are.
More precisely,
1. Firstly, a set An of l alternatives is generated; the evaluations of
these alternatives are randomly drawn from the interval [0, 99]
according to a uniform distribution.
2. Then the original model is generated randomly. The weight
vectors are drawn randomly from the uniform distribution on
the simplex in Rm . This is done in the following standard way.
The sequence of m 1 random numbers are drawn uniformly
from the [0, 1] interval. They are labeled in increasing order of
their value as follows: x1 r x2 r ⋯ rxm 1 . The uniformly distributed derived weight vector is ðx1 ; x2 x1 ; x3 x2 ; …; xm 1 xm 2 ; 1 xm 1 Þ. Such a procedure is a standard one to obtain
randomly distributed weights, see [5]. It is grounded on the
distribution of the order statistics of the uniform random
distribution in the unit interval, see [6].
3. The majority threshold λ is randomly drawn from the [0.5, 1]
interval.
4. We generate p profiles by partitioning the [0, 99] interval into
p þ 1 equal intervals. Note that proceeding in this way leads to
classes of similar “size”; an alternative option would be to draw
the profiles at random; it is difficult to assess the impact of this
choice on the result. During the elicitation process, the profiles
are considered as known and are those of the original model.
5. Original assignments are compared to inferred assignments
(see Fig. 2). To measure the closeness of the original and the
inferred model, we also generate a large set A of test alternatives. To assess how close the original and the inferred
models are, we count the number of alternatives that are
assigned to the same category by both models. Such alternatives are called congruent. The proportion of congruent alternatives in the test set is what we will call model accuracy.
A set of experiments are designed varying the complexity of
the original model, i.e. the number of criteria and the number of
categories. We also test the algorithms with different amounts of
preference information, i.e. numbers of assignment examples. The
parameter settings of the experiments are shown in Table 1.
Our experimental strategy aims mainly at observing the effect
of varying the learning set size. A secondary objective is to study
the effect of other variables describing the dataset, namely, the
number of criteria, the number of categories and, for noisy
datasets (cf. Section 4.2.3), the proportion of wrong assignments
in the learning set. To do so, we vary individually these three
variables (setting the other two to “average” values).
4.2. Experiments and results
4.2.1. Learning ability
Experiments: We study the ability of the elicitation algorithm to
retrieve the original ELECTRE TRI model. It cannot be expected that
the model can be accurately inferred when little input information
is provided. In other words, if the number of assignment examples
is limited, there should exist many ELECTRE TRI models compatible
with the assignment examples, which means that the inferred
model is almost arbitrary. It is expected that increasing the
Table 1
Parameter settings for the experimental design.
Parameters
Values considered
Number of test alternatives (set A)
Number of assignment
examples (set An)
10 000
5, 10, 15, 20, 30, …, 100
(Sections 4.2.1 and 4.2.2)
20, 30, …, 100 (Section 4.2.3)
3, 6, 9, 12
2, 4, 6, 8
Number of criteria
Number of categories
Fig. 3. Proportion of congruent assignments in the test set versus the number of
assignment examples used for learning; variable number of criteria (3, 6, 9, 12),
4 categories. Represented: median; box: 25–75%; whiskers: 10–90%. Boxes are
slightly shifted from left to right when the number of criteria increases.
Fig. 2. Workflow of the experimentation.
J. Zheng et al. / Computers & Operations Research 49 (2014) 28–40
number of assignment examples (requiring an increased cognitive
effort from the DM) will lead to inferred models closer to the
original model. This investigation studies the tradeoff between
cognitive effort and accuracy of the inferred models. We are
interested in how many assignment examples are necessary to
obtain an ELECTRE TRI model which is “close” enough to the original
one. To compare the two models, we generate randomly 10 000
alternatives (called test alternatives) and use both the original and
the elicited models to assign them to categories. The assignment
results are then compared and the proportion of congruent
assignments is calculated. The computation times are also
recorded to assess the tractability of the elicitation algorithm.
Experiments are run 500 times for each data size.
Results: Fig. 3 presents the average proportion of congruent
assignments in the test set as a function of the number of
assignment examples when the original models involve 4 categories and a variable number of criteria. Fig. 5 displays the
corresponding average computation time. In these figures, like in
most others in the rest of the paper, the values taken by the
variable of interest in the 500 repetitions of the experiment are
summarized by a box and a whisker plot. The lower and upper
sides of each box respectively represent the first and third
quartiles of the cumulative distribution of these values. The small
square inside each box indicates the median. The downward and
upward whiskers respectively extend to the percentiles 10 and 90
of this distribution.
In Fig. 3 we observe that the proportion of congruent assignments increases when more assignment examples are provided,
which means that the inferred model is getting closer to the
original model when more preference information is available. For
instance, in the case of ELECTRE TRI models with 4 categories and
6 criteria, providing 20 assignment examples ensures, in about
75% of the cases, that the algorithm infers a model which is able to
congruently re-assign 90% of the alternatives. The accuracy
reaches 95% in about 50% of the cases. It almost never (less than
10% chance) falls below 80%. For obtaining a similar level of
performance when there are 12 criteria, we need more, around
60, examples. This observation is not surprising because the more
the assignment examples, the smaller the feasible polytope
delimited by the constraints and thus the more determined the
inferred model. Fig. 3 thus indicates that, with an increased
Fig. 4. Average percentage of congruent assignments in the test set versus the
number of assignment examples used for learning; variable number of criteria (3, 6,
9, 12), 4 categories. Whiskers indicate the limits of the 95% confidence interval for
the mean percentage.
35
number of criteria, more assignment examples are required to
obtain models with the same performance.
If we hypothesize a constant “cost” for obtaining each assignment example and a tradeoff between such a cost and the
improvement in classification accuracy, the curves and box plots
in Fig. 3 can be used to determine an estimate of the “optimal”
sample size, provided an acceptable level of risk has been fixed
(e.g. 10% or 25% probability of not reaching the expected classification accuracy). Examples of exploitation of the information provided by these curves were given in the previous paragraph.
Alternatively, given the size of the available learning set, we can
predict the accuracy that can be expected. For instance, in the case
of three criteria, if we use a learning set of size 15, we may expect
that there is a 50% chance that the inferred model accuracy will lie
between 85% and 95%.
Another way of interpreting the information obtained through
our experiment is in terms of average accuracy. The 500 values of
the accuracy obtained by randomly sampling models and learning
sets allow us to estimate the expected value of the accuracy. Fig. 4
shows the mean of the 500 accuracy values computed for each
value of the experimental parameters. A 95% confidence interval
around each mean value is also represented. This diagram can be
used when the analyst formulates his/her requirements in terms of
minimal average accuracy. For instance, if (s)he wants at least 90%
classification accuracy on average (accepting a risk of 2.5% that the
mean is actually worse than that), the required number of assignment examples is the smallest sample size for which the lower
bound of the confidence intervals exceeds 0.9. Both Figs. 3 and 4
provide rich guidelines for the analyst, be it for helping to choose
the size of the learning set or to make up his/her mind regarding
the level of accuracy that can reasonably be expected.
Fig. 4 also shows a decreasing improvement rate of the mean
classification accuracy as the learning set grows. The confidence
intervals represented in the figure show that this improvement in
the mean value of the proportion of correct assignments is in
general statistically significant whenever ten additional items are
added to the learning set. It should be emphasized that the
intervals of variation depicted in Figs. 3 and 4 definitely differ in
nature: the boxplots in Fig. 3 represent the distributions of all
proportions of correct assignments obtained in the 500 repetitions
of our experiment for each datasize. In contrast, Fig. 4 depicts the
95% confidence interval obtained for the mean proportion of
correct assignments. As expected the precision of the confidence
interval for the mean proportion is quite good, since it decreases
proportionally to the square root of the number of repetitions.
The maximum average computation time of all the experiments with different settings is only 206 ms (Fig. 5), which is
acceptable even for real time implementation in a decision
support system. Furthermore, the computation time increases
(but remains acceptable) with the number of assignment examples and the number of criteria. This increase remains reasonable
given that the number of criteria impacts the number of variables
in the MIP, and each assignment example is transformed to
constraints and introduces two binary variables as well.
Fig. 6 summarizes the results of experiments in which an
inferred ELECTRE TRI model is used to assign 10 000 randomly
generated test alternatives. In these experiments, the number of
categories is varied while the number of criteria remains fixed to 7,
which is approximately the middle of the criteria range (3–12) we
considered. We observe that, when the number of assignment
examples is limited (less than 15), the proportions of congruent
assignments corresponding to experiments with different numbers of categories are clearly different. Due to the scarcity of input
information, the inference of an ELECTRE TRI model involves a certain
degree of arbitrariness. Thus a larger number of categories
means more possibility to make mistakes. When the number of
36
J. Zheng et al. / Computers & Operations Research 49 (2014) 28–40
Fig. 5. Computation time versus number of assignment examples; variable number
of criteria (3,6,9,12), 4 categories. Represented: median; box: 25–75%; whiskers:
10–90%. Boxes are slightly shifted from left to right when the number of criteria
increases.
Fig. 6. Proportion of congruent assignments versus the number of assignment
examples; 7 criteria, variable number of categories (2,4,6,8). Represented: median;
box: 25–75%; whiskers: 10–90%. Boxes are slightly shifted from left to right when
the number of categories increases.
assignment examples is greater than 15, the proportions of right
assignments for experiments with different number of categories
become almost identical. The curves representing the proportions
of congruent assignments for different numbers of categories
converge to 100% in a similar way. We explain this phenomenon
as follows. More categories make the constraints stemming from
assignment examples tighter, resulting in a smaller feasible polytope. At the same time, assigning an alternative to the right
category is more information demanding when there are more
categories. The two effects cancel out so that the number of
categories does not influence the number of assignment examples
required to infer a model “close” to the DM's preference.
4.2.2. Robustness analysis
Experiments: An important experimental issue refers to the
robustness of the inferred sorting model. Our experiment studies
Fig. 7. Number of possible assignments versus the number of assignment examples; 4 categories, variable number of criteria (3,6,9,12). Represented: median; box:
25–75%; whiskers: 10–90%. Boxes are slightly shifted from left to right when the
number of criteria increases.
the relation between the amount of input preference information
(number of assignment examples) and the robustness of the
induced model. More precisely, we consider the number jRa j of
categories to which an alternative a A A can be assigned to,
considering the preference information (see Section 3.2). We
compute the average value of jRa j, over the set A of test
alternatives, as an indicator of the robustness of the sorting model.
We vary the number of assignment examples and the complexity
of the original model (the number of categories and criteria
considered). For each parameter setting, the experiment is
repeated 50 times.
Results: The experimental results are shown in Fig. 7, where we
consider 4 categories and a variable number of criteria. A significant decrease in the average number jRa j of assignable categories can be observed while the number of assignment examples
increases. This observation is consistent with the results of the
previous experiment in which the learning ability of the inference
algorithm was tested. When the number of assignment examples
is relatively limited, the average number of assignable categories
jRa j is relatively large. This is because many ELECTRE TRI models are
compatible with the preference information, so that the alternatives are possibly assigned to several categories when using
different instances of the models compatible with the preference
information. When the number of assignment examples grows to
become relatively large, jRa j tends to decrease and becomes close
to 1, which means that almost every alternative can only be
assigned to a single category. In this case, only a few models
conform to the preference information and the inferred model is
close to the original one.
The number of criteria also has a strong effect on jRa j. For a
given number of assignment examples, jRa j increases with the
number of criteria. This is due to the fact that more criteria means
more variables in the MIP, giving more flexibility for the inferred
models, and hence leaving more possibility for an alternative to be
assigned to a category.
Fig. 8 shows the impact on robustness of varying the number of
categories (the number of criteria being kept fixed). We find that,
with a limited number of assignment examples (less than 15), jRa j
varies with the number of categories. When the number of
assignment examples is not too small (larger than 15), the number
of categories has no significant impact on jRa j. The curves
corresponding to different category numbers converge to 1 in a
J. Zheng et al. / Computers & Operations Research 49 (2014) 28–40
Fig. 8. Number of possible assignments versus the number of assignment examples; 7 criteria, variable number of categories (2,4,6,8). Represented: median; box:
25–75%; whiskers: 10–90%. Boxes are slightly shifted from left to right when the
number of categories increases.
similar way, which means that the number of categories does not
influence the robustness. The trend shown in Fig. 8 can be
explained similarly to that in Fig. 6, and they both illustrate that
the number of categories does not have any impact on the
robustness level when sufficient input preference information is
provided.
4.2.3. Inconsistency identification
Experiments: The experiments are organized as follows. Firstly,
the DM's inconsistent preference is simulated by introducing a
certain proportion of assignment errors in the set of assignment
examples. Just like in the two previous experiments, an original
ELECTRE TRI optimistic model is randomly generated. Instead of
assigning all alternatives in the learning set to the category
suggested by the original optimistic ELECTRE TRI rule, we randomly
assign a fixed number of them to one of the neighboring categories
(for example, assigning an alternative to C1 or to C3 instead of C2).
Secondly, the inconsistency resolution algorithm (Section 3.3,
program (47)–(55)) identifies a maximum subset of assignment
examples that can be represented in an ELECTRE TRI model. Lastly,
we compare the assignments made by the original ELECTRE TRI
models and the one inferred by the program (47)–(55) on a
random set of 10 000 test alternatives. Note that the assignment
examples which cannot be represented in the inferred model do
not necessarily correspond to the errors introduced in the assignments made using the original model. We thus study the following
two issues:
1. What is the proportion of assignment examples which can be
represented by the inferred model and how is this proportion
influenced by the number of assignment examples, the error
rate, the numbers of criteria and categories?
2. What is the proportion of assignment examples which are
assigned to the same category by both the original and the
inferred model and how is this proportion influenced by the
number of assignment examples, the error rate, the numbers of
criteria and categories?
The first issue is related to the capacity of the family of all ELECTRE
TRI optimistic rules to represent assignments that depart from a
37
given original rule in the family. We refer to it as the representation
issue. The second issue questions the capability of the learning
algorithm to filter out errors and restate as much as possible the
original model. We refer to it as the error correcting issue. Or to put
it another way, the latter issue amounts to identify the conditions
(size of the learning set, error rate, numbers of criteria and
categories) in which the original model is almost determined as
the model yielding the best approximation of the assignments in
the learning set.
To study these questions, for each parameter setting, we run
the experiments 500 times. Three rates of assignment errors in the
assignment examples are considered: 10%, 20%, and 30%. The
number of assignment examples varies from 20 to 100 by steps
of 10 examples. The impact of the number of criteria and
categories is also studied. Computation times are recorded.
Results relative to the representation issue: Fig. 9 displays the
mean value (over 500 runs) of the maximum proportion of
assignment examples that can be represented by an optimistic
ELECTRE TRI model under different levels of assignment errors' rates
(10%, 20%, 30%) when the original ELECTRE TRI models involve
7 criteria and 4 categories.
Fig. 9 suggests that the maximum proportion of represented
assignment examples asymptotically converges to (100 x)%
where x stands for the rate of errors introduced in the assignment
examples. When the size of the learning set is relatively small, the
flexibility of the ELECTRE TRI model makes it possible to restore more
than (100 x)% of assignment examples. When the size of the
learning set is large, the proportion of assignment examples which
cannot be represented by an optimistic ELECTRE TRI model almost
equals the proportion of assignment errors introduced. This
observation is consistent with the finding that more assignment
examples produce more determined models, implying that it
becomes much harder for an ELECTRE TRI model to “accommodate”
assignment errors.
Table 2 shows that the computation time increases with the
proportion of errors and the number of assignment examples.
Computation times remain acceptable even for the extreme cases of
experiments which elicit ELECTRE TRI models with 7 criteria, 4 categories,
using 100 assignment examples with 30% errors introduced.
Fig. 9. Proportion of represented assignment examples for ELECTRE TRI models
involving 7 criteria and 4 categories; 10%, 20%, 30% error rate. Represented:
median; box: 25–75%; whiskers: 10–90%. Boxes are slightly shifted from left to
right when the error rate increases.
38
J. Zheng et al. / Computers & Operations Research 49 (2014) 28–40
Table 2
Computation times (s) to identify inconsistencies in learning sets, 10%, 20%, 30%
error rate; 7 criteria, 4 categories.
% error
10
20
30
♯ ass. ex.
20
30
40
50
60
70
80
90
100
0.04
0.04
0.08
0.07
0.10
0.22
0.15
0.18
0.34
0.22
0.43
0.90
0.37
0.65
2.77
0.44
1.04
2.93
0.64
1.31
7.87
0.69
1.64
7.61
0.86
2.04
25.51
Fig. 11. Proportion of represented erroneous assignment examples by ELECTRE TRI;
7 criteria, 4 categories, 10%, 20%, 30% error rate. Represented: median; box:
25–75%; whiskers: 10–90%. Boxes are slightly shifted from left to right when the
error rate increases.
Fig. 10. Proportion of represented correct/erroneous assignment examples by
ELECTRE TRI; 7 criteria, 4 categories, 10% error rate.
Results relative to the “error correcting” issue: We have just
observed that the inferred model tends to assign ð100 xÞ% of the
examples to their prescribed category, where x is the error rate in
the learning set. This does not imply that the approximate x% of
examples incorrectly assigned by the inferred model indeed
correspond to the errors introduced in the learning set. One may
wonder whether the inferred model is close to the original one,
assigning correctly the same examples and incorrectly the others.
Fig. 10 depicts the proportion of represented assignment
examples (for 7 criteria, 4 categories, 10% errors and a varying
size of the learning set), and distinguishes among the represented
assignment examples, those corresponding to the correct ones
(gray part of the histogram) from those corresponding to errors
(white part of the histogram). For instance, with a learning set of
30 assignment examples, on average approximately 92% of the
assignment examples are represented, among which 89% correspond to correct assignment examples, and 3% correspond to
errors. We observe that, when the learning set size increases, the
proportion of correct represented assignment examples increases,
while the proportion of incorrect assignment examples decreases.
Another interesting question is the following: among the
erroneous assignment examples, how does the proportion of
represented erroneous assignment examples evolve with the size
of the learning set? Fig. 11 shows such proportions for a varying
size of the error rate, and varying size of the learning set. We
observe that this proportion quickly decreases with the size of the
learning set, thus showing a good ability of the algorithm to detect
errors. For instance, with 20% errors and a learning set of size 80
Fig. 12. Proportion of congruent assignments versus the number of assignment
examples in the learning set; original models involving 4 categories, 7 criteria;
variable error rates in learning set (10%, 20%, 30%). Represented: median; box:
25–75%; whiskers: 10–90%. Boxes are slightly shifted from left to right when the
error rate increases.
(consequently with 16 erroneous assignment examples), this
proportion is on average approximately 5%, which means that,
on average, less than one erroneous assignment example is
represented in the inferred model.
Recall that an example assigned to the same category by the
original and the inferred model is called a congruent assignment.
To study whether the inferred model is close to the original one,
we use a test set of 10 000 randomly generated alternatives and
we count the number of congruent assignments in this test set.
Fig. 12 displays the average proportion of congruent assignments for a varying size of the learning set with 10%, 20% and 30%
error rates in the learning set. The original models involve
J. Zheng et al. / Computers & Operations Research 49 (2014) 28–40
4 categories and 7 criteria. We observe that the proportion of
congruent assignments increases with the size of the learning set.
The inferred model is getting closer to the original one while more
preference information is added. This holds true for all error rates
although the convergence is slower for larger error rates. There is
no indication that such a tendency will still hold for very noisy
data sets (e.g. with an error rate larger than 50%); however, the
sort of decision aiding applications targeted by our algorithm
typically involve decision makers who provide “reasonably consistent” preference information (in any case, with error rates less
than 30%).
5. Conclusion and further research
This paper presents algorithms to infer the criteria weights and
majority threshold of an optimistic ELECTRE TRI rule and compute
robust assignments using this rule, in case the criteria importance
vector is not precisely known, but contained in a polytope of
acceptable values. This polytope is expressed by the DM through
assignment examples. In contrast with the pessimistic rule,
inferring an optimistic ELECTRE TRI rule from assignment examples
induces disjunctive constraints. We linearize these conditions
introducing binary variables, which permits to infer weights and
compute robust assignments through MIP. Hence, our first contribution is the development of preference elicitation tools for the
ELECTRE TRI optimistic rule, which learn parameter values and
compute the corresponding robust assignment.
Numerical experiments are conducted to investigate the performance of the algorithms with respect to learning ability,
robustness and ability to identify conflicting preferences. The
numerical tests prove the algorithms to be effective for realistic
data sets' sizes. Furthermore, the experiments provide insights
into the amount of input preference information needed to infer
an optimistic ELECTRE TRI model in a reliable way. This study gives
useful guidelines for a decision analyst involved with a DM in a
real-world application when the analyst formulates the decision
problem using the optimistic ELECTRE TRI rule.
The paper assumes that the profiles of the categories are
known a priori, and no veto threshold is taken into consideration.
Further research should be pursued in view of relaxing these
assumptions. Preliminary results have been obtained by Sobrie
et al. [31] in view of eliciting the weights and the limit profiles in
the case of the pessimistic rule. The algorithm used for the
pessimistic case could be adapted to the optimistic one although
this is not straightforward due to the added complexity of the
optimistic assignment rule. However, as soon as an algorithm for
estimating limit profiles will be available, it will be possible to
assess the expressivity of the model and its ability to reproduce
the assignments produced by various types of sorting rules as well
as the sorting of real data into categories. This is an especially
interesting question since, to the best of our knowledge, there has
been no attempt at learning ELECTRE TRI – like rules or majority rules
in machine learning to sort alternatives in ordered categories. This
is due to the fact that learning such rules on the basis of large data
sets is complicated. Yet the idea of comparing alternatives to limit
profiles using a majority rule is an appealing one for sorting in
ordered categories. It has the advantage of potentially involving all
criteria in a single decision rule while more traditional decision
rules used in classification tend to involve a small number of
criteria. This is a good opportunity to see whether global rules, i.e.
rules that simultaneously take into account all alternatives
descriptors, have a competitive descriptive power at least for
particular types of data.
39
Acknowledgments
The authors are indebted to two anonymous referees for their
remarks that greatly helped to improve previous versions of this
manuscript.
References
[1] Belacel N. Multicriteria assignment method PROAFTN: methodology and
medical application. Eur J Oper Res 2000;125(1):175–83.
[2] Bouyssou D, Marchant T. An axiomatic approach to noncompensatory sorting
methods in MCDM. I. The case of two categories. Eur J Oper Res 2007;178
(1):217–45.
[3] Bouyssou D, Marchant T. An axiomatic approach to noncompensatory sorting
methods in MCDM. II. More than two categories. Eur J Oper Res 2007;178
(1):246–76.
[4] Bouyssou D, Marchant T, Pirlot M, Tsoukiàs A, Vincke P. Evaluation and
decision models with multiple criteria: stepping stones for the analyst. New
York: Springer; 2006.
[5] Butler J, Jianmin J, Dyer J. Simulation techniques for the sensitivity analysis of
multi-criteria decision models. Eur J Oper Res 1997;103(3):531–46.
[6] Devroye L. Non-uniform random variate generation. New York: Springer;
1986.
[7] Dias L, Climaco J. On computing ELECTRE's credibility indices under partial
information. J Multi-Criteria Decis Anal 1999;8(2):74–92.
[8] Dias L, Mousseau V, Figueira J, Clímaco J. An aggregation/disaggregation
approach to obtain robust conclusions with ELECTRE TRI. Eur J Oper Res
2002;138(2):332–48.
[9] Dias LC, Sarabando P. A note on a group preference axiomatization with
cardinal utility. Decis Anal 2012;9(3):231–7.
[10] Doumpos M, Marinakis Y, Marinaki M, Zopounidis C. An evolutionary
approach to construction of outranking models for multicriteria classification:
the case of the ELECTRE TRI method. Eur J Oper Res 2009;199:496–505.
[11] Doumpos M, Zopounidis C. Developing sorting models using preference
disaggregation analysis: an experimental investigation. Eur J Oper Res
2004;154(3):585–98.
[12] Greco S, Matarazzo B, Słowiński R. Rough sets methodology for sorting
problems in presence of multiple attributes and criteria. Eur J Oper Res
2002;138(2):247–59.
[13] Jacquet-Lagrèze E. An application of the UTA discriminant model for the
evaluation of R&D projects. In: Pardalos P, Siskos Y, Zopounidis C, editors.
Advances in multicriteria analysis nonconvex optimization and its applications. Dordrecht: Kluwer Academic; 1995. p. 203–11.
[14] Jacquet-Lagrèze E, Siskos Y. Assessing a set of additive utility functions for
multicriteria decision making: the UTA method. Eur J Oper Res 1982;10:151–64.
[15] Jacquet-Lagrèze E, Siskos Y. Preference disaggregation: 20 years of MCDA
experience. Eur J Oper Res 2001;130(2):233–45.
[16] Leroy A, Mousseau V, Pirlot M. Learning the parameters of a multiple criteria
sorting method. In: Brafman R, Roberts F, Tsoukiàs A, editors. Algorithmic
decision theory. Lecture notes in computer science, vol. 6992. Berlin, Heidelberg: Springer; 2011. p. 219–33.
[17] Massaglia R, Ostanello A. N-tomic: a support system for multicriteria segmentation problems. In: Korhonen P, Lewandowski A, Wallenius J, editors. Multiple
criteria decision support. Lecture notes in economics and mathematical
systems, vol. 356. Berlin: Springer Verlag; 1991. p. 167–74.
[18] Mavrotas G, Diakoulaki D, Capros P. Combined MCDA IP approach for project
selection in the electricity market. Ann Oper Res 2003;120:159–70, http://dx.
doi.org/10.1023/A:1023382514182.
[19] Metchebon Takougang SA. Contribution à l’aide à la décision en matière de
gestion spatialisée. Etude de cas en management environnemental et développement de nouveaux outils (PhD thesis). Belgium: Faculté polytechnique,
Université de Mons; 2010.
[20] Metchebon Takougang SA, Pirlot M, Some B, Yonkeu S. Assessing the response
to land degradation risk: the case of the Loulouka catchment basin in Burkina
Faso. In: Bisdorff R, Dias L, Meyer P, Mousseau V, Pirlot M, editors. Evaluation
and decision models with multiple criteria: case studies. Berlin: Springer;
2014.
[21] Moscarola J, Roy B. Procédure automatique d’examen de dossiers fondée sur
une segmentation trichotomique en présence de critères multiples. RAIRO
Rech Opér 1977;11(2):145–73.
[22] Mousseau V, Dias L, Figueira J, Gomes C, Clímaco J. Resolving inconsistencies
among constraints on the parameters of an MCDA model. Eur J Oper Res
2003;147(1):72–93.
[23] Mousseau V, Figueira J, Naux JP. Using assignment examples to infer weights
for ELECTRE TRI method: some experimental results. Eur J Oper Res 2001;130
(2):263–75.
[24] Mousseau V, Roy B, Sommerlatt I. Development of a decision aiding tool for
the evolution of public transport ticket pricing in the Paris region. In: Colorni
A, MP, Roy B, editors. A-MCD-A aide multicritère à la décision—multiple
criteria decision aiding. Luxembourg: Joint Research Center, European Commission; 2001. p. 213–30.
[25] Mousseau V, Słowiński R. Inferring an ELECTRE TRI model from assignment
examples. J Glob Optim 1998;12(2):157–74.
40
J. Zheng et al. / Computers & Operations Research 49 (2014) 28–40
[26] Mousseau V, Słowiński R, Zielniewicz P. A user-oriented implementation of
the ELECTRE TRI method integrating preference elicitation support. Comput
Oper Res 2000;27(7–8):757–77.
[27] Perny P. Multicriteria filtering methods based on concordance/non-discordance principles. Ann Oper Res 1998;80:137–67.
[28] Roy B. Méthodologie multicritère d’aide à la décision. Paris: Economica; 1985.
[29] Roy B, Bouyssou D. Aide multicritère à la décision: méthodes et cas. Paris:
Economica; 1993.
[30] Siskos Y, Grigoroudis E, Krassadaki E, Matsatsinis N. A multicriteria accreditation system for information technology skills and qualifications. Eur J Oper
Res 2007;182(2):867–85.
[31] Sobrie O, Mousseau V, Pirlot M. Learning a majority rule model from large sets
of assignment examples. In: Perny P, Pirlot M, Tsoukiàs A, editors. ADT. Lecture
[32]
[33]
[34]
[35]
notes in computer science, vol. 8176. Berlin-Heidelberg: Springer; 2013. p.
336–50.
Williams J. Model building in mathematical programming. Chichester: John
Wiley and Sons; 1999.
Yu W. Aide multicritère à la décision dans le cadre de la problématique du tri:
méthodes et applications (PhD thesis). Paris: LAMSADE, Université Paris
Dauphine; 1992.
Zopounidis C, Doumpos M. A multicriteria decision aid methodology for
sorting decision problems: the case of financial distress. Comput Econ
1999;14:197–218, http://dx.doi.org/10.1023/A:1008713823812.
Zopounidis C, Doumpos M. Multicriteria classification and sorting methods: a
literature review. Eur J Oper Res 2002;138(2):229–46.