BULETINUL
Universităţii Petrol – Gaze din Ploieşti
Vol. LIX
No. 1/2007
19 - 28
Seria
Matematică - Informatică - Fizică
Performance Measures
for Multi-objective Optimization Algorithms
Elena Simona Nicoară
Universitatea Petrol-Gaze din Ploieşti, Bd. Bucureşti 39, Ploieşti, Catedra de Informatică
e-mail: snicoara @upg-ploiesti.ro
Abstract
In the real world, the most optimization problems are multi-objective. For the complex ones we do not
often have polynomial algorithms which return the exact solution(s) in practical time. For this reason, we
use approximation algorithms which find solution(s) (near) optimal in a practical time. Alongside the
aspects related to comparing two multi-objective solutions, there are also aspects related to measuring
the performance of an algorithm. In the paper we present the most important performance measures for
multi-objective optimization algorithms. As a practical example, we have compared three genetic
algorithms on the bi-objective JSSP test-problem ft10, and the results showed the necessity to
simultaneously consider many performance measures for the multi-objective algorithms.
Key words: multi-objective optimization, Pareto dominance, Pareto optimal solution, performance
measure, approximation of the Pareto optimal set
Multi-objective Optimization Problems
A uni-objective optimization problem consists in searching a solution x ∈ X, so that the
objective function, f(x), have a maximum or minimum value. In this case, to compare two
candidate-solutions, x(1) and x(2), means to compare their objective values, f(x(1)) and f(x(2)).
Yet many real problems involve both the simultaneous optimization of many objectives, often in
conflict one with another, and a complex large search space. Ordinarily, this space is very
difficult to explore through enumerative methods, because it contains multiple possible
solutions, placed in hardly accessible regions or having properties which slow down finding the
solutions. In these cases, we search many solutions of compromise.
Though there are procedures to aggregate the objectives in a single one and to treat the problem
as a uni-objective problem, in the case of complex problems the disadvantages of this approach
have to be considered.
A multi-objective optimization problem is defined by a set of n parameters (decision variables),
a set of k objective functions and a set of m constraints. The objective functions and the
constraints are functions of the decision variables. The aim of optimization is to
minimize y = f(x) = (f1(x), f2(x),…, fk(x)),
satisfying the constraints e(x) = (e1(x), e2(x),…, em(x)) ≤ 0,
(1)
20
Elena Simona Nicoară
where x = (x1, x2,…, xn) ∈ X, y = (y1, y2,…, yk) ∈ Y.
Here x is named the decision vector, y is named the objective vector, X the decision space and
Yf = f(Xf) the objective space. The feasible set Xf is the set of decision vectors x which satisfy the
constraints e(x) ≤ 0.
In the uni-objective optimization problems, the feasible set is totally ordered, with respect to the
objective function f, while in the multi-objective optimization problems, Xf is, generally,
partially ordered, with respect to the objective function f. [8] To compare two candidatesolutions in the multi-objective optimization problems there have been defined the concepts of
Pareto dominance and Pareto optimal solutions. Based on the Pareto dominance we can also
introduce the optimality criterion for the problem to be solved.
Without restricting the generality, we can consider that all the partial objective functions are to
be minimized. The following definitions refer to such optimization problems.
Definition 1. x(1) dominates x(2) if the both following conditions are satisfied [2]:
x(1) is not weaker than x(2) in every objective, namely x (1) i p x ( 2 ) i , ∀ i ∈ {1,2,..., k} ;
x(1) is strictly better than x(2) at least in one objective, namely ∃i ∈ {1,2,..., k } so that
x (1) i p x ( 2 ) i .
The symbol p is generally accepted to designate the dominance relation.
If x(1) dominates x(2) we say that x(2) is dominated by x(1).
Consequently, for any two candidate-solutions, one of these situations can be true:
º
º
one solution dominates the other (is better considering all the objectives),
the solutions can not be compared (does not exist a dominance relation between them).
Definition 2. A decision vector x ∈ X is nondominated relative to a set A ⊆ X iff [8]:
¬∃a ∈ A so that a dominate x .
(2)
Definition 3. The decision vector x is Pareto optimal if and only if it is nondominated relative to
the entire set X.
In other words, a decision vector (a candidate solution) is Pareto optimal if it can not be
improved in any objective without causing a degradation in at least one another objective
(Pareto, 1896) [8].
Generally speaking, it does not exist a single Pareto optimal solution of a problem, but a set of
Pareto optimal solutions. None of them can be identified as better than the others, in the absence
of preference information for a certain objective.
The Pareto optimal solutions form the Pareto optimal set, and the correspondent objective
vectors form the Pareto optimal front.
Definition 4. Let A ⊆ X be a set of decision vectors. A is a local Pareto optimal set iff [8]:
∀a ∈ A, ¬∃x ∈ X so that x dominate a , x −a < ε and f ( x) − f (a) < δ ,
where
.
(3)
is a metrics, and ε > 0, δ > 0 very small values.
In other words, A contains those decision vectors which are not dominated by the decision
vectors in a small neighborhood, in the objective space.
Definition 5. The set A is a global Pareto optimal set iff [8]:
Performance Measures for Multi-objective Optimization Algorithms
21
∀a ∈ A, ¬∃x ∈ X so that x dominate a.
(4)
So, A is formed by the nondominated decision vectors. All the Pareto optimal solutions are not
mandatory for a global Pareto optimal set, and every global Pareto optimal set is also a local
Pareto optimal set. The Pareto optimal set includes the global optimum solutions.
Fig. 1. Illustration of a global Pareto optimal front and a local Pareto optimal front
in a bi-objective space, for a minimization problem
In the multi-objective optimization, the aim is to find as many different solutions as possible
(near) Pareto optimal. A multi-objective optimization algorithm has to perform two tasks [2]:
º
º
to guide the search towards the global Pareto optimal region and
to maintain the population diversity (in the objective space, in the parameters space or in
both of them) in the current nondominated front.
Performance Measures for Multi-objective Optimization Algorithms
In the multi-objective optimization, it is often impossible to know the Pareto optimal set. In
these conditions, to measure the performance of a search algorithm is difficult enough.
The general performance criteria for the multi-objective optimization algorithms are:
º
º
º
accuracy - how close the generated nondominated solutions are to the best known
prediction;
coverage - how many different nondominated solutions are generated and how well they
are distributed;
the variance for every objective - which is the maximum range of nondominated front,
covered by the generated solutions (fraction of the maximum range of the objective in the
nondominated region, covered by a nondominated set).
Starting from these criteria, there were developed various performance measures for the search
algorithms, originating from various areas (statistics, computer science, biology etc.). In the
following sections of the paper we present the most important performance measures, which
also allow us to compare algorithms and to adjust their parameters for better results.
A Diversity Performance Measure
A comparing measure, from the solutions diversity point of view, is described by Deb et al. [3].
It is based on the successive distances of the solutions in the first front of the final population.
The set of solutions in this front is compared with a uniform distribution, and then the deviation
from it is determined by the formula [3]:
22
Elena Simona Nicoară
F0
di − d
i =1
F0
∆=∑
.
(5)
Here, di is the Euclidean distance between two successive solutions in the first front (F0), in the
objective space, and⎯d is the average of these distances. In order for the distribution to take into
account the entire space of the real front, we have included the frontier solutions also in the
front F0. We note that the deviation measure is computed for every run where the distinct
solutions are more than three. The average of these deviations for ten runs, for example, can be
a comparing measure for different algorithms. Hence, in terms of the ability to obtain more
„dispersed” solutions, more uniformly distributed solutions on the first front, an algorithm with
a smaller average deviation is a better option.
The Expected Number of Evaluation for Success
The performance measure ENES (Expected Number of Evaluation for Success) is determined
by the following formula, which indicates the average number of evaluations to obtain the
success.
ENES = NE / NS.
(6)
NE denotes the number of evaluations for the objective function in 20 independent runs and NS
the number of successes in the 20 runs.
Quantitative Performance Measures If the Pareto Optimal Set Is Known
Talbi [7] submitted separate performance measures, for cases when the Pareto optimal set is
known, respectively unknown. The Pareto optimal set is known when it can be identified using
an enumerative method (for example Branch & Bound).
If the Pareto optimal set (OP) is known, Talbi defines the absolute efficiency of the algorithm as
the proportion occupied by the Pareto solutions in the approximated Pareto optimal set (OP*),
relative to the real Pareto optimal set [7]:
E=
OP * ∩ OP
OP
.
(7)
A solution belonging to OP*, without being Pareto optimal, is not necessarily a weak solution.
The smaller the distance between OP and OP*, the better the algorithm.
Let d(x,y) be a distance between two solutions in the objective space (the Tchebycheff norm):
n
d ( x , y ) = ∑ λi f i ( x ) − f i ( y ) ,
(8)
i =1
where λi is a parameter which allows to normalize the various criteria.
The distance between a solution in OP* and a solution y can be defined by the following
formula [7]:
d (OP* , y ) = min d ( x, y ), x ∈ OP* .
(9)
One must notice that we also can use other distances between two solutions in the objective
space besides the Tchebycheff norm.
Performance Measures for Multi-objective Optimization Algorithms
23
Based on these definitions, we can consider many quantitative measures for algorithms [7]:
º
the longest distance between the sets OP* and OP [7]:
WD = max d (OP * , y ), y ∈ OP .
º
(10)
the average distance between the sets OP* and OP [7]:
∑ d (OP , y)
*
MD =
º
y∈OP
OP
.
(11)
a measure for the uniformity of the set OP* [7]:
DIV =
WD
MD .
(12)
Qualitative Measures Based on Approximations of the Pareto Optimal Set
If the Pareto optimal set is not known, then we work with an approximation of it.
M.P. Hansen and A. Jaskiewicz have introduced a family of comparing relations between
approximations in 1998 [7].
Definition 6. An approximation A weakly dominates an approximation B if [7]:
A ≠ B and ND ( A ∪ B ) = A ,
(13)
where ND(S) is the set of solutions nondominated by the set S. This means that A weakly
dominates B if for every solution xB∈B there is a solution xA∈A which dominates xB or is equal
to xB and at least a solution in A is not in B (see figure 1.(a)).
Definition 7. An approximation A strongly dominates an approximation B if [7]:
ND ( A ∪ B ) = A and B − ND ( A ∪ B ) ≠ ∅.
(14)
In other words, for every solution xB∈B there is a solution xA∈A which dominates xB or is equal
to xB and at least a solution xB∈B is dominated by a solution xA∈A (see figure 1.(b)).
Definition 8. An approximation A totaly dominates an approximation B if [7]:
ND ( A ∪ B ) = A and B ∩ ND ( A ∪ B ) = ∅.
(15)
This means that every solution xB∈B is dominated by a solution xA∈A (see figure 1.(c)).
(a)
(b)
(c)
(d)
Fig. 2. The dominance relations between two approximations, for two/objective minimization:
(a) weak dominance, (b) strong dominance, (c) total dominance, (d) incomparable approximations.
24
Elena Simona Nicoară
Figure 1.(d) illustrates two incomparable approximations – neither dominates the other. Each
above relation defines a partial order in the approximations set. Moreover, these relations allow
only a qualitative comparison between approximations.
To judge an approximation as satisfactory, attributes such as coverage, uniformity, cardinality
and adjacency are important under two aspects: the rules to stop an approximation algorithm
and the procedures to tune its parameters.
The Contribution of a Metaheuristic Relative to Another Metaheuristic
In [7] is also submitted to research a comparing measure for two approximation methods, two
metaheuristics, M1 and M2, named the relative efficiency measure. It allows to compare the
methods by counting the Pareto solutions, identified by M1, dominated by those identified by
M2. Let OP*i be the Pareto optimal set identified by Mi and let C be the Pareto solution set,
common to both OP*i. Let W1, W2, be the solutions sets in OP*1, respectively OP*2 which
dominate the solutions in OP*2, respectively OP*1. Let L1 and L2 be the solutions sets in OP*1,
respectively OP*2 dominated by the solutions in OP*2 and respectively OP*1.
The set of solutions in OP*1, OP*2 which have dominance relation with no solution in OP*2,
respectively OP*1 will be then [7]:
N i = OPi* − (C ∪ Wi ∪ Li ), i ∈ {1,2}
.
(16)
Let OP* be the set of Pareto optimal solutions found by both methods [7]:
OP* = C ∪ W1 ∪ N1 ∪ W2 ∪ N 2 .
(17)
The contribution of the metaheuristic M1 relative to M2 is the ratio of the solutions in OP*
produced by M1 [7]:
C
C
+ W1 + N 1
2
Cont ( M 1 / M 2 ) =
= 2
C + W1 + N 1 + W2 + N 2
+ W1 + N 1
OP *
.
(18)
We note that if the two metaheuristics provide the same solutions, then
Cont(M1/M2) = Cont(M2/M1) = ½, and if all the solutions produced by M2 are dominated by the
solutions produced by M1, then Cont(M2/M1) = 0.
The Distance Between a Set of Solutions and a Reference Set
To evaluate a set of solutions Sj, Czyzack and Jaszkiewicz (1998) proposed a measure based on
the distance from a reference set (the Pareto optimal set or a near Pareto optimal set); more
precisely, the average distance from every reference solution to the nearest solution in Sj. This
measure, entitled D1R by Knowles and Corne (2002), is defined in terms of a reference set S*, as
follows [4]:
D1R ( S j ) =
1
S*
∑ min{d
y∈S *
xy
x ∈ S j}
.
(19)
Here, dxy is the distance between a solution x and a reference solution y in the N-dimensional
objective space [4]:
Performance Measures for Multi-objective Optimization Algorithms
d xy = ( f1* ( y ) − f1* ( x)) 2 + ... + ( f N* ( y ) − f N* ( x)) 2
,
25
(20)
where f i * (.) is the ith objective, normalized using the reference set S*.
The smaller D1R(Sj), the better the solutions set Sj. Note that D1R is not the average distance
from every solution in Sj to the nearest solution in S*; this corresponds to the general distance.
While the general distance can evaluate only the proximity of the set Sj to S*, D1R(Sj) can
evaluate both proximity to S* and the distribution of Sj.
If Sj proves to be a good approximation for the real Pareto optimal set, the solution x in Sj,
chosen by the decident, can approximate the best solution x*. In this case, the loss caused by the
choice of x instead of x* can be approximately measured by the distance between x and x* in the
objective space. If the solution x* is not known, we can not directly measure this distance. But
an expected value for this distance can be coarsely estimated by the average distance from every
Pareto optimal solution to the nearest solution available. The measure D1R corresponds exactly
to this approximation.
With the aim of evaluate a set of solutions Sj, let S = S1 ∪ S 2 ∪ ... ∪ S J be the union of J sets
of solutions. A direct performance measure for the set Sj relative to S is the proportion of the
solutions in Sj which are not dominated by any other solution in S [4]:
R NDS ( S j ) =
S j − {x ∈ S j ∃y ∈ S : y p x}
Sj
.
(21)
y p x denotes the dominance of y on x. The bigger is ratio RNDS(Sj), the better is the set Sj.
Another performance measure is the front diversity relative to the solutions space. Because the
Pareto optimal front is defined relative to the objective space, it is natural that the performance
measures based on the Pareto optimal front should also be defined relative to the objective
space.
Other Performance Measures
In [1] authors define the following performance measures for multi-objective algorithms:
º
º
º
The convergence order of the algorithm;
Local performance measures, which evaluate the algorithm power to improve the
population (of solutions) state from the generation T to the generation T+1;
Global performance measures to evaluate the algorithm’s behavior (behavior in infinity
conditions) in the long term. These return the needed computer resources and time
resources to attain the optimum.
Other comparing measures are:
º
º
º
º
º
The best objective value found in X runs;
The average duration, in 10 runs, to obtain the same solution(s);
The consistency associated to the localization of all the optima;
The progress rate from one generation to the next;
The progress velocity (the average distance in the search space, made in the beneficial
direction / the number of evaluations for the objective function).
26
Elena Simona Nicoară
Case study. Analysis of results
In order to validate the need for considering many performance measures for algorithms used to
solve multi-objective problems, we tested three genetic algorithms (GA) on the bi-objective Job
Shop Scheduling Problem (JSSP) ft10. It consists in scheduling 10 jobs, each formed of 10
operations, on 10 machines, being known the precedence relations and the processing times.
Minimization of the makespan and minimization of the number of late jobs against the deadline
1000 units are the objectives.
Although this problem is a small-sized one, it is among the most difficult JSSP test-problems. A
solution was identified only after 24 years from its enunciation, and after 2 more years it was
demonstrated that the optimal solution (in the uni-objective formulation) has the makespan 930
units. The tested genetic algorithms are: an elitist genetic algorithm, NSGA_II (Non-dominated
Sorting Genetic Algorithm II) designed by Deb [3] and NSGA_II_DAR (NSGA_II with
Dynamic Application of genetic operators and partial population Reinitialization). The last one
was proposed by the author in [6].
For the ft10 problem, the Pareto optimal set is not known. The performance measures for the
tested algorithms and their values are presented in table 1 below.
Table 1. The performance measures for three algorithms solving the bi-objective ft10
ALGORITHM
PERFORMANCE MEASURE
The best makespan solution in makespan
the final front
number of late operations
The best aggregated objective value
(coefficients 0.8, 0.2)
The average aggregated objective values in the final front
The worst aggregated objective values in the final front
Diversity in the schedules space
The number of different nondominated solutions, per test
Minimum schedule overlapping
Maximum schedule overlapping
Average schedule overlapping
Diversity in the objective space
The diversity performance measure described by Deb
Minimum
Average
The variance for the first objective (the most important)
Range dimension for the makespan
Run time, per generation* (seconds)
ELITI
ST GA
1054
5
NSGA
_II
1175
16
NSGA_II
DAR
1096
8
844.2
943.2
878.4
823.5
893.8
934.6
992.1
890.3
956.5
4.3
0.87
0.99
0.95
5.5
0.80
0.99
0.92
8.36
0.83
0.99
0.94
0.07
9.30
0.01
5.33
0.01
6.24
412
3.6
456
2.4
213
3.2
(*) The results were obtained on a processor AMD Athlon 1600 MHz and 256 MB RAM.
In the table, the bold values indicate the best value for the current performance measure. We
notice that, from the solution fitness point of view, the best algorithm is the elitist genetic
algorithm. If this would be the single perspective to consider, as in the uni-objective case, we
may conclude that other analysis is not necessary, and the best algorithm is the elitist GA. But
the distribution of the solutions in the space is very important for the multi-objective problems.
The level of distribution was measured by many indicators, grouped in three classes: diversity in
the schedules space, diversity in the objective space and the variance for the first objective (the
most important).
Performance Measures for Multi-objective Optimization Algorithms
27
For the permutation encoding (associated with scheduling solutions) we used to measure the
diversity of found solutions the number of different nondominated solutions, per test and the
indicator named schedule overlapping, defined in [5]. The last one takes into account both the
order of operations in the schedule sequence and the start processing and stop processing times
of the operations. It quantifies the similarity of two schedules so those identical schedules have
a measure equal to 1 and two completely different schedules a measure equal to 0.
Consequently, reduced values for this measure indicate a better distribution of the solutions in
the schedules space. From the diversity point of view, we note that NSGA_II is the best
algorithm closely followed by NSGA_II_DAR. If the time resources are strongly limited, the
user would consider also the run time needed. And from this perspective, the NSGA_II proves
to be, on the average, the best option to take. As we can see, there is no one single algorithm to
prove the best from all perspectives, namely: fitness and diversity of the solutions and time
complexity. These data prove that we can not efficiently evaluate the quality of an algorithm
form a single perspective.
In conclusion, to better assist the user to decide what algorithm to use for a certain problem at a
given time, under particular circumstances, we must consider many performance measures.
Conclusion
In the multi-objective optimization, the aim is to find as many different solutions as possible
(near) Pareto optimal. Because there are many types of complex problems (multimodal,
deceptive etc.) and no single algorithm is the best for every type and even for every instance,
many methods / algorithms have been developed in the literature to solve these problems.
The described performance indicators allow to measure the performance of an algorithm, to
adjust the parameters of an algorithm to obtain better results and also to compare different
algorithms.
The measures can be quantitative or qualitative. We can treat separately the performance, when
the Pareto optimal set is known or unknown. These measures take into account: the solution
diversity, the expected number of evaluations for attaining the success, the proportion occupied
by the Pareto solutions in the approximated Pareto optimal set, the proportion of the solutions
found by an algorithm in the set of solutions found by two algorithms, the distance between an
identified set of solutions and a reference set of solutions, the convergence order of the
algorithm, the power to improve the population from a generation to another, the run time for
attaining a certain set of solutions etc.
Ordinarily, for the multi-objective optimization, we compare many sets of solutions obtained by
different algorithms or obtained by the different parameters specifications of the same
algorithm.
We recommend the simultaneous use of many performance measures, because it is impossible
to evaluate all the aspects in every set of solutions using a single measure. The above presented
case study is a proof for this recommendation.
References
1.
2.
B e y e r , H . G . , S c h w e f e l , H . P . , W e g e n e r , I . - How to analyse evolutionary
algorithms, Theoretical Computer Science 287, pp. 101-130, 2002
D e b , K . - Multi-objective genetic algorithms: Problem difficulties and construction of test
problems, Evolutionary Computation Journal, 7(3), pp. 205-230, 1999
28
Elena Simona Nicoară
3. Deb,
K . , A g r a w a l , S . , P r a t a p , A . , M e y a r i v a n , T . - A fast elitist
nondominated sorting genetic algorithm for multi-objective optimization: NSGA-II,
IEEE Transactions on Evolutionary Computation, 6(2), pp.182–197, 2002
4.
5.
6.
7.
8.
H o l l a n d , J . H . - Genetic algorithms, Scientific American, 267(1), pp. 44-50, 1992
J e n s e n , M . T . - Robust and flexible scheduling with evolutionary computation, doctoral thesis,
Dissertation Series DS-01-10, Aarhus University, Denmark, 2001
N i c o a r ă , E . S . - Contribuţii privind utilizarea algoritmilor evolutivi în conducerea proceselor
de fabricaţie în industria medicamentelor, doctoral thesis, Ploieşti, 2007
T a l b i , E . G . - Métaheuristiques pour l'optimisation combinatoire multi-objectif: Etat de l'art,
C.N.E.T (France Telecom) report, Oct. 1999
Z i t z l e r , E . , D e b , K . , L o t h a r , T . - Comparison of multiobjective evolutionary
algorithms: Empirical results, Evolutionary Computation Journal, 8(2), pp. 125-148, 2000
Indicatori de Performanţă
pentru Algoritmi de Optimizare Multiobiectiv
Rezumat
În lumea reală, majoritatea problemelor de optimizare sunt multiobiectiv. Pentru cele complexe şi de
mari dimensiuni adeseori nu avem la dispoziţie algoritmi de complexitate polinomială care să redea
soluţia(soluţiile) exactă(e) în timp util. De aceea, se folosesc algoritmi aproximativi care găsesc soluţii
(aproape) optime în timp util. Împreună cu aspectele legate de compararea a două soluţii multiobiectiv,
apar şi aspectele legate de măsurarea performanţei unui algoritm şi de comparare a doi algoritmi. În
lucrare sunt prezentaţi cei mai importanţi indicatori de performanţă pentru algoritmi de optimizare
multiobiectiv. Drept exemplu, am comparat trei algoritmi genetici aplicaţi pe problema-test biobiectiv
JSSP ft10, iar rezultatele au demonstrat necesitatea considerării simultane a mai multor indicatori de
performanţă pentru algoritmi multiobiectiv.