Generalizing the Improved Run-Time Complexity
Algorithm for Non-Dominated Sorting
Félix-Antoine Fortin
Simon Grenier
Marc Parizeau
[email protected] [email protected] [email protected]
Laboratoire de vision et systèmes numériques
Département de génie électrique et de génie informatique
Université Laval, Québec (Québec), Canada G1V 0A6
ABSTRACT
years 2000 use Pareto domination to guide search, for example non-dominated sorting genetic algorithm II (NSGAII) [8], strength Pareto evolutionary algorithm 2 (SPEA2)
[16], Pareto archive evolution strategy (PAES) [13], Pareto
envelope-based selection algorithm (PESA) [4], and Paretofrontier differential evolution (PDE) [1]. Even today, Zhou
et al. [15] observed that a majority of MOEAs in both research and application areas still share a common framework as that of NSGA-II. More recently Deb et al. [7] proposed modifications to NSGA-II that improves convergence
for problems with high number of objectives. The time complexity of these algorithms is generally dominated by the
Pareto ranking of problem solutions and therefore the optimization of this procedure is key to faster MOEAs.
The straightforward brute-force procedure for determining Pareto ranking of problem solutions is an O(M N 3 ) algorithm, where M is the number of objectives, and N the
number of solutions. Essentially, one may first compare in
O(M N 2 ) time the dominance matrix of the O(N 2 ) solution
pairs for all M objectives in order to obtain the first Pareto
front. But this process must be repeated for O(N ) possible fronts, in the worst case, after removal of the solutions
from the previous fronts. In 2002, by efficiently computing
and managing domination counts for solutions, Deb [8] introduced the so-called “Fast Non-Dominated Sorting” approach
with a reduced O(M N 2 ) complexity. Soon after in 2003,
Jensen [12] was able to further reduce this time complexity
to O(N logM −1 N ), allowing much improved runtimes for
common problem sizes and number of objectives. However,
this algorithm assumes that no solutions can share identical
values for any objective. The author claims that removing
this assumption is easy, to which we beg to differ. There are
multiple non-trivial cases that have to be considered when
applying this algorithm to the general case. Furthermore, in
the likely event of two solutions sharing an identical value
for one of the objectives, the fronts computed by the Jensen
algorithm can be erroneous, leading to suboptimal results.
To tackle real-world multi-objective problems, both fast and
correct algorithms are definitely needed.
The incompleteness of Jensen’s algorithm and its repercussion on the proper solving of optimization problems have
been reported previously by Fang et al. [10]. They came
to the conclusion that the algorithm proposed by Jensen is
unable to generate the same non-dominated fronts as the
original NSGA-II algorithm on the same populations of the
DTLZ1 benchmark [9]. Even though their explanation for
this discrepancy is somewhat incorrect, they nevertheless
This paper generalizes the “Improved Run-Time Complexity
Algorithm for Non-Dominated Sorting” by Jensen, removing
its limitation that no two solutions can share identical values for any of the problem’s objectives. This constraint is
especially limiting for discrete combinatorial problems, but
can also lead the Jensen algorithm to produce incorrect results even for problems that appear to have a continuous
nature, but for which identical objective values are nevertheless possible. Moreover, even when values are not meant
to be identical, the limited precision of floating point numbers can sometimes make them equal anyway. Thus a fast
and correct algorithm is needed for the general case. The
paper shows that generalizing the Jensen algorithm can be
achieved without affecting its O(N logM −1 N ) time complexity, and experimental results are provided to demonstrate
speedups of up to two orders of magnitude for common problem sizes, when compared with the correct O(M N 2 ) baseline
algorithm from Deb.
Categories and Subject Descriptors
F.2.2 [Analysis of Algorithms and Problem Complexity]: Nonnumerical Algorithms and Problems—Sorting and
searching
General Terms
Algorithms, Performance
Keywords
Non-dominated sort; time complexity
1.
INTRODUCTION
Some of the most well-known multi-objective evolutionary algorithms (MOEAs) developed at the beginning of the
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. To copy otherwise, to
republish, to post on servers or to redistribute to lists, requires prior specific
permission and/or a fee.
GECCO’13, July 6–10, 2013, Amsterdam, The Netherlands.
Copyright 2013 ACM 978-1-4503-1963-8/13/07 ...$15.00.
615
1: procedure NonDominatedSort(P, M )
S ← ({f (p) | p ∈ P }, <1:M )
2:
3:
for all s ∈ S do
4:
frt(s) ← 1
5:
end for
6:
NDHelperA(S, M )
7:
for i = 1, . . . , max{frt(s) | s ∈ S} do
8:
Fi ← {}
9:
end for
10:
for all s ∈ S do
11:
i ← frt(s)
12:
Fi ← Fi ∪ {p ∈ P | f (p) = s}
13:
end for
14:
return F1 , F2 , . . .
15: end procedure
1: procedure NDHelperA(S, k)
if |S| < 2 then return
2:
3:
else if |S| = 2 then
(1)
Figure 1: Main procedure of the generalized non-dominated
sorting. It receives a population P where each individual
has a M objectives fitness and returns a sequence of Pareto
fronts on which reside these individuals.
Figure 2: Recursive procedure NDHelperA. It attributes
front indices to all fitnesses in S for the first k objectives.
1: procedure SweepA(S)
T ← {s(1) }
2:
3:
for i = 2, . . . , |S| do
demonstrated that correcting this deficiency is non-trivial.
Moreover, the rectification of the algorithm is strongly advised as it is currently used in real-world applications, for
example in Davoodi et al. [5] who use the algorithm to solve
a multi-objective discrete robot path planning problem.
The main contribution of this paper is the clear presentation of a generalized algorithm for the case where solutions
can share an arbitrary number of equal objective values.
We show that the additional steps for solving the general
case do not affect the overall time complexity. Finally, we
show through experiments that very interesting speedups are
achievable in practice, over a wide range of common problem
sizes.
The remainder of this paper is structured as follows. Algorithm changes required for solving the general case are
presented next, in Section 2. Then, the impact of these
changes on time complexity is analyzed in Section 3, and
experimental speedups are provided in Section 4. Conclusions are finally drawn in Section 5.
2.
(2)
4:
if s1:k ≺ s1:k then
5:
frt(s(2) ) ← max{frt(s(2) ), frt(s(1) ) + 1}
6:
end if
7:
else if k = 2 then
SweepA(S)
8:
9:
else if |{sk | s ∈ S}| = 1 then
10:
NDHelperA(S, k − 1)
11:
else
12:
L, H ← SplitA(S, k)
13:
NDHelperA(L, k)
14:
NDHelperB(L, H, k − 1)
15:
NDHelperA(H, k)
16:
end if
17: end procedure
(i)
4:
U ← {t ∈ T | t2 ≤ s2 }
5:
if U 6= {} then
r ← max{frt(u) | u ∈ U }
6:
7:
frt(s(i) ) ← max{frt(s(i) ), r + 1}
8:
end if
9:
T ← T \ {t ∈ T | frt(t) = frt(s(i) )}
10:
T ← T ∪ {s(i) }
11:
end for
12: end procedure
Figure 3: Two-objectives sorting procedure SweepA. It attributes front index to lexicographically ordered elements of
S based on the first two objectives using a line-sweep algorithm.
∀i < j. This order is preserved when S is later split into
subsets.
A front index is then associated with every fitness (lines
3 to 5). It is initially set to one and can only be increased
as fitnesses are compared to each other. The recursive procedure NDHelperA is next applied (line 6) on the fitnesses
for the M objectives. When the procedure returns, each fitness has been associated with its proper front index. The
fronts are finally assembled from individual solutions in P
based on their fitness (lines 7 to 13).
Procedure NDHelperA is presented in Figure 2. Since
at least two fitnesses are required to establish a dominance
relationship, the first step is to exit the procedure (line 2)
when there are less than two elements in S. If the number of
fitnesses is two (line 3), they are compared to each other directly. Since fitnesses are already lexicographically ordered,
s(1) cannot be dominated by s(2) . We therefore only need to
check whether s(2) is dominated by s(1) and, if so, to adjust
its front.
Next, if the problem has only two objectives (line 7), it
can be solved via a sweep procedure described below. This
situation was not considered in the original algorithm as it
was presented for problems where M ≥ 3 and because of
ALGORITHM DESCRIPTION
To simplify notations, instead of sorting solutions, the presented algorithm works directly on fitnesses. The fitness
q = f (p) of an individual solution p is given by function f
that returns a list of M objective values. In the algorithm,
we assume that all objectives need to be minimized. Let qk
denote the kth objective of fitness q, q1:k the first k objectives, and q1:k ≺ t1:k the dominance of fitness q over fitness
t for the first k objectives. The Pareto front associated with
fitness q is retrieved via function frt(q). It can also be
modified with the assignment operator ←.
The algorithm is divided into 7 procedures presented in
Figure 1, 2, 3, 5, 7, 8, and 9. The changes to Jensen’s
algorithm are highlighted in these figures; the procedures of
Figure 3 and 9 are new.
The main procedure is described in Figure 1. The function
takes two arguments: the population P and the number of
objectives M . The first step (line 2) is to create a totally ordered set S = [s(1) , . . . , s(|S|) ] of unique fitnesses associated
with the individuals p ∈ P . The elements of the set are lexicographically ordered over the M objectives, s(i) <1:M s(j) ,
616
4
4
2
3
3
5
6
5
5
4
4
3
2
1
1
F1
2
3
4
objective 1
5
6
1
2
5
5
F2
2
F3
objective 2
objective 2
6
4
3
4
objective 1
5
3
3
F1
1
F4
3
4
objective 1
5
(d) i = 3, U = {s(1) , s(2) }, r = 2
6
3
4
objective 1
5
6
F4
6
4
5
F2
2
F3
3
3
F1
F3
4
3
F2
3
2
1
1
2
2
(c) i = 2, U = {s(1) }, r = 1
4
4
2
1
1
1
6
(b) Initialization
6
F1
1
1
(a) Fitnesses of S
2
2
1
1
F2
2
3
1
objective 2
2
objective 2
objective 2
5
6
objective 2
6
5
F1
1
1
1
2
3
4
objective 1
5
6
(e) i = 4, U = {s(1) , s(2) , s(3) }, r = 3
1
2
3
4
objective 1
5
6
(f) i = 5, U = {s(1) }, r = 1
Figure 4: Step-by-step illustration of the SweepA procedure for a simple example with 5 fitnesses. Subfigures represent steps
of the procedure, and captions indicate important variables at these steps.
the distinct fitness values assumption, k could not decrease
inside NDHelperA. For the general case, however, this is
no longer true.
When all values for objective k are equal (line 9), this
objective cannot be used to rank the fitnesses, and we simply
skip to the next objective. This is the only condition that
causes the value of k to be decreased inside this procedure
and it is the reason why the sweep procedure presented in
Figure 3 is required when k = 2.
Procedure SweepA handles the special case where the fitnesses in S all have identical values for objectives 3 through
M . Their front indices can therefore be determined solely
on their respective dominance on the first two objectives.
The procedure is similar to the non-dominated sorting algorithm for two objectives presented by Jensen. However, this
sweep takes into account that fitnesses can have initial front
indices greater than 1. The algorithm starts by defining a
set T of fitnesses associated with front indices. There can be
at most one fitness per front index in this set. The elements
of T are referred to as stairs in the original algorithm. Set T
is initialized by the first element of S. In sequence, for each
s(i) ∈ S, a set U is built with all elements of T that have a
lower or equal second objective (line 4). If U is not empty, it
means that some other fitness dominates s(i) , and its front
index must thus be adjusted according to the highest front
found in U (lines 6 and 7). Thereafter, if a fitness in T has
the same front index as s(i) , it is first removed from the set
(line 9), and s(i) is always added to T (line 10).
An example of this sweep procedure is represented graphically in Figure 4. We start with a lexicographically ordered
set containing five fitnesses numbered from 1 to 5. For the
sake of this example, we assume that the initial front index
of each is 1. The elements of T are represented in this figure
by horizontal lines; the so-called “stairs”. The first element
of S initializes set T and defines the current stair for front
1. Because the second element has a higher 2nd objective,
it will be added to set T after having its front index incremented to 2. The third element is more interesting. It has
the same value for objective 2 as one of the existing stairs. It
will force the creation of a new stair for front index 3. Next,
element 4 has the same value for objective 1 as element 3,
but the value of its 2nd objective is necessarily higher, because S is always lexicographically sorted. Again, this will
create a new front index 4. As for the fifth element, set U
will only contain the first element corresponding to front 1.
It will thus replace in T the representative of front 2.
Back to Figure 2, if the problem has more than two fitnesses and two objectives, it is partitioned into smaller subproblems and solved by recursion. S is split into two sets
L and H around the median of the kth objective (line 12).
Contrary to the original algorithm, these two sets are not
guaranteed to be of equal size since multiple fitnesses can
share the value of the median. The impact of this on the
run-time complexity will be analyzed in Section 3.
The SplitA procedure is detailed in Figure 5. It receives
set S and splits its fitnesses into two output sets according to
objective k. The median value for objective k is first determined (line 2). Next, sets L and H are created with fitnesses
that respectively have values lower (line 3) and higher (line
4) than this median. Those with values equal to the median
617
1: procedure SplitA(S, k)
2:
med ← median{sk | s ∈ S}
3:
L ← {s ∈ S | sk < med}
4:
H ← {s ∈ S | sk > med}
if |L| ≤ |H| then
5:
6:
L ← L ∪ {s ∈ S | sk = med}
else
7:
8:
H ← H ∪ {s ∈ S | sk = med}
end if
9:
10:
return L, H
11: end procedure
1: procedure NDHelperB(L, H, k)
if L = {} or H = {} then return
2:
3:
else if |L| = 1 or |H| = 1 then
4:
for all h ∈ H, l ∈ L do
if l1:k h1:k then
5:
6:
frt(h) ← max{frt(h), frt(l) + 1}
7:
end if
8:
end for
9:
else if k = 2 then
10:
SweepB(L, H)
11:
else if max{lk | l ∈ L} ≤ min{hk | h ∈ H} then
12:
NDHelperB(L, H, k − 1)
13:
else if min{lk | l ∈ L} ≤ max{hk | h ∈ H} then
L1 , L2 , H1 , H2 ← SplitB(L, H, k)
14:
15:
NDHelperB(L1 , H1 , k)
16:
NDHelperB(L1 , H2 , k − 1)
17:
NDHelperB(L2 , H2 , k)
18:
end if
19: end procedure
Figure 5: Partition procedure SplitA. It divides in two a
set of fitnesses S around its median for the objective k and
uses the elements equal to the median to balance the sets.
6
objective k
5
Figure 7: Recursive procedure NDHelperB. It attributes
a front index to fitnesses of H for the first k objectives by
comparing them to those of L.
4
3
2
1
1
2
3
4
5
index
6
7
8
1: procedure SweepB(L, H)
2:
T ← {}
3:
i←1
4:
for j = 1, . . . , |H| do
9
Figure 6: Extreme case handled by the SplitA function where more than half of fitnesses have an equal
value for objective k.
The resulting split is: L =
{s(1) , s(2) , s(3) , s(4) , s(5) , s(6) , s(7) } and H = {s(8) , s(9) }.
5:
6:
7:
8:
9:
10:
11:
12:
13:
14:
15:
16:
17:
18:
19:
are then assigned to the set having the smallest number of
elements. In this way, fitnesses are partitioned into balanced sets, as much as possible. In the case where objective
k has many equal values, the split may become unbalanced.
For example, Figure 6 illustrates an extreme situation where
more than half of the values are equal and centered around
the median.
After the split of S over objective k, NDHelperA proceeds with recursive calls to solve for L and H (lines 13 and
15). However, through procedure NDHelperB (line 14), in
between these two calls, the initial front indices of H need
to be adjusted according to the fronts of L. After the first
recursive call, the fronts of L are final, because their elements cannot be dominated by those of H. Moreover, only
the remaining k − 1 objectives need to be taken into account
by NDHelperB, since the SplitA procedure has already
made sure that no element of H can dominate an element
of L according to objective k.
Procedure NDHelperB is presented in Figure 7. If any
of L or H is empty, the procedure simply returns (line 2);
nothing needs to be done. If one of the sets contains a single
fitness (line 3), then every fitness in the other set (line 4) is
directly compared to this fitness on objectives 1 to k (line
5). If l1:k is dominating, that is l1:k ≺ h1:k , then the front
index of h needs to be at least one more than the index
of l (line 6). The same is true if l1:k = h1:k , since it was
determined by previous recursive calls that l dominates h
for every objective greater than k.
(i)
(j)
while i ≤ |L| and l1:2 ≤ h1:2 do
(i)
R ← {t ∈ T | frt(t) = frt(l(i) ) ∧ t2 < l2 }
if R = {} then
T ← T \ {t ∈ T | frt(t) = frt(l(i) )}
T ← T ∪ {l(i) }
end if
i←i+1
end while
(j)
U ← {t ∈ T | t2 ≤ h2 }
if U 6= {} then
r ← max{frt(u) | u ∈ U }
frt(h(j) ) ← max{frt(h(j) ), r + 1}
end if
end for
end procedure
Figure 8: Two-objectives sorting procedure SweepB. It attributes front indices to H based on the first two objectives
by comparing them to elements of L using a line-sweep algorithm.
Next, if the problem has only two objectives (line 9), a
sweep-line algorithm akin to procedure SweepA is applied.
The SweepB procedure is presented in Figure 8. The algorithm starts by defining a set of stairs T . However, contrary
to SweepA, this set is initially empty (line 2). An index
counter i is also initialized (line 3) and will be used to access
elements of L in order. Then, in sequence, for each element
h(j) ∈ H, the algorithm determines whether it is possible to
add stairs to set T . The stairs are only composed of the fitnesses of L and these elements are added in order. A fitness
l(i) is eligible to be a stair if it is lexicographically smaller
than the current element h(j) on the first two objectives (line
5). When such a fitness is found in L, a set R is built with
618
1: procedure SplitB(L, H, k)
2:
if |L| > |H| then pivot ← median{lk | l ∈ L}
3:
else pivot ← median{hk | h ∈ H}
4:
end if
5:
L1 ← {l ∈ L | lk < pivot}
6:
L2 ← {l ∈ L | lk > pivot}
7:
H1 ← {h ∈ H | hk < pivot}
8:
H2 ← {h ∈ H | hk > pivot}
if |L1 | + |H1 | ≤ |L2 | + |H2 | then
9:
10:
L1 ← L1 ∪ {l ∈ L | lk = pivot}
11:
H1 ← H1 ∪ {h ∈ H | hk = pivot}
else
12:
13:
L2 ← L2 ∪ {l ∈ L | lk = pivot}
14:
H2 ← H2 ∪ {h ∈ H | hk = pivot}
15:
end if
16:
return L1 , L2 , H1 , H2
17: end procedure
and another with higher values (lines 6 and 8). To balance
the partition, the fitnesses with values equal to the pivot
are attributed to the pair of sets with the smallest summed
cardinality.
After the partition of L and H into subsets, it is known
that: L1 and H1 share a common interval for the objective k; L2 and H2 also share a common interval; and L1
contains fitnesses with strictly lower values than H2 for the
kth objective. Based on this knowledge, the assignment of
front indices to elements of H based on elements of L can
be achieved by three recursive calls. First, H1 is ranked according to L1 on the first k objectives (line 15). Second, it is
known that elements of L1 have a strictly lower objective k
than H2 , therefore, a recursive call is made to assign ranks
to H2 according to L1 using the first k − 1 objectives (line
16). Finally, H2 is ranked according to L2 on the first k
objectives (line 17).
Figure 9: Partition procedure SplitB. It divides two sets
of fitnesses L and H around the median of the longest set
for the objective k into four sets L1 , L2 , H1 and H2 . The
values equal to the median are used to balance the pair of
sets with the least number of elements.
3.
COMPLEXITY ANALYSIS
The running time of the algorithm depends on whether
the fitness splitting is balanced or unbalanced, which in turn
depends on the distribution of the values for each objective.
If the splitting is balanced, the runtime is asymptotically
as fast as the algorithm presented by Jensen. However, if
the splitting is unbalanced, because many values are equal,
the algorithm can be asymptotically as slow as the fast nondominated sort of Deb [8]. In this section, we analyze the
performance of the algorithm under the assumptions of balanced and unbalanced splits. We also address the space
complexity of the algorithm.
all the elements of T that have the same front index as l(i)
and a smaller second objective (line 6). This is because a
stair representing a front index is always associated with the
fitness of the lowest second objective encountered so far in
L for that front. If set R is empty, the fitness can be added
as a stair in T . As in SweepA, if a fitness in T has the same
front index as l(i) , it is first removed from the set (line 8)
and l(i) is always added to T (line 9). This loop is executed
until every element of L has been processed or until element
l(i) is lexicographically greater than h(j) on the first two objectives. Next, again as in SweepA, a set U is built with all
elements of T that have a lower or equal second objective
than h(j) (line 13). If U is not empty, it implies that some
elements of T dominate h(j) , and its front index must be
adjusted according to the highest front in U (lines 15 and
16).
Back to Figure 7, if none of the preceding cases occurs, the
problem might have to be subdivided again. When all values
for objective k in L are either lower or equal to any value
for this objective in H (line 11), no further process is needed
for this objective, and the procedure is called recursively to
solve for objective k − 1 (line 12).
When there is an overlap between values of objective k
for L and H (line 13), then L and H need to be split again
around a common pivot. This is achieved by procedure
SplitB of Figure 9 that produces L1 , L2 , H1 , and H2 . In the
original algorithm, sets L and H can be split independently
around the median of objective k, since it is assumed that
the median is unique. But this assumption no longer holds,
L and H need to be split jointly. Neglecting this case can
result in elements of L2 dominating elements of H1 , which
is contrary to a hypothesis formulated by Jensen.
SplitB basically acts as procedure SplitA. However, instead of receiving a single set, it receives both L and H.
The partition point is first determined as the median of the
set with the highest cardinality for the objective k (lines 2
to 4). Next, each set L and H is partitioned into two subsets around the pivot, one with lower values (lines 5 and 7)
3.1
Best-case: balanced split
The best-case occurs when the split procedures can partition the sets evenly at each recursive call. This happens
whenever there are no more than a constant number of values equal to the median, for each objective k. The recurrence relations for the running times TA and TB of procedures NDhelperA and NDhelperB are those formulated
by Jensen [12]:
TA (N, M ) = 2TA
N
2
, M + TB
N
2
,
N
2
,M − 1
+ Tsplit (N )
TB (L, H, M ) = TB (L1 , H1 , M ) + TB (L2 , H2 , M )
+ TB (L1 , H2 , M − 1) + Tsplit (L, H)
where N = L + H = |S|. Similar recurrence relations were
investigated by Monier [14] and based on his results and the
demonstration made by Jensen, we know that
TB (L, H, M ) = O(N logM −1 N ).
By replacing TB in the recurrence equation for TA , it simplifies to
TA (N, M ) = 2TA N2 , M + O(N logM −2 N ).
The results
of Bentley [2] gives the fact that if T (N ) =
2T N2 + O(N logM N ), then T (N ) = O(N logM +1 N ). We
can therefore conclude that the time complexity for TA is
TA (N, M ) = O(N logM −1 N ).
Furthermore, based on the study by Bentley, we know that
the runtime bound can be reduced by a factor O(log N ) for
619
3.2
objective 2
each objective where all values are equal. In the absolute
best case, every objective k ≥ 3 is equal and the presented
algorithm is bounded by the lexicographical sort and the
2-D sweep resulting in a complexity of O(N log N ).
Worst-case: unbalanced split
As stated by Bentley [2], if the multidimensional divideand-conquer strategy employed here was applied to onedimensional problem, the resulting algorithm would be similar to a quicksort that always partitions around the median
of the set. From this, we can deduce that the worst-case of
our algorithm is similar to the one of the quicksort [3], that
is when the split procedure produces one set, either L or H,
containing |S| − 1 elements and the other with only 1 element for each objective k ∈ [3, M ]. Then, the TA recurrence
relation becomes:
objective 1
Figure 10: Hypercube benchmark example with M =2,
T =25 and where we sampled 19 points distributed along
8 distinct Pareto fronts. Points in light shade of grey were
not sampled.
TA (N, M ) = TA (N − 1, M ) + TA (1, M )
+ TB (N − 1, 1, M − 1) + Tsplit (N )
In the average case, the number of values equal to the
median represents a very small fraction of the total number
of values to be split in two. In most cases, the resulting
partition will therefore be almost even and the resulting time
complexity will be the same as the one identified for the best
case O(N logM −1 N ).
with the following base cases:
TA (1, M ) = O(1)
TA (2, M ) = O(M )
TA (N, 2) = O(N log N )
TB (1, N − 1, M ) = O(M N )
3.4
TB (N − 1, 1, M ) = O(M N )
Tsplit (N ) = O(N )
And TA can be simplified to:
TA (N, M ) = TA (N − 1, M ) + O(1) + O(M N ) + O(N )
= TA (N − 1, M ) + O(M N )
Then, by telescoping the sum of these equations:
TA (N, M ) = TA (N − 1, M ) + CM N
TA (N − 1, M ) = TA (N − 2, M ) + CM (N − 1)
..
.
4.
TA (3, M ) = TA (2, M ) + CM (3)
EXPERIMENTS
To compare the runtimes of our generalized algorithm, we
use three benchmarks. Each benchmark presents a different
Pareto front structure where each objective is minimized.
They simulate different possible cases and give an approximation of the average runtimes for different population sizes
and different numbers of objectives.
The first benchmark is a sampled hypercube. We start
by computing a number k of discrete values per objective,
with k = dT 1/M e, based on a fixed maximum population
size T and the number of objectives M . Then, a regular
M -dimensional grid composed of kM points is built, and N
points are randomly sampled on this grid. In the example of
Figure 10, M = 2, T = 25, and N = 19. The points in light
shade of grey are the ones that were not sampled. Given the
fixed problem structure and high probability of equal values,
this benchmark is used to both confirm the correctness of
the generalized algorithm and measure the time impact of
the added special cases. For our experiments, we arbitrarily
set T = 106 .
The second benchmark is a random sampling of the objective space. N points are sampled randomly in space [0, 1]M .
For this benchmark, the probability of having equal objective values is very low.
TA (2, M ) = C + CM (2)
and cancelling out the recurrent terms, we obtain the worstcase time complexity of TA (N, M ) = O(M N 2 ). Therefore,
under specific configurations of fitness values, the worst possible time complexity of our algorithm is equivalent to the
time complexity of the original non-dominated sort algorithm proposed by Deb et al. [8].
3.3
Space complexity
The algorithm proposed by Deb et al. [8] requires a space
complexity in the order of O(M N + N 2 ). To correct the
Jensen procedure which uses O(M N ) space, we only added
a structure that keeps unique fitnesses in addition to the
original set of individuals P . Given the case where all fitnesses are unique, the storage requirement is also O(M N ),
therefore the total space complexity is O(2M N ) or simply
O(M N ). Consequently, even in the worst-case, our algorithm has the same time complexity as the Deb et al. algorithm, but requires N times less space which is another
remarkable advantage.
Average case
When splitting with SplitA, if at least d|S|/2e elements of
S have the same value for an objective k, the mode and the
median will coincide. Whether this configuration leads to
the worst case depends on the distribution of values around
the median. If exactly one value is greater than the median,
the worst case split occurs, and if this happens for every
objective k ≥ 3, the resulting time complexity is the worstcase O(M N 2 ). This implies that at least N/2 fitnesses share
the same value for the objective M , (N − 1)/2 for the objective M − 1, etc., which is highly improbable given the
typical values for N used with multi-objective optimization
problems.
620
103
The last benchmark is the DTLZ1 minimization problem
defined by Deb et al. [9]. The number of xi variables in
a solution is n = M + k − 1, where M is the number of
objectives and k is a problem parameter that establishes the
number of local Pareto fronts (11k −1). For the experiments,
k was set to 5 as suggested by Deb et al. [9].
Each problem was executed with M = 2, 3, 4, 6 and 8
objectives, and for 19 populations sizes N = 100, 200, . . . ,
900 and N = 1000, 2000, . . . , 10000. For each combination
of N and M , both the classical O(M N 2 ) and generalized
O(N logM −1 N ) algorithms were executed 25 times. Over
7125 different populations were thus Pareto sorted using
both algorithms and results compared to validate correctness. Speedups were calculated as the average classical over
generalized runtimes. Both algorithms were implemented
in Python using the DEAP [11] framework for evolutionary computations. The benchmarks were executed under
CPython 2.7.3 on a single node of the Calcul Québec supercomputer Colosse supercomputer (dual quad-core Intel
Nehalem CPUs at 2.8 GHz).
The speedup curves for each problem are illustrated in
Figure 11. Plots have been drawn with log-log scales. The
relation between the population size and the speedups in
logarithmic scales being fairly linear, a curve of equation
f (x) = 10b xm has been fitted to each point series. The
parameters m and b of each curve and their residual r are
presented in Table 1.
From the graphs, we can see that the speedup is directly
related to the number of objectives and the population size.
The highest speedups are achieved when the number of objectives is low and the population is large. They reach
their paroxysm with the DTLZ1 problem when M = 2 and
N = 10000, where the improved algorithm is 515 times faster
than the baseline algorithm. However, even for low population sizes and high number of objectives, the resulting
speedup is always strictly greater than one.
The results obtained with the hypercube problem confirm
the theoretical time complexity for the average case. As a
matter of fact, the fitnesses sampled for this problem are
known to share identical values. Therefore, even though it
is possible that the split may not partition the set of fitnesses evenly, the algorithm average runtime is unaffected
and remains superior to the ordinary non-dominated sorting algorithm.
In some cases for high objective numbers and low population size, we observe that some of the speedup curves
intersect. These small anomalies can be explained by the
fact that for high objective number and small population
size, the overheads implied by the different function calls
can be greater than the actual processing time required to
sort these fitnesses. Furthermore, Deb et al. [6] have demonstrated that the size of the population should grow exponentially with the number of objectives in order to maintain a
certain level of selection pressure. For example, to obtain at
most 30% of the solutions on the first non-dominated front,
he estimates a minimum population size around 800 individuals for a problem with 6 objectives and more than 2000
individuals for a problem with 8 objectives.
5.
2
3
4
6
8
speedup
102
objectives
objectives
objectives
objectives
objectives
101
100
102
103
population size
104
(a) Hypercube
103
2
3
4
6
8
speedup
102
objectives
objectives
objectives
objectives
objectives
101
100
102
103
population size
104
(b) Random
103
2
3
4
6
8
speedup
102
objectives
objectives
objectives
objectives
objectives
101
100
102
103
population size
104
(c) DTLZ1
Figure 11: Speedup achieved by the generalized algorithm
as a function of population size for different numbers of objectives for the three benchmark problems.
lem [10], which was one of the benchmark problem used by
Jensen himself.
We have adapted Jensen’s algorithm to the general case
where fitnesses can share an arbitrary number of equal values for any objective. This adaptation also allowed us to
combine into a single algorithm the two specials cases, for
M = 2 and M ≥ 3, that were previously treated separately
by Jensen. We have demonstrated that the run-time complexity remains unaffected at O(N logM −1 N ), for the best
case as well as the expected case, and that the worst-case
CONCLUSION
In this paper, we have addressed the issue of equal value
fitnesses that can lead the Jensen non-dominated sort algorithm to fail, for instance when applied on the DTLZ1 prob-
621
Nb.
obj.
2
3
4
6
8
m
0.751
0.814
0.840
0.693
0.515
Hypercube
b
r
-0.288 0.017
-0.986 0.003
-1.419 0.003
-1.367 0.030
-0.965 0.013
m
0.724
0.757
0.710
0.539
0.443
Random
b
-0.228
-0.903
-1.174
-1.030
-0.821
r
0.011
0.000
0.003
0.006
0.003
m
0.760
0.767
0.703
0.520
0.426
DTLZ1
b
-0.303
-0.916
-1.125
-0.864
-0.619
r
0.009
0.000
0.003
0.005
0.002
Table 1: Fitted parameters of the equation f (x) = 10b xm and the residual r for each combination of number of objectives
and problem.
degenerates to an upper bound of O(M N 2 ), corresponding
to the classical Deb algorithm, but using only O(M N ) storage instead of O(N 2 ). Experiments using Deb’s algorithm as
baseline have confirmed that we are indeed able to generalize
the Jensen algorithm while maintaining the expected performances, and that our speedup results even exceed those
reported by Jensen. Furthermore, we explicitly made sure
that both sorts produced the exact same output in order
to assess the algorithm correctness. We can thus conclude
that our modifications of the Jensen algorithm should be
used to gain significant performance improvement over the
Deb algorithm, for all common problem sizes, and that it
should also be used instead of Jensen’s original algorithm to
guarantee the correctness of the non-dominated sort.
The new implementation of the general non-dominated
sort is publicly available for download under LGPL in DEAP
release 1.01 . It is not specific to NSGA-II and could easily be
adapted to a multitude of other MOEAs that use dominance
ranking.
[7] K. Deb and H. Jain. Handling many-objective
problems using an improved NSGA-II procedure. In
Evolutionary Computation (CEC), 2012 IEEE
Congress on, pages 1–8, june 2012.
[8] K. Deb, A. Pratap, S. Agarwal, and T. Meyarivan. A
fast and elitist multiobjective genetic algorithm:
NSGA-II. IEEE Transactions on Evolutionary
Computation, 6(2):182–197, Apr. 2002.
[9] K. Deb, L. Thiele, M. Laumanns, and E. Zitzler.
Scalable multi-objective optimization test problems.
In Congress on Evolutionary Computation (CEC
2002), pages 825–830. IEEE Press, 2002.
[10] H. Fang, Q. Wang, Y.-C. Tu, and M. F. Horstemeyer.
An efficient non-dominated sorting method for
evolutionary algorithms. Evolutionary Computation,
16(3):355–384, 2008.
[11] F.-A. Fortin, F.-M. De Rainville, M.-A. Gardner,
M. Parizeau, and C. Gagné. DEAP: Evolutionary
algorithms made easy. Journal of Machine Learning
Research, 2171–2175(13), Jul. 2012.
[12] M. Jensen. Reducing the run-time complexity of
multiobjective EAs: The NSGA-II and other
algorithms. IEEE Transactions on Evolutionary
Computation, 7(5):503–515, Oct. 2003.
[13] J. Knowles and D. Corne. Approximating the
nondominated front using the pareto archived
evolution strategy. Evolutionary computation,
8(2):149–172, 2000.
[14] L. Monier. Combinatorial solutions of
multidimensional divide-and-conquer recurrences.
Journal of Algorithms, 1(1):60–74, 1980.
[15] A. Zhou, B.-Y. Qu, H. Li, S.-Z. Zhao, P. N.
Suganthan, and Q. Zhang. Multiobjective evolutionary
algorithms: A survey of the state of the art. Swarm
and Evolutionary Computation, 1(1):32–49, 2011.
[16] E. Zitzler, M. Laumanns, and L. Thiele. SPEA2:
Improving the strength Pareto evolutionary algorithm.
Technical Report 103, Computer Engineering and
Networks Laboratory (TIK), Swiss Federal Institute of
Technology (ETH) Zurich, Gloriastrasse 35, CH-8092
Zurich, Switzerland, May 2001.
Acknowledgment
This work was supported by the Fonds de recherche du
Québec - Nature et technologies (FRQNT) and by the Natural Sciences and Engineering Research Council of Canada
(NSERC). Computations were made on the Colosse supercomputer at Université Laval, under the administration of
Calcul Québec and Compute Canada.
6.
REFERENCES
[1] H. Abbass, R. Sarker, and C. Newton. PDE: A
pareto-frontier differential evolution approach for
multi-objective optimization problems. In Proceedings
of the 2001 Congress on Evolutionary Computation,
volume 2, pages 971–978, 2001.
[2] J. L. Bentley. Multidimensional divide-and-conquer.
Commun. ACM, 23(4):214–229, Apr. 1980.
[3] T. H. Cormen, C. E. Leiserson, R. L. Rivest, and
C. Stein. Introduction to Algorithms, Third Edition.
MIT Electrical Engineering and Computer Science
Series. The MIT Press, 3rd edition, 2009.
[4] D. Corne, J. Knowles, and M. Oates. The pareto
envelope-based selection algorithm for multiobjective
optimization. In Parallel Problem Solving from Nature
PPSN VI, pages 839–848. Springer, 2000.
[5] M. Davoodi, F. Panahi, A. Mohades, and S. N.
Hashemi. Multi-objective path planning in discrete
space. Applied Soft Computing, 13(1):709–720, 2013.
[6] K. Deb. Multi-Objective Optimization Using
Evolutionary Algorithms. Wiley, 2001.
1
http://code.google.com/p/deap
622