On the Relation between Solution Quality and Computing Times
of Intelligent Algorithms − A Comparative Study for Standard Cell
Placement in VLSI Design
Karl Kurbel
Europe University Viadrina Frankfurt (Oder), Business Informatics, POB 776,
D-15207 Frankfurt (Oder), Germany; E-mail: [email protected]
Abstract
Many heuristic methods for NP hard optimization problems were developed in the past. Recently several new
approaches based on analogies to processes of nature
have been proposed. Considering the variety of methods
available, a crucial question is which one to choose in
practice. The behavior of any method depends strongly
on parameters. Furthermore, time constraints often play
an important role and the stability of a method may be a
relevant factor to consider. In this paper, some assistance for the decision which method to select will be
given. The methods under investigation are: simulated
annealing, tabu search, threshold accepting, genetic algorithms, and a hybrid called parallel recombinative
simulated annealing (PRSA). The findings reported in
this paper were derived from a large number of tests.
About 220,000 test runs were performed over a time period of about 8 months.
1 Introduction
Many real-world optimization problems are NP hard.
Examples of such problems are job-shop scheduling in
short-term production planning [7], placement of standard cells in the design of VLSI chips [12], and the
traveling salesman problem [9]. Exact optimization is
usually not feasible when problems of practical size are
attacked. Therefore heuristic methods have been developed in large numbers.
In the eighties and nineties, several heuristics derived by
analogy from nature have been proposed for optimization problems. Some of those heuristics are based on the
principles of evolution and selection whereas others use
analogies to physical processes. The methods under
study in this paper are:
• Simulated annealing,
• Tabu search,
• Threshold accepting,
• Genetic algorithms,
• Parallel recombinative simulated annealing (PRSA).
When new methods emerge, promises and expectations
are often quite high. For users in practice, exposed to a
variety of new approaches, it is interesting to know how
the methods are in relation to each other and of course,
which method is the best one. Asking for the "best"
method is somewhat ambiguous, however. Is the best
method, in practice, the one which computes the best
value of the objective function, or is it the fastest one, or
is it the most stable one? For example, is method I better
than method II if it has given excellent solutions in very
few cases but quite bad ones many other times?
Solution quality, runtime and stability all are criteria to
be taken into consideration when judging a method.
Furthermore, those criteria are not independent of each
other. An obvious trade-off to be made, for example, is
the one between solution quality and runtime. All methods have parameters (to be set by the user), and the parameter values have severe impact on both solution
quality and runtime. It will be shown later that all methods can be made fast or slow, making solutions "bad" or
"good", respectively.
Therefore, the effects of parameter settings will be discussed in section 3. An important question for practical
use of the methods will be addressed in section 4: How
do the methods behave when a time constraint to
achieve a solution is imposed? This means, for example,
is it better to run a fast method very often or to run a
slow method only few times within the same given time
limit? Finally, the stability of the methods will be analyzed in section 5. Stability is also a practical concern. It
means: Can the user reasonably expect to get the same
quality of solutions and the same runtimes "as always"
when the same method is applied the next time?
The findings reported in this paper were derived from a
large number of tests. About 220,000 test runs were per-
formed over a time period of about 8 months (elapsed
computing time on four Sun SparcStations20 running in
parallel).
2 The Underlying Test Problem
The problem used for our tests is placement of standard
cells on VLSI chips. The process of chip design goes
through several steps [13]. One of them is physical design. In this stage, standard cells have to be placed in
given rows on a chip as shown in figure 1. Standard
cells are logic modules with pre-designed internal layout. The cells are of the same height but their widths
vary depending on their functionalities. Interconnections
between cells are pre-defined by so-called netlists. A
netlist consists of nets which in turn are made up of
those cells that are to be connected by wires. For a given
number of cells with their dimensions (height and
widths) and a given netlist, the task is to place the cells
on the given layout in such a way that the total length of
interconnecting wires will be minimal [11, 12]. The sum
of the wire-lengths of all nets thus constitutes the objective function.
in terco n n e ctio n
fe ed th ro u g h c ell
ro u tin g c h an n el
b o n d p ad
m ac ro b lo ck
sta n d ard ce ll
w asted sp a ce
Fig. 1: Chip layout after standard cell placement [8]
For all the methods discussed subsequently, the layout
of figure 1 was mapped to a rather simple representation. Since the positions of macro blocks are already
given in this stage, they do not need to be considered
explicitly. Therefore just the order of cells and their positions on the chip (indicated by coordinates, in a given
unit of measurement) have to be mapped. The corresponding information is represented by a list of triplets
as shown in the example of figure 2. The order in the list
defines the order of the cells on the layout. Obviously
cells A to E are in one row as they all have the same y
coordinate.
A set of eight standard cell placement scenarios was
used for the tests. Those scenarios comprise placement
problems of various sizes (from 20 to 100 cells) and
various complexities (from simple to complicated netlists).
Cell
A
B
C
D
E
F
G
H
I
J
X
0
20
50
75
90
0
30
55
70
95
Y
0
0
0
0
0
50
50
50
50
50
Fig. 2: Representation of standard cell positions
as a list (example)
3 Parameters of the Methods under Study
and their Impact on Solution Quality and
Runtime
An extensive study of the effects of parameter settings
on both solution quality (i.e. value of the objective
function) and computing times has been presented elsewhere [6]. The most important findings of that study are
summarized subsequently.
3.1 Simulated Annealing
The simulated annealing (SA) algorithm starts with an
initial feasible solution and a very high temperature.
Then a flip operator is used to interchange two randomly
chosen cells in the list representation as in figure 2. The
new solution is accepted if its wire-length is less than the
previous one; otherwise it might be accepted with a
probability that depends on the temperature. Then the
next pair of cells is selected and treated in the same way.
In each iteration, the temperature is lowered by a small
amount. This process continues until the temperature
reaches a value near zero or until some other termination
criterion is satisfied.
Parameters the user has to set refer to the cooling schedule. The most important one is the cooling factor. In our
tests we used factors of 0.99 (SA-1), 0.999 (SA-2),
0.9999 (SA-3), and 0.99999 (SA-4). Each SA version
was run 20 times for each of the 8 scenarios. Both the
solution quality and the computing times exhibit a wide
distribution. SA-1 and SA-2 are quite fast, whereas SA3 and SA-4 take rather long. However, the solutions calculated by the latter ones are much better. This effect is
shown in figure 3. The underlying scenario is a fairly
complex one (70 cells, typical netlist). The points of the
figure represent average values from the 20 runs of each
version of SA. Note that the runtimes are given on a
logarithmic scale.
Obviously there is a trade-off between solution quality
and computing time. While SA-1 is quite fast, its solutions are very bad. On the other hand, SA-4 gave excellent solutions, but the average runtimes where by orders
Wire-length
of magnitude larger (2:42 minutes versus 4:12:49
hours).
6300
6200
6100
6000
5900
5800
5700
5600
5500
100
SA-1
SA-2
SA-3
SA-4
1000
10000
CPU-time [s]
100000
Fig. 3: Relation between solution quality, computing
time, and cooling factor in simulated annealing (scen. 6)
3.2 Tabu Search
Tabu search is an exhanced local search method. To get
out of local minima, tabu search generates and explores
several solutions at a time. Then it chooses the best one
for further search, even if that solution is not better than
the currently best solution. In this way, not only new
points of the solution space but also old ones, i.e. points
which have been visited before, can be reached again.
To avoid repetition of the same search steps, the concept
of a tabu list is used. Whenever a new solution is chosen
for further search, the moves applied to the old solution
in order to generate the new one are entered into the
tabu list. If the tabu list is full and a new move has to be
recorded, the oldest entry is deleted. This means that
some solutions which were tabu before can be reached
again and thus new regions of the search space may be
explored.
Important parameters are the number n of neighbors to
be searched in one iteration and the length l of the tabu
list [4, 15]. Furthermore, the number c of iterations without change (i.e. after how many iterations in which no
improvement of the current solution has been found
should the algorithm terminate?) has to be set.
We tested six combinations of parameter values because
parameters here are not independent from each other
(see [6] for details). Again, some versions were found to
produce very good solutions but were quite slow
whereas others were by orders of magnitude faster but
came out with bad results. (The best average solutions
were obtained by versions with n = 50 - 100, l = 15 - 45,
and c = 10,000.) A similar trade-off as in figure 3 was
observed.
3.3 Threshold Accepting
Threshold accepting [3] starts with an initial solution
and tries to improve that solution step by step. A socalled threshold value is used to decide whether a new
solution will be kept or not. If the difference between
the old value of the objective function and the new one
is less than the threshold, the new solution is accepted.
Afterwards the threshold is reduced by a given factor.
The initial threshold t is a parameters which has to be
set. Like tabu search, threshold accepting searches the
neighborhood of the currently best solution. Thus the
number of neighbors to be examined (n) has to be fixed.
The number of iterations without any change of the objective function (c) serves as a stopping criterion as in
tabu search.
Seven combinations of parameters were tested (see [6]
for details). The best solutions were obtained by the version with n = 1,500, t = 500, and c = 1,000. This version
was very slow. On the other hand, the fast versions gave
bad results. The strongest influence on solution quality
was found to come from the number of neighbors to be
examined in one iteration (n), as in tabu search. A larger
n means that more solutions will be checked, thus the
probability to find a better solution than the current one
will also increase.
3.4 Genetic Algorithms
The number of parameters of a genetic algorithm [2, 5]
is quite big. First of all, different types of cross-over,
mutation, and inversion operators may be chosen. Second, the respective rates − to how much percent of a
population will cross-over, mutation and inversion be
applied in each generation? − have to be set. Third, the
population size (number of individuals) has to be fixed.
Fourth, a maximum number of generations to be processed may also be predefined if the algorithm is desired
to terminate before complete convergence of the population.
In our tests, three of those parameters were considered:
the cross-over rate cr, the population size s, and the
number of generations m. Based on combinations of the
parameters, four versions were tested:
Parameter
s
m
cr
GA-1
32
1 500
0.9
GA-2
64
2 000
0.975
GA-3
64
2 500
0.99
GA-4
128
2 000
0.975
All of them performed relatively badly, but the trade-off
between solution quality and computing time can still be
observed in figure 4. In that table, the best results rowwise are boldfaced and the worst results are underlined.
Scenario
Result quality
GA-1
GA-2
GA-3
GA-4
Wire-length
CPU-time
Wire-length
CPU-time
Wire-length
CPU-time
Wire-length
best
3810
00:01:56
3780
00:05:43
3840
00:07:12
3750
CPU-time
00:12:14
1
worst
4080
00:01:58
3930
00:05:54
3960
00:07:28
3990
00:17:03
(20 cells)
average
3936
00:01:57
3863
00:05:49
3888
00:07:22
3873
00:14:23
best
3000
00:01:53
2940
00:05:36
2940
00:07:07
2940
00:11:49
2
worst
3210
00:01:55
3150
00:05:51
3179
00:07:18
3240
00:16:54
(20 cells)
average
3139
00:01:54
3056
00:05:43
3075
00:07:13
3072
00:14:19
best
3600
00:01:53
3570
00:05:43
3600
00:07:16
3450
00:11:41
3
worst
3870
00:01:58
3840
00:06:18
3690
00:07:26
3660
00:16:53
(20 cells)
average
3693
00:01:56
3644
00:05:52
3639
00:07:21
3611
00:13:47
best
2138
00:06:04
2062
00:17:29
2024
00:22:29
1980
00:35:43
4
worst
2495
00:06:14
2381
00:18:14
2411
00:23:23
2317
00:53:21
(40 cells)
average
2284
00:06:08
2217
00:17:55
2246
00:22:54
2177
00:41:42
best
3185
00:06:00
3157
00:17:37
3221
00:22:53
3093
00:36:13
5
worst
3696
00:06:16
3516
00:18:34
3494
00:23:40
3427
00:39:27
(40 cells)
average
3451
00:06:11
3344
00:18:10
3335
00:23:25
3272
00:37:37
best
6523
00:16:56
6390
00:49:59
6446
01:04:49
6162
01:39:22
6
worst
7221
00:17:38
6868
00:51:53
6927
01:06:26
6753
01:46:50
(70 cells)
average
6812
00:17:16
6617
00:51:10
6679
01:05:31
6408
01:42:18
best
7339
00:16:42
7278
00:50:02
7260
01:04:56
7000
01:40:25
7
worst
8024
00:17:49
7828
00:52:12
7988
01:06:34
7735
01:30:34
(70 cells)
average
7756
00:17:16
7550
00:51:22
7617
01:05:43
7391
01:43:34
best
3333
00:31:08
3048
01:33:40
3089
01:57:12
2825
01:33:29
8
worst
4378
00:35:07
4032
01:38:25
3988
02:07:40
3566
04:35:52
(100 cells)
average
3758
00:32:50
3535
01:36:15
3516
02:04:44
3179
03:27:38
Fig. 4: Results of genetic algorithms with different parameter settings
3.5 Parallel Recombinative Simulated
Annealing (PRSA)
The name PRSA indicates that this method uses both the
population concept, including recombination operators,
from genetic algorithms and features like cooling from
simulated annealing [10]. PRSA generates new individuals by applying genetic recombination (cross-over)
and mutation (flip) operators. New populations are constructed by selection from old and new individuals with
the help of Boltzmann trials [1]. In contrast to normal
genetic algorithms, the selection process is controlled by
the temperature.
In this hybrid method, the number of parameters and parameter combinations is very high. However, most parameters had been tested with the respective methods
before, and their effects in PRSA were expected to be
more or less the same. Therefore our investigations focussed on the selection strategy. This parameter represents very well the basic idea of PRSA, namely combining SA and GA features for generating new solutions.
In principle, both old (parent) and new (child) individuals could be considered as candidates for the next
population. Therefore they compete against each other
in PRSA, two children and two parents at a time. Four
types of competition were tested:
1. Children compete against the worst parents.
2. Each child competes against one of its own two
parents.
3. Children and parents compete in reverse order of
their fitness values: the best child against the worst
individual from the set of all parents, the second-best
child against the second-worst parent, etc.
4. Each child competes against one random individual
from the set of parents.
The winners of each competition are then selected for
the next generation with a certain probability (Boltzmann criterion).
Out of the four selection strategies, the second one gave
the best solutions. This was somewhat counter-intuitive
because we had expected that a strategy which replaces
explicitly bad individuals, like a) or c), would be superior. Again the runtimes of the best version were much
longer than those of the worse versions.
In relation to each other, the methods behaved as follows (for details see [6]):
1. Simulated annealing was the overall best method,
both from the solution quality and the runtime point
of view.
2. Tabu search came relatively close, with the exception that two of the six versions tested gave the overall worst solutions.
3. Threshold accepting had similar results as the first
two methods, with solutions approximately 5 %
worse.
4. PRSA produced good solutions, even a little better
than threshold accepting, but runtimes were by far
the longest ones of all methods under study.
5. The genetic algorithms gave the worst solutions of
all methods, excepting the two version of tabu search
mentioned above.
Method
Simulated
annealing
Tabu
search
Version
fast
SA-0
slow
SA-4
fast
TS-1
slow
TS-6
Threshold
accepting
fast
slow
TA-2
TA-7
Genetic
algorithm
fast
GA-1
slow
GA-4
fast
slow
PRSA-0
PRSA-4
PRSA
4 Impact of a Given Time Constraint on
the Behavior of the Methods
Parameters
cooling factor 0.98
cooling factor 0.99999
n = 10, l = 6, c = 1000
n = 75, l = 45,
c = 10000
n = 8, t = 500, c = 1000
n = 1500, t = 500,
c = 1000
s = 32, m = 1500,
cr = 0.9
s = 128, m = 2000,
cr = 0.975
s = 16, strategy a)
s = 64, strategy b)
In the first series of tests, each version of each method
ran exactly 20 times. When the same time limit was allowed in the second series, all methods ran more often
(with the exception of PRSA-2). The impact of the same
time on the number of runs is quite remarkable. Figure 5
shows the distribution of runs that were actually carried
out. Some very fast versions ran several thousand times
− in the same time in which PRSA-2 ran only around 10
and SA-4 between 22 and 44 times.
In this section, the question of how time constraints, or
more general, how fixed time budgets affect the above
ordering will be discussed. In section 3, the findings
were derived from a uniform number of 20 runs for each
version of each method. This is not really a fair basis for
comparison if practical constraints are taken into consideration, because some versions/methods take hours or
days whereas others need only seconds. Therefore a
second series of test runs was designed.
For all methods except PRSA-2, the effect was observed
that increasing the number of runs improved the solution
quality. The results of both test series are presented in
figure 6. For each version/method, the data from the
"same runtime" tests are given in the respective left column and the data from the "20 runs" test are given in the
respective right column, set in italics for distinction. As
before, best results are set in bold face and worst results
are underlined.
In this series, all methods were given the same amount
of time, implying that a "fast" method could run more
often than 20 times whereas a "slow" method would
probably run less often. However, each method should
be able to run a certain minimum number of times so
that random effects would be eliminated to some extent.
For this purpose, the average runtime recorded for the
slowest method was taken for each scenario and multiplied by 10 (i.e. the minimum number of times each
method ran was approximately 10).
A look at the respective two columns of each version reveals that there are both improvements and changes to
the worse regarding the wire-lengths in the new versions. Most versions now found better solutions than before. For example, SA-0, the fast version of simulated
annealing, performed better or the same as before in all
scenarios (see rows "best"). The reason is obviously that
it ran far more often than just 20 times (e.g for scenario
In the second set of test runs, only one fast version and
one slow version per method were selected. As discussed above, the fast versions usually gave bad results
and the slow versions usually gave good results. The
following versions were used for the comparisons:
Scenario
SA-0
SA-4
TS-1
TS-6
TA-2
TA-7
PRSA-0
PRSA-2
GA-1
1 (20 cells)
9946
22
3616
62
3621
21
80
9
178
GA-4
29
2 (20 cells)
10468
24
3656
70
3809
22
82
8
152
25
3 (20 cells)
12085
27
4358
79
4422
25
97
12
202
33
4 (40 cells)
7856
29
8752
103
7026
59
115
12
193
33
5 (40 cells)
7752
28
8444
93
6565
56
96
10
243
41
6 (70 cells)
5420
44
19569
179
7275
133
104
12
621
108
7 (70 cells)
5389
44
18811
182
7184
131
100
10
689
118
8 (100 cells)
3814
32
32226
316
7485
229
146
11
433
74
Fig. 5: Distribution of runs executed within the same time
Scen.
Res.
qual.
PRSA-0
PRSA-2
SA-0
SA-4
GA-1
GA-4
TS-1
TS-6
TA-2
TA-7
same
20
same
20
same
20
same
20
same
20
same
20
same
20
same
20
same
20
same
20
time times time times time times time times time times time times time times time times time times time times
runs
runs
runs
runs
runs
runs
runs
runs
runs
runs
1
(20 c.)
2
(20 c.)
3
(20 c.)
4
(40 c.)
5
(40 c.)
6
(70 c.)
7
(70 c.)
8
best
3750 3780 3750 3750 3750 3810 3750 3750 3750 3810 3750 3750 3780 3780 3750 3750 3750 3840 3780 3780
worst
4050 4050 3840 3810 4170 4050 3780 3750 4170 4080 3960 3990 3990 3960 3930 3930 4320 4140 3930 3930
average 3874 3863 3777 3771 3907 3959 3753 3750 3931 3936 3864 3873 3900 3887 3780 3791 4009 3999 3839 3839
best
2940 2940 2940 2940 2940 2940 2940 2940 2940 3000 2940 2940 2940 3000 2940 2940 2940 3030 2940 2940
worst
3240 3120 3000 3030 3330 3210 2940 2940 3330 3210 3179 3240 3210 3180 3060 2970 3480 3300 3060 3090
average 3050 3039 2966 2948 3068 3074 2940 2940 3122 3139 3049 3072 3080 3101 2958 2957 3165 3192 3004 2997
best
3450 3450 3450 3450 3450 3450 3450 3450 3509 3600 3450 3450 3450 3510 3450 3450 3450 3600 3450 3450
worst
3780 3720 3540 3600 3840 3690 3450 3450 3870 3870 3750 3660 3750 3690 3600 3600 4020 3900 3630 3660
average 3613 3620 3480 3510 3628 3621 3450 3450 3677 3693 3627 3611 3639 3629 3518 3552 3729 3717 3554 3557
best
1976 2047 1967 1891 1894 2086 1857 1911 2065 2138 2005 1980 2167 2346 1913 1916 1955 2166 1925 1974
worst
2365 2309 2085 2080 2578 2384 2006 1991 2600 2495 2336 2317 2589 2483 2129 2129 2756 2487 2186 2103
average 2166 2166 2012 2001 2199 2196 1940 1942 2305 2284 2168 2177 2392 2410 1994 1991 2264 2279 2036 2037
best
2927 3065 2885 2907 2952 3069 2859 2885 3121 3185 3081 3093 3264 3425 2859 2885 2942 3171 2877 2877
worst
3409 3375 3047 3068 3736 3425 2983 2976 3764 3696 3461 3427 3747 3648 3244 3103 3829 3645 3215 3180
average 3192 3217 2950 3001 3275 3266 2904 2927 3437 3451 3279 3272 3539 3515 2998 3010 3341 3376 3042 3040
best
5722 5971 5639 5539 5688 5994 5452 5367 6276 6523 6068 6162 6632 6820 5486 5541 5740 6147 5604 5637
worst
6498 6397 5924 6024 6695 6344 5775 5761 7515 7221 6841 6753 7471 7202 6028 5911 6954 6788 6101 6227
average 6093 6136 5789 5806 6200 6154 5578 5558 6826 6812 6474 6408 7075 7051 5660 5682 6321 6397 5858 5827
best
6700 6776 6515 6568 6600 6838 6325 6349 7159 7339 7024 7000 7609 7803 6343 6422 6693 6985 6458 6458
worst
7478 7360 6917 6875 7644 7501 6656 6646 8347 8024 7733 7735 8469 8205 6902 6834 7876 7731 7112 7027
average 7067 7102 6718 6736 7135 7099 6454 6467 7766 7756 7378 7391 8038 8020 6607 6658 7278 7363 6771 6778
best
2434 2428 2218 2160 2255 2335 1982 2059 2801 3333 2705 2825 3918 4222 1971 1949 2300 2682 2097 2255
worst
3340 3413 2580 2654 3458 3130 2205 2193 4657 4378 3767 3566 5210 4812 2424 2253 3997 3439 2949 2725
(100 c.) average 2859 2870 2368 2404 2787 2809 2095 2115 3767 3758 3214 3179 4570 4577 2154 2115 3039 2989 2483 2487
Fig. 6: Results of test series 1 and 2
1 it ran actually 9,946 times). However, the worst solutions also became much worse. Apparently the variance
increased. The average wire-lengths did not change significantly overall. SA-4, the slow version, did not only
remain the best one as before but still improved somewhat. In four out of five scenarios it found better solutions (for scenarios 1 to 3 it had already reached the optimum before). The averages also improved. SA-4 obtained the best solutions on average both in the sameruntime and in the 20-times runs.
The second question asked in section 1 can be answered
now. In particular:
a) Does the same-time constraint have any effect on the
decision which method is the best one?
b) Will the solutions be better if a fast version of a
method is run very often or if a slow version is run
only a few times?
Looking at the result tables, question a) is easy to answer. Although almost all methods became better in the
sense that better solutions were found, SA-4 proved to
be by far the best one in the second test series as well.
Therefore simulated annealing, with very slow cooling,
is the overall winner. The time constraint did not change
this situation.
The second question may be put as follows: Is it better
to set the parameters in a way that, even though one individual solution is unlikely to be very good, among the
large number of solutions obtained within the given time
hopefully there will be an outstanding one? Or is it better to tune the parameters so that better solutions are
likely to be computed from the beginning on but due to
the time limit there will only be a few ones to select
from?
Our results show that in none of the cases a "bad" version of a method from the 20-times test series became
better than the corresponding "good" version by running
very often. As can be seen from figure 6, the differences
in the numbers of runs are quite big. However, although
in some cases (e.g. simulated annealing, tabu search) the
fast versions were run 100 times more often than the respective slow versions, the latter ones still found better
solutions. The answer to the second question is therefore
that, according to our test results, it is better to run slow
versions, with good parameter settings, only a few times
instead of fast versions many times.
only once the probability that the result will be as good
as the average result is quite high.
5 Stability of the Methods
The final question to be discussed is related to the
problem examined in the previous section. For a prospective user in an application field, stability of the results obtained by a method is an important factor to
consider. This means: can the user expect that the solutions computed for one particular problem will be approximately of the same quality and need the same runtime "as always"?
The results given in sections 3 and 4 have shown that the
variances of the wire-lengths and runtimes are sometimes quite big. For the purpose of comparison of the
methods, the coefficients of variation were computed
[14]. Such a coefficient indicates the standard deviation
of the results relative to the average. A small coefficient
implies that application of the corresponding method is
likely to produce a result close to the average.
In figure 7, the coefficients of variation are shown both
for the average wire-lengths and for the average runtimes obtained. For each method, one fast and one slow
version was considered. The basis for calculation is the
results of the 20-times runs given in section 2.
4,00%
3,50%
3,29% 3,14%
3,00%
2,89% 2,87% 2,85%
2,38%
2,50%
2,04%
2,00%
1,83%
1,65%
1,50%
0,88%
1,00%
0,50%
SA-4
TS-1
TS-6
PRSA-2
TA-7
PRSA-0
SA-0
GA-4
TA-2
0,00%
GA-1
Coefficient of variation [%]
Coefficients of variation for wire-length
Method
SA-4
TS-1
TS-6
PRSA-2
TA-7
PRSA-0
SA-0
GA-4
TA-2
45,00%
37,27%
40,00%
32,57%
35,00%
30,73%
30,00%
26,13%
25,00%
18,48%
20,00%
15,35%
15,00%
10,01%
10,00%
2,95%
5,00% 1,22%
1,29%
0,00%
GA-1
Coefficient of variation [%]
Coefficients of variation for CPU-time
Method
Fig. 7: Coefficients of variation for wire-length
and CPU time [14]
The coefficients of variation for wire-length are not very
big. They are between 0.88% (SA-4) and 3.29% (GA-1),
i.e. solutions are mostly around the average. From these
values it may be concluded that when a method runs
On the other hand, the coefficients of variation for CPU
time vary significantly. For the slow version of tabu
search (TS-6) the coefficient is 37.27% but for the fast
genetic algorithm (GA-1) it is only 1.22%. Thus the
runtime of TS-6 may vary quite a lot from one run to
another. In contrast to this, methods like simulated annealing with slow cooling (SA-4), the fast genetic algorithm (GA-1), and the slow version of threshold accepting (TA-7) most probably will take almost the same
amount of CPU-time in different runs for the same
problem.
From the stability point of view, simulated annealing in
version SA-4 again exhibited the best behavior. The
runtimes for all scenarios were very close to the respective average runtimes, and the final values of the objective function were also quite near together. This means
that the user does not need to fear random effects but
may expect a good solution in an average time.
6 Summary and Conclusions
The purpose of this paper was to evaluate several heuristic methods for sequencing problems which have been
proposed in the past years. Those methods are simulated
annealing, tabu search, threshold accepting, genetic algorithms, and PRSA (parallel recombinative simulated
annealing). The focus of the investigation was set on
questions that arise when the methods are to be applied
in practice: How to set the parameters? Should they be
set in a way that a method becomes slow and hence can
only run few times within a given time limit? Or is it
better to make a method fast and run it many times instead? Furthermore, can the user expect that the results
of a certain method will be of the same quality as before
when the method runs the next time?
Answers to those questions were derived from a large
number of test runs with different parameter settings for
each method. The general findings may be summarized
as follows. Appropriate parameter settings are of outstanding importance. Any method can behave very well
or very badly depending on the values and combinations
of its parameters. The user has to make a trade-off between solution quality and runtime. Usually it is better to
choose good parameter values, knowing that those values will slow the method down and permit only few runs
to be carried out within a certain time limit. In none of
the test scenarios we found any indication that the random effect of running the "bad" version of a method
very often would allow some extraordinarily good solution to be captured on the way. Most likely the "bad"
versions get stuck in local minima from which they can-
not escape due to the parameter settings. Furthermore, in
no case the fast version of a method outperformed the
respective slow method in terms of solution quality by
running a large number of times.
son, D.W., Steele, N.C., Albrecht, R.F. (eds.), Artificial Neural Nets and Genetic Algorithms − Proceedings of the International Conference in Ales,
France; Vienna 1995, pp. 491-494.
The best method of both test series was simulated annealing, when the parameter controlling the cooling
schedule was set in a way that the method became very
slow (SA-4). How strong the impact of the cooling
schedule is had been observed in our first experiments
where we employed a cooling factor of 0.98. The
method was extremely fast but the solutions were so bad
that at the first moment we even considered not to use
simulated annealing any more!
[9] Lawler, E.L., Lenstra, J.K., Rinnooy Kan, A.H.G.,
Shmoys, D.B.: The Traveling Salesman Problem:
A Guided Tour of Combinatorial Optimization;
New York, NY 1985.
Simulated annealing with very slow cooling (SA-4,
cooling factor 0.99999) proved to be the most stable
method as well. Both solution quality and computing
time were always very close to the respective averages.
Based on the results from the two test series, SA-4 may
be considered as the best method for the underlying test
problem.
References
[1] Aarts, E.H.L., Korst, J.H.M.: Simulated Annealing
and Boltzmann Machines − A Stochastic Approach
to Combinatorial Optimization and Neural Computing; Chichester, New York, et al. 1989.
[2] Davis, L.: Handbook of Genetic Algorithms; New
York, NY et al. 1991.
[3] Dueck, G., Scheuer, T.: Threshold Accepting: A
General Purpose Optimization Algorithm Appearing Superior to Simulated Annealing; Journal
of Computational Physics 90 (1990), pp. 161-175.
[4] Glover, F.: Tabu Search − Part I; ORSA Journal
on Computing 1 (1989) 3, pp. 190-206.
[5] Goldberg, D. E.: Genetic Algorithms in Search,
Optimization, and Machine Learning; Reading,
MA 1989.
[6] Kurbel, K.: A Computational Study of the Effects
of Parameters on the Performance of Intelligent
Algorithms; in: Proceedings of Fourth World Congress on Expert Systems 1998, Mexico City (to
appear).
[7] Kurbel, K., Ruppel, A.: Integrating Intelligent Jobscheduling into a Real-world Production-scheduling System; Journal of Intelligent Manufacturing
7 (1996), pp. 373-377.
[8] Kurbel, K., Schneider, B., Singh, K.: VLSI Standard Cell Placement by Parallel Hybrid Simulated-Annealing and Genetic Algorithm; in: Pear-
[10] Mahfoud, S.W., Goldberg, D.E.: Parallel Recombinative Simulated Annealing; IlliGAL Report No.
92002 (Illinois Genetic Algorithms Laboratory),
University of Illinois, Urbana, IL 1992, pp. 1-27.
[11] Sait, S.M., Youssef, H.: VLSI Physical Design
Automation − Theory and Practice; London, New
York et al. 1995.
[12] Shahookar, K., Mazumder, P.: VLSI Placement
Techniques; ACM Computing Surveys 23 (1991)
2, pp. 143-220.
[13] Sherwani, N.: Algorithms for VLSI Physical Design Automation; Boston, MA 1993.
[14] Singh, K., Bochynek, R.: Comparison of Heuristic
Search Methods for Sequencing Problems − A
Computational Study; in: First International Conference on Operations and Quantitative Management, Proceedings: Volume II; Jaipur, India, Janu-ary 5-8, 1997, pp. 451-458.
[15] Werra, de D., Hertz, A.: Tabu Search Techniques:
A Tutorial and an Application to Neural Networks;
Operations Research Spektrum 4 (1989) 11, pp.
131-141.