lDepartment of Biostatistics, School of Public Health and Curriculum in
Operations Research and Systems Analysis.
2Analytic Services, Inc., Falls Church, Virginia.
THE GRADIENT LINEAR OPTIMIZATION METHOD
by
Richard H. Shachtman
1
and Eileen P. Neely
2
Department of Biostatistics
University of North Carolina at Chapel Hill
Institute of Statistics Mimeo Series No. 1041
November 1975
THE GRADIENT LINEAR OPrIMIZATION METHOD
Richard H. Shachtroan 1 and Eileen P. Neely2
1.
Introduction
Due to the notable expense necessarily incurred in the solution
of large-scale linear programs with the algorithms which are currently
available, efforts to streamline the solution techniques continue.
Sare
studies have suggested tha.t the key to nore efficient solution may be
the decision rule (rules) which is (are) used to detennine the pivot elarent
for change of basis: [3,4,5].
This paper presents an algori thIn which
is a m:xlification of the Bounding Hyperplane Method (BHPM) , [4] .
In
particular, a different criterion is suggested for selection of the vector
which is leaving the basis.
The proposed Gradient Linear Optimization (GLO) approach is one
nernber of the class of primal-dual algorithms for the solution of linear
programs [1,
Chp. 8].
In these algorithms, a decision is made at each
iteration, based on the current state of the problem, whether to use a
pri.nal or a dual iteration to nove to the next basic solution for the
problem.
The purpose of such a design is to reduce the number of iterations and tine
required for solution.
As pointed out by Saksena and Cole [4], this type
of procedure is not restricted to novement wi thin the feasible region of
the linear program.
At each iteration, the algorithm can lead to a solution
loepa.rt.rr'ent of Biostatistics, School of PublJic Health and Curriculum in
Operations Research and Systems Analysis.
2Anaiytic Services, Inc., Falls Church, Virginia.
-2-
which is either primal feasible [1, chp.
1], dual feasible [1, chp.
roth primal and dual feasible, or neither primal nor dual feasible.
8] ,
A
primal feasible basic solution corresponds to an extrerre point of the
polytope generated by the constraints, while a dual feasible basic solution
which is not also primal feasible does not lie on the polytope.
A basic
solution which is roth primal and dual feasible is optimal and is an extrerre
point of the polytope.
In sane cases this increased freedan of IIDvement
rray allow improved efficiency as compared to primal and/or dual algorithms.
Another advantage introduced by this freedcrn of IIDverrent is that selection
of an initial basic solution is easy since primal feasibility is not required.
2.
The Linear Program
The Gradient Linear Optimization method is designed to be used for
•
the solution of linear programs of the for:m:
where
and
b
x
is an
is an
In
n x 1
x 1
Maximize
c'x
Subject to
~::
•
e.,
vector, A is the
vector.
In X
n
matrix of coefficients,
(Note that since the sign of
a feasible region defined by Ax > b
is also admissible.)
!?
is unirrp::>rtant,
After the in-
equalities are converted to equalities by the introduction of a vector
of slack variables, where each slack variable has an associated cost of
s
0,
the problem can be written as:
Maximize
c'x + O's
Subject to
A.x+ Is
= !?,
- - Q,
x
>
- ->
s
O.
•
J
At each iteration, the current solution is either primal feasible or dual
feasible, or else the problem can be identified as infeasible.
Assuning
that the problem is not identified as being infeasible, then the dual class
algorithm will be used in the event that the problem is dual feasible;
otherwise, an algorithm designated as primal class 3 will be employed.
In
the event that a solution is both primal feasible and dual feasible, the
solution is optimal and no further iterations are needed.
Thus at any iteration,
the choice between algorithms is clearly defined.
For purposes of discussion of the GID method, it is necessary to define
sorre notation.
Throughout the paper,
~,
A,
~,~
are used to represent the
solution vector, matrix of constraint coefficients, the vector of costs, and
the vector of reduced costs, respectively.
an element of
(Here, using standard notation,
Z is defined as
Z.:: z. - c.. The vector Z can then be
J
J
J
interpreted as the normal to a family of hyperplanes where each hyperplane
L z.
is of the fonn
let
I
and
J
of the matrix A.
II
=
{i
E
defined by
I
I
J1
x.
=K
for an arbitrary constant
K.)
In addition,
jEJ J J
be the set of indices of the raws and columns, respectively,
let the subset of row indices
Xi < O}
=
{j
E
II
C
I
be defined by
furthernore, let the subset of column indices
J
I
Zj < O}.
Then
J
l
C
J be
II contains the indices of
constraints which are currently primal infeasible, while
indices of columns which are currently dual infeasible.
J
l
contains the
Finally, let Irod
Z
be the rrodified reduced cost hyperplane which is obtained by replacing the
negative coefficients (if any) of the reduced cost hyperplane by their rrDduli;
3The tenn "primal class" will be used to designate the algorithm which is used
for primal feasible problems. It should be noted that this algoritlml is also
used in circumstances where the problem is neither primal nor dual feasible.
-4-
i. e., m:x1
Z is a typical hyperplane in the family of hyperplanes having
IzJ.Ix.J = K,
the fo:rm
3.
where
K is an arbitrary constant.
The Algorithm
GLO is an iterative procedure in which the first step in each iteration
must be a dete:rmination of the current status of the solution.
Because the
primal-dual approach does not require that the solution at each iteration
be primal feasible, it is always possible to take the initial basic solution
directly from the problem.
The solution then progresses according to the
following rules:
(1)
If the current solution is both primal feasible (for all
i E I, x. > 0)
1
-
j E J, Z. > 0) ,
and dual feasible (for all
J -
then the current solution is optimal.
(2)
If the current solution is not dual feasible, Le. , it is
either primal feasible or else neither primal nor dual
feasible, two cases are possible, i . e., for
0i ~ 0
a.
If
b.
If there exists
o.1
as defined below,
for all i E I, then the primal is unbounded; Stop.
i
such that
o.1
< 0, use the primal
class algorithm to proceed to the next iteration.
(3)
If the current solution is dual feasible but not primal
feasible, again tv,{) cases are possible, i . e. ,
a.
If for sorce
j
E
i
o
E
I, x.
1
< 0
o
and a.
1
0
J, then the primal is infeasible.
. > 0
J Stop.
for ali
•
-5-
b.
If conditions (1) and (3a) are not net, but the current
solution is dual feasible, proceed to the next iteration
by using the dual class algorithm.
The GW Pri.nal Class Algorithm
If there exists
j EJ
with
z.
J
< 0,
a restricted problem is generated
by rerroving all coluzms which correspond to variables which are dual feasible.
In order to select the pivot
reM,
the following value is calculated for each
constraint in the reduced problem:
O.
1
If
e.1
I
. J
=
JE 1
n
1J
a 2..
(I
a .. Z.) /
J
j=l
1J
represents the angle between the normal to the
hyperplane and
ith
the normal to the reduced cost hyperplane, then
cos
e·1 =
a .. Z.) /
1J
«
J
n
I
j=l
2
a .. )
1J
n
(I z.
j=l
2
Since the second factor in the denominator is constant over
that
o.
is proportional to
1
cos
e..
1
I o.1
T = {i E I
if rET
it is clear
O. < 0
1
is maximum.
and then, from am:mg these
M)re concisely, if
< O}, then the rth row is chosen as the pivot raw if and only
-
= max I oil. This selection results in IIDverrent to the
r
iET
having the normal which forms the smallest angle with the normal
and
hyperplane
i,
The exit vector is chosen by restricting
consideration to the constraints for which
choosing that raw for which
»
J
I0 I
to the negative redoced cost plane of the reduced problem.
The enter vector
is selected in the usual rranner (Le., as for the simplex algorithm) in order
to maximize the increase in the objective function.
(See, for example, [1, chp. 4].)
Maxtrix calculations are then executed for the whole tableau.
-6-
The GW Dual Class Algorithm
Z. > 0
J -
If
j
for all
E
but there exists an
J
i
E
I
with
x.
< 0,
1
a restricted problem is generated by rerroving all rows corresponding to variables
which are primal feasible.
'Ib select the exit vector for the dual class
algorithm, we define
n
a..
1
The
kth
Since
=
n
(I
a .. Z.
1J
j=l
/
J
(I
j=l
2
1J
a ..
vector is selected as the exit vector if and only if
a. is proportional to
1
cos
e.,
1
lakl = max
10.··1·
iEI
1
the effect of this decision rule
iJ-
to nove to the hyperplane characterized by having the nonnal which fonns
the largest
angle with the nonnal to the reduced cost hyperplane.
The
enter vector is then chosen in the usual manner in order to maximize the
decrease in the objective function.
4.
(See, for exarrple, [1, chp. 8].)
•
Comparison of GLO and BHPM
The GID procedure described above is consistent with BHPM [4]
in that t\'oO classes of algoritlnns are defined, one of which is to be used in
the event that the current solution is dual feasible,
with the secorrl algorithm
to be used if the criterion of dual feasibility is not rret.
The t\\O rrethods
diverge, hoNever, with regard to the specification of the decision rules for
choosing the exit vector.
In both procedures, the exit vector is selected
first, whereas in the sirrplex algorithm the enter vector is selected first.
In BHPM,
the goal is to nove to the nearest bounding hyperplane (as
vie~
frcm the current extreme point) in the direction of the nonnal to the reduced
cost plane.
In contrast, the GID approach uses a positional criterion (as
measured by the angle between nonnals) rather than a distance criterion.
-7-
5.
Examples
Example 1.
This exanq;>le was incllrled in [4] as Example 2 and is introduced
here in order to p:::>int up the difference in the sequence of basic solutions
which can occur basoo on the choice of decision rules.
Maximize
0.75X
Subject to
l
-
150~
0.25x l
60~
The problem is to
+ 0.02x - 6.0X4
3
- 0.04x + 9.0x4 < 0
3
x
< 1
-
3
x.
J
->
0,
j=1,2,3,4.
saksena and Cole introduced this as an unpublished example of cycling (due
to Beale) for the simplex IIEthod.
t\oK)
Solution with the GI..O approach requires
iterations, which is the sane number required by the BHPM approach.
The
tableaus generated during the solution of this problem by the GID approach
are shown in Tables I-III.
this paper with the
Canparison of the tableaus in Tables I-III of
corresp:::>nding tableaus in Tables V-VIII of [4] reveals
a difference in the sequence of pivotal elements.
(Note:
In Example 1 as
found in [4], use of the GID IIEthod to find the optimal solution resulted
in the
saIIE
sequence of basic solutions as for BHPM.)
e
e
•
e
'rnBLE I
Initial Tableau for Example 1
P j -+
+i
c+
.75
P2
-150
P
3
0.02
P
4
-6
.25
-60
-.04
9
PI
P5
0
P
6
0
P
7
0
x+
1
0
0
0.0 -0.00308
Basis
1
P
5
Calculations:
-
O.l.
Primal Algorithm
ANGLE
89.77°
REMARKS
3 j
E:
J
~
Z. < 0, so
J
2
P
6
0.5
-90
3
P
7
0
0
Z.
-.75
150
-.02
3
0
1
0
0.0
-0.00416
89.68°
1*
0
0
0
1
1.0 -0.01414
88.92 °
-.02
6
0
0
0
I
co
I
J
Note:
The Pivotal Elerrent is indicated by an asterisk.
0
Primal Class
Algorithm is
used.
TABLE II
Tableau after one iteration for Exanq;>le 1
Pj+
+i
c+
PI
.75
P
2
-ISO
P
3
0.02
P4
P
-6
0
s
P
6
0
Calculations :
P7
0
x+
-
Basis
0.
1
Prirra1 Algorithm
1
P
s
0.2S
-60
0
9
1
0
0.04
0.04
-0.00309
89.76
2
P
6
P
3
O.S*
-90
0
3
0
1
0.02
0.02
-0.00416
89.68
0
1
0
0
0
LO
LO
3
0
0.0
REMARKS
ANGLE
0
0
90.00°
3
j
E:
J
3
Z. < 0, so
J
Prirra1 class
Algorithm is
used. With the
BHPM. Algorithm,
I
0'\
I
the basis after
one iteration
included PS '
PI and P7 •
Z.
J
Note:
0.7S
ISO
0
6
0
0
0.02
The Pivotal E1errent is indicated by an asterisk.
0.02
•
e
e
e
'I2illLE I I I
Tableau after two iterations for Example 1
Pj +
PI
P2
c+
.75
-150
+i
P
3
0.02
P4
P5
-6
P
0
P
6
0
0
Basis
I
Calculations:
7
xi-
1
P
5
0
-15
0
7.5
1
-0.5
0.03
0.03
2
PI
1.0
-180
0
6
0
2
0.04
0.04
o
r-l
I
3
P
Z.
J
3
0
0
1
0
0
0
1
1.0
0
15
0
10.5
0
1.5
0.05
0.05
O.l.
ANGLE
None Required
REMARKS
Solution is
optimal. Value
of the objective function is 0.05.
-11-
Example 2.
Consider the linear program
Maximize
1.0x + 4.0x
l
2
Subject to
-2x
1
- 7.0x
2
<
-163
-5x
1
- 8.0x
2
<
-227
-4x
1
- 1.0x
2
<
- 52
- 1.0x
2
-<
- 17
-4x
1
+ 3.0x2 <
-
76
-l~
+ 1.0x2 <
-
28
-2x
1
+
3.0~
-<
98
-lx
1
+ 3.0x2
-<
118
-lx
1
+ 7.0x2 <
-
310
3x
l
+ 8.0x2 <
-
520
l~
+
1.0~
<
-
95
5~
+
4.0~
-<
433
5~
+
3.0~
-<
396
4x
l
- 1.0~
-<
208
lX
l
- 1.0x
2
<
-
34
lX
- 2.0x
<
-
14
->
0
->
O.
l
2
xl
x
2
The diagram of the feasible region for this problem is shown in Figure 1.
Solution by rooans of the GI.D approach required two
iterations, while
five iterations were requried using the BHPM awroach.
'Ihe simplex algorithm,
where rroverrent is restricted to the polytope of the feasible region shown
in Figure 1,
~uld
require additional iterations due both to the Phase I
rerroval of artificial variables as well as the requirenent (which is inherent
in the simplex algorithm) of rroving to neighboring extrerre points.
-12-
Figure 1
50
40
(57,37)
(8,36)
e
(5,32)
30
(58,24)
20
10
a
40
Objective: Maximize
60
5
lx
1
+ 4x2
•
TABlE N
Initial Tableau for Example 2
I {-i
7 8
j-+
1
2
3 4
5
6
c-+
1
4
0
0
0 0
0
0
9 10
11
12
0
0
0
0
13 14
0
0
15
16
17
18
0
0
0
0
Calculations :
x{-
Basis
1
6
7
8
3
P4
P
5
P
6
P
7
P
8
P
9
P10
9
10
P11
P12
11
P
13
P14
P15
2
3
4
5
I
."
~
I
P
12
13
14
15
16
P
16
P
17
P18
Z.
J
Noh...
-2
-7
-5
-8
-4
-1
3
1
-2
-1
3
3
-1
3
1
5
7
8*
5
3
1
1
- 17
76
1
28
98
118
1
1
1
1
310
520
95
433
1
1
1
4
1
1
4 -1
1 -1
-2
-1
-4
396
208
1
0
0
0
0 0
0
0
0
0
0
0
0
The Pivotal Element is indicated by an asterisk.
0
0
0
34
1
14
0
0
REMARKS
:3
j
E:
J ~
z.
J
< 0,
so Primal class
- 52
1
1
1
MQ.E
-163
-227
1
-4 -1
0 -1
6.l.
Pri.rcal AlgorithIn
algorithm is used;
-1.569
-1. 732
-2.673
-3.317
-3.781
-4.069
-2.887
-3.240
-2.874
67.63 °
65.16°
49.59°
36.45°
23.51°
9.32 °
45.56°
38.20°
45.82°
P12 is selected to
leave an:i P2 to
enter.
e
TABLE V
Tableau After One Iteration
.63
3
-3.63
4
.38
5
-5.13
1'"o
-1.38
, 7
PC)
-3.13
i
0
D
410
-2.13
'-J
v
-3.63*
6
.l.
~
"11
1
1
1
1
1
1
1
48
used;
-119 -0.5253 137.95°
-.13
-37 -0.4399 128.47°
leave and
-.38
-97 -0.5299 138.54°
to
-.38
-77 -0.5256 138.01°
-.88
-145 -0.58281145.50°
.13
2
1.j
PiS
3.88
.38
1'16
4.38
.13
1"17
1.38
.13
1. 75
.25
~
16
-,
, 18
i
,__-.2J
;~ote:
0.5
The Pivotal Elerrent is indicated by an asterisk.
P11
-.38
::'2
...
hence Dual Class
E
.13
.63
3.5
293
-
Algorithm is
t
14
Z.
J
13
I •.
13
1
> 0,
j
292
.13
.38
J
e
Exarrp1e 2
1.0
1
P2
L',-'
e
.88
1
-2
J
-.13
-.5
J,
is
selected to
P1
enter.
65
1
30
1
0.5~:
173
1
201
1
273
1
99
1
144
260
I
I
I
.
.
i
I
TABLE VI
Tableau After Tv.o Iterations for Exanple 2
+i
J+
1
2
3
4
5
6
7
8
9
10
c+
1
4
0
0
0
0
0
0
0
0
11
12
13
14
15
16
17
0
0
0
0
0
0
0
Calculations
18
0
BASIS
X+
,-
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
P3
P
4
P
5
P
6
P
7
P
8
P
9
P
10
P1
P
2
P
13
P
14
P
15
P
16
P17
P
18
Zj
1
1
1
1
1
1
1
1
1
.17
.72
267
.55
1.48
373
1
158
is optim
- 1
1
8.1 ANGLE REMARKS
Solution
.1
.03
33
Value of
-1.4
.86
86
the ob-
-.38
.21
18
jective
-.86
.38
28
function
-.59
.14
8
-.28
.24
40
.1
.03
50
.17
-.28
.97 -1.34
1.07 -1.31
1.21 -.93
.38
-.21
.48
-.17
.138
.379
5
1
1
33
46
1
98
1
44
1
1
74
240
is 240.
.,
e
e
e
TABLE VII
'
~----------t"
: c;","!l,'r
I
',1\
i
kt.'(/>.- -
._--!--rJ'r"hll,m~
ID
...
Vpctor h
(RHS)
Vector c
_.---_.- ----
27
25
I
i 20x201
xed RIIS
% Sparse
1 20
xed RIIS
% Sparse
i,
II
i
5
xed RHS
Sp.lrse
i
;;20 '
!V
I~Ox20 I
I
:O;;t,,)!
i
~
.-l
I
2tJ
2ox/,ol
JO
,--
_2\1"l,
"
GLO
BHPM
GLO
BHPM
GLO
BHPM
GLO
BHPM
1GLO---
26.593
2.007
22.592
5.213
0.820
0.384
2.147
0.366
0.082
0.909
0.474
1.918
0.365
0.085
1000
[-100,100]
50.400
28.880
1. 745
19,472
9.532
'J.770
0.442
1. 742
0.316
0.163
0.855
0.526
1.625
0.345
0.161
Ell
1000
Ell
Ell
Ell
Ell
Ell
2.116
20.613 10.480
0.635
0.268
2.369
0.324
0.152
0.720
0.350
2.057
0.321
0.152
65.600
29.200
2.247
19.667
6.611
1.046
0.436
2.399
0.290
0.106
1.150
0.530
2.170
0.304
0.114
[-100,100] 150.800
24.920
6.051
36.507
9.478
3.776
0.558
6.767
0.999
0.228
4.052
0.689
5.881 11.107
0.234
0.401
3.678
1.097
3.353 11.059
0.396
1000
Ell
] 000
[-100,100] 13R.923
Ell
36.846
1000
[-100,100] 119.346
1:
3.437
0.958
5.877
2.901
0.437
5.293
37.578
Ell
8.921
3.332
0.577
---
No test problems were affected by this
restriction when the GLO algorithm was used.
converge in
175
Some problems, however, failed to
iterations with the BHPM approach.
The means and standard
deviations which are artificially low due to this truncation are indicated in
NOTE~'
the table by the symbol
ffl.
In the case of a
matrix, the maximum number of iterations allowed was
100.
20x20
Only one problem failed to converge in
100
1.129
Ell
0.329
Ell
In order to contain expenses for computer time, the maximum number of iterations
175.
6.638
Ell
24.833
0.951
Ell
44.349 13.248
--
allowed for anyone problem was
3.588
Ell
20.308
I
I
Ell
37.550 14.318
3.770
Ell
I
I
20.360
--~--
~;OTlc
BHPM
Std. DC'via liOll
BHPM-t
GLO
53.370
Ell
~J
BHPM'f
GLO GLO
[-100,100]
ying RIIS
Sparse [-1000,1000] [-100,100] 131.433
I
C~-!~."l~ i.~.!!~ ~~<:J!l!:F:tlL__.,
Mean in Seconds
1000
Ell
II
20;;!+0
GLO
Std. Deviation
43.080
xed RHS
;~ Sparse
o!
~
I3HPM~
CPU TIME (ITERATIONS ONLY)
Mean in Seconds
[-100,100]
xed HIlS
'l.. Sparse
I
•.~')
BHPM
Std. Dl'viutlon
ying RIIS
Sparse [-1000,1000] [-100,100]
xed RHS
Sparse
i
..:)
------ -
~--_.-
------
---,-~,
120x20 I
.
25
~riplion
i---rI
_ _ _ _ _ _ _. J
Mean
'T"sL
SLr.e
__._--
ITERATION COUNT
iterations.
5.775
1.025
Ell
3.100
Ell
0.559
5.546
0.219
3.540
1.226
Ell
Ell
0.712
4.972
0.330
!
I
1.124 10.220
i
-
-176.
Test Results
The Gradient Linear Optimization Method was employed for the solution
of a set of test problems of various sizes, and the results were oompared
with the results obtained when the sarre problems were solved by the BHPM
approach.
These tests were run on an IBM 370/165 using a series of programs
designed mainly by Jarres O. Kitchen of the University of North Carolina
GOmputation Center [2].
Constraint coefficients were randomly generated
in the interval [-100, 100].
The results of these tests are shown in Table
VII.
Examination of the test results in Table VII reveals that for every
group of test problems, on the average, the GID algorithm required fewer
iterations than did the BHPM algorithm.
Similarly, the rrean CPU tines
(for iterations only as well as for the entire problem) were consistently
lower with GLO.
In fact, on alnost every individual problem, iteration oomt
and CPU tine were lower for GLO than BHPM.
in Table VII, tests were also run on
Again,
GIJ)
SJITVC
In addition to the results shown
larger problems,
35 x 55 or larger.
was better, although time and rroneyconstrained us from rmning a
sufficient number for statistical significance.
The results of test runs with the simplex algorithm are not included due
to a narked diSParity in coding techinques between the available simplex programs and those used for the BHPM and GLO algorithms.
The latter two programs
were coded similarly, which pennitted a comparison of CPU ti..nes.
Ti..nes should
only be used for relative oomparisons, in the sense that they were made on
the sarre machine with the
Barre
coding techniques.
to use the rrost efficient coding techniques.
That is, no effort was made
With resPect to iteration comt,
the rrean number of iterations required for solution by the GLO method ranged
-18-
from
1. 015
ID
to
1. 842
ID.
No p:rrticular relationship between the sparseness
of the coefficient matrix and the efficiency of the GLO rrethod was observed.
7.
Degeneracy and Cycling
Although the authors do not have a proof that degeneracy and cycling
can be avoided, in the hundreds of test problems which ~re run, no cycling was
observed.
Included in the testing
cycling with other algorithms
~re
several problems which do cause
(e.g., Example 1 of this paPer), and each of
these converged with the GLO rrethod.
•
REFERENCES
1.
Hadley, G.:
2.
Kitchen, J .0., Shachtman, R.H.: "A Linear Prograrrrning System". This
is a system of seven programs designed to corrpare the BHPM rrethod,
variants of GIJJ rrethods and mixtures. It accepts the corresponding
matrices for existing examples or generates randan problems for
testing.
3.
Kuhn, H. W., Quandt, R. E.: "An Experirrenta1 Study of the Sirrp1ex
~thod", Proceedings of Syrrposia in Applied Math, 15 (1963) 107-124.
4.
Saksena, C. P., Cole, A. J.: "The Bounding Hyperplane Method of
Linear Prograrrming", Operations Research, 19 (1971) 1-18.
5.
Zionts, S.: "The Criss-cross ~thod for SOlving Linear Progranming
Problems" , Management SCience, 15 (1969) 426-445.
Linear PrograIml;ing,
Addison-wes1ey, 1962.