Introduction to Optimization
(iii) Lagrange Multipliers and
Kuhn-tucker Conditions
Water Resources Systems Planning and Management: M2L3
D Nagesh Kumar, IISc
Objectives

To study the optimization with multiple decision variables
and equality constraint : Lagrange Multipliers.

To study the optimization with multiple decision variables
and inequality constraint : Kuhn-Tucker (KT) conditions
2
Water Resources Systems Planning and Management: M2L3
D Nagesh Kumar, IISc
Constrained optimization with equality
constraints
A function of multiple variables, f(x), is to be optimized subject to one or
more equality constraints of many variables. These equality constraints, gj(x),
may or may not be linear. The problem statement is as follows:
Maximize (or minimize) f(X), subject to gj(X) = 0, j = 1, 2, … , m
where
3
 x1 
x 
 
X   2
 
 xn 
Water Resources Systems Planning and Management: M2L3
(1)
D Nagesh Kumar, IISc
Constrained optimization…
 With the condition that
m  n ; or else if m > n then the problem
becomes an over defined one and there will be no solution. Of the
many available methods, the method of constrained variation and the
method of using Lagrange multipliers are discussed.
4
Water Resources Systems Planning and Management: M2L3
D Nagesh Kumar, IISc
Solution by method of Lagrange
multipliers

Continuing with the same specific case of the optimization problem with
n = 2 and m = 1 we define a quantity λ, called the Lagrange multiplier as


f / x2
g / x2
(2)
(x1* , x 2* )
Using this in the constrained variation of f [ given in the previous lecture]
 f g / x1 f 
df  

dx1  0

 x1 g / x2 x2  (x1 *, x 2 *)
And (2) written as
5
 f
g 


0



x

x
1  (x * , x * )
 1
1
2
(3)
 f
g 


0


x2  (x * , x * )
 x2
1
2
(4)
Water Resources Systems Planning and Management: M2L3
D Nagesh Kumar, IISc
Solution by method of Lagrange multipliers…
 Also, the constraint equation has to be satisfied at the extreme point
g ( x1 , x2 ) ( x * , x * )  0
1
(5)
2
 Hence equations (2) to (5) represent the necessary conditions for the point
[x1*, x2*] to be an extreme point.
 λ could be expressed in terms of g / x1 as well g / x1 has to be non-
zero.
 These necessary conditions require that at least one of the partial
derivatives of g(x1 , x2) be non-zero at an extreme point.
6
Water Resources Systems Planning and Management: M2L3
D Nagesh Kumar, IISc
Solution by method of Lagrange multipliers…
 The conditions given by equations (2) to (5) can also be generated by
constructing a functions L, known as the Lagrangian function, as
L( x1 , x2 ,  )  f ( x1 , x2 )   g ( x1 , x2 )
(6)
 Alternatively, treating L as a function of x1,x2 and , the necessary
conditions for its extremum are given by
L
f
g
( x1 , x2 ,  ) 
( x1 , x2 )  
( x1 , x2 )  0
x1
x1
x1
L
f
g
( x1 , x2 ,  ) 
( x1 , x2 )  
( x1 , x2 )  0
x2
x2
x2
L
( x1 , x2 ,  )  g ( x1 , x2 )  0

7
Water Resources Systems Planning and Management: M2L3
(7)
D Nagesh Kumar, IISc
Necessary conditions for a general problem
 For a general problem with n variables and m equality constraints the
problem is defined as shown earlier
 Maximize (or minimize) f(X), subject to gj(X) = 0, j = 1, 2, … , m
where
 x1 
x 
 
X   2
 
 xn 
 In this case the Lagrange function, L, will have one Lagrange multiplier
j
for each constraint as
L( x1 , x2 ,..., xn, 1 , 2 ,..., m )  f ( X)  1 g1 ( X)  2 g 2 ( X)  ...  m g m ( X)
8
Water Resources Systems Planning and Management: M2L3
(8)
D Nagesh Kumar, IISc
Necessary conditions for a general problem…
 L is now a function of n + m unknowns,
x1 , x2 ,..., xn , 1 , 2 ,..., m , and the
necessary conditions for the problem defined above are given by
m
g j
L f

( X)    j
( X)  0,
xi xi

x
j 1
i
L
 g j ( X)  0,
 j
i  1, 2,..., n;
j  1, 2,..., m
(9)
j  1, 2,..., m
 which represent n + m equations in terms of the n + m unknowns, xi and j.
The solution to this set of equations gives us
 x1* 
 *
x 
X 2 
 
 xn* 
9
and
Water Resources Systems Planning and Management: M2L3
 1* 
 *
 
*   2 
 
m* 
(10)
D Nagesh Kumar, IISc
Sufficient conditions for a general problem
 A sufficient condition for f(X) to have a relative minimum at X* is that each
root of the polynomial in Є, defined by the following determinant equation
be positive.
L11  
L12
L1n
g11
g 21
g m1
L21
L22  
L2 n
g12
g 22
gm2
Ln1
Ln 2
Lnn   g1n
g2n
g mn
0
10
g11
g12
g1n
g 21
g 22
g2n
g m1
gm2
g mn
Water Resources Systems Planning and Management: M2L3
0
0
0
0
(11)
D Nagesh Kumar, IISc
Sufficient conditions for a general problem…
where
2L
Lij 
( X* ,  * ),
xi x j
g pq 
g p
xq
( X* ),
for i  1,2,..., n and
j  1,2,..., m
(12)
where p  1,2,..., m and q  1,2,..., n
 Similarly, a sufficient condition for f(X) to have a relative maximum at X*
is that each root of the polynomial in Є, defined by equation (11) be
negative.
 If equation (11), on solving yields roots, some of which are positive and
others negative, then the point X* is neither a maximum nor a minimum.
11
Water Resources Systems Planning and Management: M2L3
D Nagesh Kumar, IISc
Example
2
2
Minimize f ( X)  3x1  6 x1 x2  5 x2  7 x1  5 x,2
Subject to
x1  x2  5
Solution
g1 (X)  x1  x2  5  0
L( x1 , x2 ,..., xn, 1 , 2 ,..., m )  f ( X)  1 g1 ( X)  2 g 2 ( X)  ...  m g m ( X)
with n = 2 and m = 1
L = 3x12  6 x1 x2  5 x22  7 x1  5 x2  1 ( x1  x2  5)
L
 6 x1  6 x2  7  1  0
x1
1
 x1  x2  (7  1 )
6
1
 5  (7  1 )
6
12
Water Resources Systems Planning and Management: M2L3
or
1  23
D Nagesh Kumar, IISc
Example…
L
 6 x1  10 x2  5  1  0
x2
1
(5  1 )
2
1
 3( x1  x2 )  2 x2  (5  1 )
2
 3 x1  5 x2 
1
x2 
2
Hence,
x1 
11
2
11 1 
X*   ,  ; λ*   23
2 2
L12
g11 
 L11  


L
L


g
22
21   0
 21
 g
g12
0 
 11
13
Water Resources Systems Planning and Management: M2L3
D Nagesh Kumar, IISc
Example…
 2L
L11  2
 6
x1 ( X*,λ*)
 2L
L12  L21 
x1x2
g11 
g1
x1
or
( X*,λ* )
1
( X*,λ* )
g12  g 21 
The determinant becomes
 6
 2L
L22  2
 10
x2 ( X*,λ*)
g1
x2
1
( X*,λ* )
6
1
 6 



6

10


1

0
 1
1
0 

(6)[1]  (6)[1]  1[6  10 ]  0
 2
Since  is negative, X*, λ * correspond to a maximum
14
Water Resources Systems Planning and Management: M2L3
D Nagesh Kumar, IISc
Kuhn – Tucker Conditions

KT condition: Both necessary and sufficient if the objective function is
concave and each constraint is linear or each constraint function is
concave, i.e., the problems belongs to a class called the convex
programming problems.
15
Water Resources Systems Planning and Management: M2L3
D Nagesh Kumar, IISc
Kuhn-Tucker Conditions: Optimization
Model
Consider the following optimization problem
Minimize f(X)
subject to
gj(X) ≤ 0
for j=1,2,…,p
where the decision variable vector
X=[x1,x2,…,xn]
16
Water Resources Systems Planning and Management: M2L3
D Nagesh Kumar, IISc
Kuhn-Tucker Conditions
Kuhn-Tucker conditions for X* = [x1* , x2* , . . . xn*] to be a local minimum are
m
f
g
  j
0
xi j 1 xi
17
i  1, 2,..., n
j g j  0
j  1, 2,..., m
gj  0
j  1, 2,..., m
j  0
j  1, 2,..., m
Water Resources Systems Planning and Management: M2L3
D Nagesh Kumar, IISc
Kuhn Tucker Conditions …

In case of minimization problems, if the constraints are of
the form gj(X) ≥ 0, then λj have to be non-positive

If the problem is one of maximization with the constraints
in the form gj(X) ≥ 0, then λj have to be nonnegative.
18
Water Resources Systems Planning and Management: M2L3
D Nagesh Kumar, IISc
Example 1
Minimize
f  x  2 x  3x
2
1
2
2
2
3
subject to
g1  x1  x2  2 x3  12
g 2  x1  2 x2  3x3  8
19
Water Resources Systems Planning and Management: M2L3
D Nagesh Kumar, IISc
Example 1…
Kuhn – Tucker Conditions
g
g
f
 1 1  2 2  0
xi
xi
xi
j g j  0
gj  0
j  0
20
2 x1  1  2  0
(14)
4 x2  1  22  0
(15)
6 x3  21  32  0
(16)
1 ( x1  x2  2 x3  12)  0
2 ( x1  2 x2  3x3  8)  0
(17)
(18)
x1  x2  2 x3  12  0
(19)
x1  2 x2  3x3  8  0
(20)
1  0
2  0
(21)
Water Resources Systems Planning and Management: M2L3
(22)
D Nagesh Kumar, IISc
Example 1…
From (17) either 1 = 0 or x1  x2  2 x3  12  0 ,
Case 1: 1 = 0
 From (14), (15) and (16) we have x1 = x2 =
 Using these in (18) we get
2 / 2 and x3 = 2 / 2
22  82  0,  2  0 or  8
 From (22), 2  0 , therefore,
2 =0,
 Therefore, X* = [ 0, 0, 0 ]
This solution set satisfies all of (18) to (21)
21
Water Resources Systems Planning and Management: M2L3
D Nagesh Kumar, IISc
Example 1…
Case 2:
x1  x2  2 x3  12  0
 Using (14), (15) and (16), we have
or 171  122  144
1  2 1  22 21  32


 12  0
2
4
3
 But conditions (21) and (22) give us 1  0 and 2  0
simultaneously, which cannot be possible with 171  122  144
Hence the solution set for this optimization problem is
X* = [ 0 0 0 ]
22
Water Resources Systems Planning and Management: M2L3
D Nagesh Kumar, IISc
Example 2
Minimize
f  x12  x22  60 x1
subject to
g1  x1  80  0
g 2  x1  x2  120  0
23
Water Resources Systems Planning and Management: M2L3
D Nagesh Kumar, IISc
Example 2…
Kuhn – Tucker Conditions
g
g
f
 1 1  2 2  0
xi
xi
xi
j g j  0
gj  0
j  0
24
2 x1  60  1  2  0
(23)
2 x2  2  0
(24)
1 ( x1  80)  0
2 ( x1  x2  120)  0
(25)
x1  80  0
(27)
x1  x2  120  0
(28)
1  0
2  0
(29)
Water Resources Systems Planning and Management: M2L3
(26)
(30)
D Nagesh Kumar, IISc
Example 2…
From (25) either 1 = 0 or ( x1  80)  0 ,
Case 1
 From (23) and (24) we have
 Using these in (26) we get
x1  
2
2
 30 and x2  
2
2
2  2 150  0
 2  0 or  150
 Considering
2  0 , X* = [ 30, 0]. But this solution set violates (27)
and (28)
 For
25
2  150 , X* = [ 45, 75]. But this solution set violates (27)
Water Resources Systems Planning and Management: M2L3
D Nagesh Kumar, IISc
Example 2…
Case 2: ( x1  80)  0
 Using x1  80 in (23) and (24), we have
2  2 x2
1  2 x2  220
(31)
 Substitute (31) in (26), we have 2 x2  x2  40   0
 For this to be true, either
26
x2  0 or x2  40  0
Water Resources Systems Planning and Management: M2L3
D Nagesh Kumar, IISc
Example 2…
 For x2  0 , 1  220
 This solution set violates (27) and (28)
 For x2  40  0 ,
1  140 and 2  80
 This solution set is satisfying all equations from (27) to (31) and hence
the desired
 Thus, the solution set for this optimization problem is
X* = [ 80, 40 ]
27
Water Resources Systems Planning and Management: M2L3
D Nagesh Kumar, IISc
BIBLIOGRAPHY / FURTHER READING
1. Rao S.S., Engineering Optimization – Theory and Practice, Fourth
Edition, John Wiley and Sons, 2009.
2. Ravindran A., D.T. Phillips and J.J. Solberg, Operations Research –
Principles and Practice, John Wiley & Sons, New York, 2001.
3. Taha H.A., Operations Research – An Introduction, 8th edition, Pearson
Education India, 2008.
4. Vedula S., and P.P. Mujumdar, Water Resources Systems: Modelling
Techniques and Analysis, Tata McGraw Hill, New Delhi, 2005.
28
Water Resources Systems Planning and Management: M2L3
D Nagesh Kumar, IISc
Thank You
Water Resources Systems Planning and Management: M2L3
D Nagesh Kumar, IISc