CHAPTER 5
NON-LINIER OPTIMIZATION
5.1. Introduction
Problem estimisasi secara umum
MaX (Min) f (X)
s.t.
Xy
F disebut objektif function.
X disebut himpunan feasible.
Himpunan feasible biasanya digambarkan dengan persamaan
atau pertidaksamaan yang disebut kendala.
Contoh linier programming problem :
MaX
s.t.
ctX
AX = b
X≥0
X = {XRn| AX = b, X ≥0}
Jika tidak ada kendala maka X = Rn,
dan problem ini disebut optimisasi tanpa kendala
5.2. Global dan Local Optima
5.2.1 Definisi
f = D  R dimana D  Rn. Fungsi f dikatakan memiliki global
(absolute) maximum pada X*  D jhj f(X) ≤ f(X*)  X  D.
Titik X* disebut global maximizer (or maximizer) dari f dan
f(X*) disebut global maximum dari f dan sebaliknya.
Global minimum f(X) ≥ f (X*)  X  D.
Fungsi f dikatakan memiliki local (relative) maximum pada X*
 D jhj terdapat neighborhood N (X*) seperti :
f(X) ≤ f(X*)  X  N (X*)  D.
Titik X* disebut local maximizer dari f dan f(X*) disebut local
maximum dari f.
Dan sebaliknya adalah:
Fungsi local minimum X*  D jhj f(X) ≥ f(X*)  X  N(X*)  D
5.2.2 Contoh Local dan Global Optimum
f(X)
0
X1
X2
X3
X4
X5
X6
D
X1, X3, X5 adalah local maximum, X5 global max
X2, X4, X6 adalah local minimum, X2 global min
5.2.3 Remark
(1) Global max digunakan secara relatif to domain D dari f
yaitu global maximizer berarti global maximizer to
domain
(2) Istilah optimum digunakan baik maximum maupun
minimum
(3) Suatu global maximizer (global minimizer) adalah local
maximizer (local minimizer)
(4) Local maximum dapat lebih kecil dibanding local
minimun
(5) Suatu fungsi tidak memiliki baik maximizer ataupun
minimizer dalam domainnya. f(X) = X, tidak memiliki
max/min dalam interval (0,1)
Catatan : Interval ini tidak tertutup meskipun interval
tertutup tetapi tidak memiliki batas (bounded)
5.2.4 Theorema
Suatu fungsi continuous real-valued didefinisikan atas
suatu compact subset S dari Rn memiliki maximizer &
minimizer dalam S.
5.3 Necessary (or First-Order) Condition
untuk Local Optimum
5.3.1 Theorema
Andaikan f : D  R, dimana D ≤ Rn, dan andaikan turunan
partial dari f  X(S) pada suatu interior point X* dari D X*
adalah local optimizer dari f, maka f’(X) = 0. Proof :
5.3.2 Remark
(1) Kasus satu variabel dari theorema 5.3.1
f(X)
f’(X*)=0
f(X)
f’(X*)=0
0
X*
X
0
X*
X
(2)Syarat bahwa turunan exist pada local optimizer adalah
necessary dalam teorema 5.3.1
contoh :
f(X) = X2/3
f’(X) = 2/3X-1/3
Fungsi f memiliki local minimum pada X*=0 but f’(0)≠0.
Contoh di atas tidak differentiable pada X*=0 (ini berarti
bukan sufficient).
f(X) = X2/3
f(X)
Gambar 5.4
X
X*
(3) Syarat suatu X* adalah interior point dari domainnya
adalah necessary. Contoh andaikan gambar 5.5 domain D
adalah interval tertutup [0,1].
f’(X*)0
Gambar 5.5
D
X*
0
1
X
Memenuhi local minimum pada X*=0 tetapi f’(X*)0.
Note: X* tidak pada interior point dari D.
(4) Kebalikan dari teorema 5.3.1 tidak benar
Contoh :
f(X) = X3
f’(X) = 3X2
f’(0) = 0 tetapi f tidak memiliki local optimum pada X*=0.
f(X)
f’(X)=X3
X
Gambar 5.6
Contoh tiga dimensi
f(X1,X2) = X12 – X22
Gambar 5.7 gradien f adalah
f’(X1,X2) =
2X1
-2X2
Pada X*=[0,0]t, f(X*)=0 tetapi untuk titik dekat X* pada sb
X1–axis, f(X1,X2)>0 dan titik dekat X* pada sb X2 f(X1,X2)<0.
Consekuensi X*=0 mendekati local maximizer dan local
minimizer sehingga titik tersebut disebut saddle point.
X3
Gambar 5.7. Saddle point
X2
X1
5.3.3 Definisi
Titik X* dimana f’(X*)=0 disebut stationary point f. stationary
point dan bukan titik optimum disebut saddle point dari f.
Dalam pandang theorema 5.3.1 titik optimizer f pada domain D
ditemukan.
(a) Stationary point (figure 5.7.1(a))
(b) The boundary point (figure 5.7.1(b))
(c) f’(X) tidak exist (figure 5.7.1(c))
X*
D
X*
X*
D
D
c. f’(X*)=n tidak diperoleh
b. f’(X*)  0
Gambar 5.7.1
Kemudian sufficient condition untuk suatu stationary point
menjadi local optimizer ketika f setidaknya dapat di
diferensialkan 2 kali secara continuous.
a. f’(X*) = 0
5.4 Sufficient (Second-Order) Condition for
Local Optima : The One-Variable Case
5.4.1 Theorema
Andaikan D ≤ R . f : D  R memiliki continuous second-order
derivative atas open interval I yang berisi X*. Andaikan f’(X*)=0.
(i) Jika f”(X*) > 0, X* adalah local minimizer dari f
(ii) Jika f”(X*) < 0, X* adalah local maximizer dari f
Proof :
5.4.2 Contoh
Let f : R  R
f(X) = 2X3 – 3X2 – 12X + 1
f’(X) = 6X2 – 6X – 12
Setting f’(X) = 0 diperoleh X* = 2, X** = -1
f”(X) = 12 X – 6
f”(2) = 12(2) – 6 = 18
f”(-1) = 12(-1) – 6 = -18
Ini berarti bahwa X* = 2 adalah local maximizer dari f
X** = -1 adalah local minimizer dari f
5.4.3 Remark
Ketika f”(X*)=0, tidak ada kesimpulan yang dapat dibuat.
Kita
gunakan
theorema
berikutnya
untuk
menggeneralisasikan theorema 5.4.1
5.4.4 Theorema
Let D C R dan f : D  R memiliki continuous nth-order derivative
pada N(X*) C D.
Suppose f’(X*) = f”(X*) = … = f(n-1)(X*) = 0 dan f(n)  0
(i) Jika n bilangan genap dan f(n)(X*) > 0, maka X* adalah local
minimum dari f
(ii) Jika n bilangan genap dan f(n)(X*) < 0 maka X* adalah local
maXimum dari f
(iii) Jika n bilangan ganjil, maka X* adalah saddle point dari f
Proof :
5.4.5 C ontoh
Necessary condition :
Stationary Point
:
Sufficient condition :
Kasus ini n=4 dan
minimizer pada f.
f(X) = (X – 5)4
f’(X) = 4(X-5)3 = 0
X* = 5
f”(X) = 12(X-5)2,
f”(5) = 0
f(3)(X) = 24(X–5),
f3(5) = 0
f4(X) = 24,
F4(5) = 24
f4(5) > 0, sehingga X*=5 adalah local
5.5 Sufficient (or Second-Order) Condition for
Local Optima : The n-Variable Case
5.5.1 Theorema
Let D C Rn + f : D  R mempunyai continous second order
partial derivative pada N(X*) C D dan f’(X*) = 0.
(i) Jika f”(X*) adalah positif definit, X* adalah local
minimizer dari f
(ii) Jika f”(X*) adalah negatif definit, X* adalah local
maximizer dari f
(iii) Jika f”(X*) indefinite, X* adalah saddle-point dari f
Proof :
5.5.2 Remark
Ketika Hessian Matriks f”(X*) adalah positif atau negatif
semi definit, tidak ada kesimpulan yang dapat ditarik.
5.5.3 Contoh
f(X) = -X12 – 3X22 – 2X32 + 2X1 – 12X2 + 8X3 – 5
Gradien dari f :
f’(X) =
-2X1 + 2
-6X2 – 12
-4X3 + 8
Setting f’(X) = 0, maka titik stationer
X* = [2
-2
Hessian matriks dari f pada X* adalah
-2 0 0
f”(X* ) = [f’’ij(X*)] = 0 -6 0
0 0 -4
Memiliki leading principle minor adalah
|[-2]| = -2 < 0
-2
0
0
-6
= 12 > 0
2]t
-2 0 0
0 -6 0
0 0 -4
= -48 < 0
Dari theorema 1.12.15 bahwa f”(X*) adalah negatif definite.
Alternatif, catatan bahwa f”(X*) adalah matriks diagonal, maka
eigenvalue adalah elemen diagonal (theorema 1.11.6). Sejak
eigenvalue negatif maka f”(X*) adalah negatif definit (by
Theorema 1.12.6). Sehingga X* adalah local maximizer dari f
(by theorema 5.5.1).
5.5.4 Contoh
f(X) = 4X13 + X1X2 – 3/2X12 + ½X22 + 10
Gradient dari f adalah
f’(X) =
12X12 + X2 – 3X1
X1+ X2
Setting f’(X) = 0 diperoleh titik stationer :
X* =
0
0
, X** =
1/3
-1/3
Hessian matriks dari f adalah
f”(X) = 24X1-3 1
1
1
Pertimbangankan titik stationer X* maka Hessian matrix dari f
pada X* adalah
f”(X*) =
24(0)-3
1
1
1
-3
1
=
1
1
Matriks Real symmetric dimana eigenvalue adalah
1 = 1 + 5 ,
2 = 1 – 5
Apabila 1 > 0 dan 2 < 0 sesuai teorema 1.12.6 bahwa f”(X*)
adalah indefinite. Konsekuensinya, X* adalah saddle-point dari f
by theorema 5.5.1
Hessian matrix pada X** adalah
f”(X**) =
24(1/3)-3
1
1
1
=
5
1
1
1
Matriks Real symmetric dimana Leading Principal Minors (LPM)
adalah
5
1
|[5]| = 5 > 0 ,
1
1 =4>0
Theorema f”(X**) adalah positif definite. Alternatif, calculate
eigenvalue dari f”(X**);
1 = 3 + 5
,
2 = 3 – 5
1>0 dan 2>0, by theorem 1.12.6 f”(X**) adalah positif definite.
sehingga X** sebagai local minimizer dari f by theorem 5.5.1
5.5.5 Contoh
Let f : R2  R didefinisikan dengan
f(X) = X12 + X24
Gradient dari f adalah
2X1
3
4X
2
f’(X) =
Setting f’(X) = 0 , diperoleh stationary point
X* = [ 0
0]t
Hessian matriks dari f adalah
f”(X) =
2
0
0
12X22
Pada X*, Hessian matriks dari f adalah
f”(X*) =
2
0
0
0
Eigenvalue adalah 1=2 dan 2=0 sesuai theorema 1.12.6 that
f”(X*) adalah positive semidefinite tetapi tidak positif definit
sehingga tidak dapat diputuskan.
5.5.6 Contoh
Let f : R2  R didefinisikan dengan
f(X) = X12 – X24
Gradient pada f adalah
2X1
f’(X) = - 4X22
Setting f’(X) = 0 , diperoleh titik stasioner
X* = [ 0
0]t
Hessian matriks dari f adalah
f”(X) =
2
0
0
-12X22
Pada X*, Hessian matriks dari f adalah
f”(X*) =
2
0
0
0
Theorem 5.5.1 tidak dapat digunakan tetapi f(X*)=0 dan
untuk titik X mendekati X* pada X1–axis, f(X)=(X1)2 > 0
sedang untuk titik X mendekati X* pada X2–axis, (f(X)=(X2)4 < 0. Sehingga X* adalah saddle-point dari f (lihat
Danao, 244-247).
5.5.7 Contoh Profit Maximization
q = output, R(q), C(q), (q). R dan C continuous second-order
derivatives.
(q) = R(q) – C(q)
’(q*) = R’(q*) – C’(q*) = 0
R’(q*) = C’(q*)  optimal profit
R’(q*) < C’(q*)  q  
R’(q*) > C’(q*)  q  
5.5.8 Contoh
The Method least Squares.
Garis regresi :
Y = 0 + 1X
Untuk setiap i
i = Yi – (0 + 1Xi)
Untuk estimasi 0 dan 1 n
n
(i)2 =  (Yi – 0 – 1Xi)2
i=1
i=1
adalah minimum.
n
 disederhanakan sebagai .
i=1
f(0, 1) = (Yi – 0 – 1Xi)2
f’(0, 1) = 0 untuk minimum
2(Yi – 0 – 1Xi)( -1) = 0
2(Yi – 0 – 1Xi)(-Xi) = 0
Maka
n 0 + (Xi)1 = Y1
(Xi)0 + (X12)1 = YiXi
Normal equation dalam bentuk matrix
n
Xi
B0 = yi
yiXi
Xi
Xi2
B1
Asumsi coefficient matriks adalah non-singular, menggunakan
Cramer’s Rule, titik stasioner :
1* =
nXiYi – (Xi) (Yi)
n(Xi)2 – (Xi)2
0* = 1 Yi –
n
1 (X )  *
i
1
n
to verify * is a global minimizer dari f, dengan melihat Hessian
matrix dari f :
2
f”(0, 1) = 2X
i
2Xi
2Xi2 =
2n
2Xi
2Xi
2Xi2
LPM adalah :
|[2n]| = 2n>0 ;
2n
2Xi
2Xi 2(Xi)2
= 4n(Xi)2 – 4(Xi)2
= 4[n(Xi)2 – (Xi)2]
= 4[(n-1)(Xi)2 – 2 Xi Xj] >0
Dari persamaan (5-2) Hessian matriks f”(0,1) adalah positive
definite and so the stationary point * minimizer f.
5.6 Optima of Concave and Convex Functions
5.6.1 Definisi
Let X,Y  Rn, titik Z = X + (1–)Y
Dimana 0 ≤  ≤ 1 disebut convex combination dari X dan Y
5.6.2 Remark
Geometrically, the set of all convex combinations of X and Y is
the line segment joining X and Y
5.6.3 Definisi
A subset C of Rn adalah convex jhj line segment berhubungan
dari titik dalam C berada dalam himpunan C. In symbols, C
adalah convex jhj
X,Y  C, 0 ≤  ≤ 1
(X + (1- )Y)  C
5.6.4 Remarks
1) Himpunan kosong (O) dan himpunan hanya satu elemen
adalah convex set
2) Rn adalah convex set
y=1/X
Convex
Set
Strictly Convex
Not Strictly Convex
Not
ConveX
Set
3) The following sets are convex :
(a) The closed half-spaces :
(b) The hyperplane :
(c) The Non-negative orthant :
(d) The positive orthant :
H+(P,) = {X  Rn| PtX  }
H-(P,) = {X  Rn| PtX ≤ }
H(P,) = {X  Rn | PtX = }
Rn+ = {XRn | X  0}
Rn++ = {XRn|X>0}
5.6.5 Theorema
Interseksi dari convex set adalah convex.
Proof :
5.6.6 Definisi
A subset C of Rn adalah strictly convex jhj
X,Y  C, X  Y, 0 <  < 1
[X + (1-)Y]  int (C)
5.6.7 Remarks
An open convex set is strictly convex. A closed disk is strictly
convex while a closed triangle is not. Intuitively, a closed strictly
convex set does not have a flat portion on its boundary.
5.6.8 Definisi
A function f : C  R defined on a convex subset C of Rn is said to
be concave on C if and only if
X,Y  C, 0 ≤  ≤ 1,
f(X+(1- )Y) ≥ f(X)+(1- )f(Y)
5.6.10 Definisi
f : C  R defined on a convex subset C of Rn is said to be strictly
concave on C if and only if
X,Y  C, X ≠ Y, 0 <  < 1,
f(X+(1-)Y) > f(X)+(1- )f(Y)
5.6.11 Definisi
f : C  R defined on a convex subset C of Rn is said to be strictly
convex on C if and only if
X,Y  C, X ≠ y, 0 <  < 1,
f(X+(1- )Y) < f(X)+(1-)f(Y)
5.6.12 Theorema
A concave (convex) function is continuous in the interior of its
domain
f((X1)+(1- )X2)
f(X2)
f(X1)
Concave function
f(X1)+(1- )f(X2)
X1
(X1)+(1- )X2
X2
f(X1)+(1-)f(X2)
f(X1)
f(X2)
f(X1+(1- )X2
X1
X1+(1-)X2
X2
Convex function
Y Y=ln(X)
X
Strictly concave
function
Concave function
Y
Y=1/X
Strictly convex
function
X
Convex function
Neither concave nor convex functions
Figure 5.11
5.6.13 Theorema
f : C  R dimana C adalah convex subset of Rn
(i) if f is concave on C then UCf() = {X  C|f(X) ≥ }
is convex for every   R
(ii) if f is convex on C then LCf() = {X  C|f(X) ≤ }
is convex for every   R
5.6.14 Definisi
f : D  R adalah function defined on a subset D of Rn.
- UCf() = {X  D| f(X) ≥ ,   R}
disebut upper contour set of f
- LCf() = {X  D| f(X) ≤ ,   R}
disebut lower contour set of f
- Cf() = {X  D| f(X) = ,   R}
disebut contour (or level) set of f
5.6.15 Remark
5.6.16 Theorem
f : C  R adalah continuously differentiable on convex
subset of Rn
(i) f adalah concave on C if and only if
X,Y  C,
f(Y) - f(X) ≤ [f’(X)]t (Y-X)
(ii) f adalah convex on C if only if
X,Y  C,
f(Y) - f(X) ≥ [f’(X)]t (Y-X)
5.6.17 Theorem
f : I  R adalah continuously differentiable on open
interval I
(i) f adalah concave on I if only if
X,Y  I,
f(Y) - f(X) ≤ f’(X) (Y-X)
(ii) f adalah convex on I if only if
X,Y  I,
f(Y) - f(X) ≥ f’(X) (Y-X)
5.6.19 Theorem
f : C  R adalah continuously differentiable on open convex
subset C of Rn
(i) f adalah strictly concave on C if only if
X,Y  C, X ≠ Y
f(Y) - f(X) < [f’(X)]t (Y-X)
(ii) f adalah strictly convex on C if only if
X,Y  C, X ≠ Y
f(Y) - f(X) > [f’(X)]t (Y-X)
5.6.20 Theorem
f : C  R adalah twice continuously differentiable on open
concex subset C of Rn then
(i) f is concave on C jhj Hessian matrix f”(X) is negatif
semidefinite on C
(ii) f is convex on C jhj Hessian matriX f”(X) is positive
semidefinite on C
5.6.21 Theorem
f : I  R be twice continuously differentiable on an open
interval I
X{Ci | if the hessian matriks f”(X) adalah negatif
(i) f is concave on I jhj f”(X) ≤ 0 on I
(ii) f is conveX on I jhj f”(X) ≥ 0 on I
5.6.22 Theorem
f have continuous second-order partial derivative on an open
convex subset C of Rn
(i) If Hessian matriks f”(X) is negative definite for every X  C,
then f is strictly concave on C
(ii) If Hessian matriks f”(X) is positive definite for every X  C,
then f is strictly convex on C
Proof : Polak
5.6.23 Remark
f : R  R defined f (X) = X4.
f is continuous second-order derivative on R.
f is strictly convex on R. But f”(0) = 0 which is not positive
definite.
5.6.24 Theorema
f : I  R adalah twice continuously differentiable
on an open interval I
(i) If f”(X) < 0 for every X  I then f is strictly concave on I
(ii) If f”(X) > 0 for every X  I then f is strictly convex on I
Sum and compositions of concave and convex function
5.6.25 Theorem
f : C  R dan g : C  R where C is a convex subset of Rn
(i) If f dan g adalah concave on C then
(a) f + g is concave on C
(b) f is concave on C for each  > 0
(c) f is convex on C for each  < 0
(ii) If f dan g adalah convex on C then
(a) f + g is convex on C
(b) f is convex on C for each  > 0
(c) f is concave on C for each  < 0
5.6.26 Theorem
f : C  R dan g : C  R; dimana C is a convex subset on Rn
(i) If f dan g adalah strictly concave on C then
(a) f + g is strictly concave on C
(b) f is strictly concave for each  > 0
(c) f is strictly convex for each  < 0
(ii) If f dan g adalah strictly convex on C then
(a) f + g is strictly convex on C
(b) f is strictly convex on C for each  > 0
(c) f is strictly concave on C for each  < 0
5.6.27 Theorem
Let : f : C  R be defined on a convex subset C of Rn such that f(C)
is convex. Let g : f(C)  R be defined on f(C)
(i) If f is concave on C and g is concave and increasing of f (C),
then the composition of f and g is concave on C.
(ii) If f is convex on C and g is convex and increasing on f (C),
then the composition of f and g is convex on C.
5.6.28 Theorem
Let : f : C  R be defined on a convex subset C of Rn such that f(C) is
convex. Let g : f(C)  R be defined on f(C)
(i) If f is strictly concave on C and g is strictly concave and increasing
of f (C), then the composition of f and g is strictly concave on C.
(ii) If f is strictly convex on f(C), then the composition of f and g is
strictly convex on C.
5.6.29 Example
(1) f(X) = X12 + X1X3 – X2 + X22 + X2X3 + 3X32
Hessian matrix of f is
f”(X) =
LPM are
2 0 1
0 2 1
1 1 6
[2] > 0
2 0
0 2
=4>0
2 0 1
0 2 1
1 1 6
= 20 > 0
f”(X) is positive definite for each X  R3.
Therefore, f is strictly conveX on R3 .
(2) Consider the production function f : R++2  R
f(L,K) = L – K, 0<,<1 , +<1
The gradient of f is
L-1 K
f’(L,K) =
L
K -1
The Hessian matrix of f is
f”(L,K) =
(-1)L-2K
L-1K -1
L-1K -1
(-1)LK -2
and its Leading Principal Minors are
|[ (-1)L-2K]|
= (-1)L-2K < 0
(-1)L-2K  L-1K  -1
L-1K -1
(-1)LK -2
= [(-1)(-1) – 22]L2-2K2  -2
=  [1-(+)]L2-2K2  -2 > 0
If follows that this function is strictly concave on the positive
quadrant.
(3) Let f : R++  R be defined by
f(X) = Ln(X)
f”(X) =
-1
<0
2
X
For all X  R++. It follows from theorem 5.6.24
that f is strictly concave on R++
(4) Let f : R+  R defined by
f(X) = X
f’(X) = X-1
f”(X) = (-1)X-2
Hence, f is strictly concave if  < 1
f is strictly convex if  > 1
f is concave and convex if  = 1
Hence, f is concave if  ≤ 1
f is convex if  ≥ 1
(5) Let f : R++n  R defined by
n
f(X) = CtX = CiXi
i=1
then the function g : R++n  R definey by
g(X) = ln
(CtX)
n
= ln ( CiXi)
i=1
is concave on R++n, since f is concave and ln is a
concave and incrasing function.
(6) Let f : R++2  R defined by
f(X) = 1 ln (X1) + 2 ln (X2) ,
claim that f is strictly concave on R++2
1,2 > 0
5.6.30 Theorem
(i) Every local maximizer of a concave function is a global
maximizer
(ii) Every local minimizer of a convex function is a global
minimizer
Proof :
5.6.31 Theorem
(i) A local maximizer of strictly concave function is unique
(ii) A local minimizer of strictly convex function is unique
5.6.32 Example : Consider example 5.6.29(1)
f(X) = X12 + X1X3 – X2 + X22 + X2X3 + 3X32
It was shown that f is strictly convex on R3. The stationary point
are obtained by setting f’(X) = 0, i.e.
2X1 +
X3 = 0
2X2 + X3 = 1
2X1 + X2 + 6X3 = 0
Stationary point :
X* = [1/20,
11/
20,
2/
t
20]
Hessian matriks f’’(X*) is positive definite then theorem 5.5.1,
X* is a local minimizer of f. But f is strictly convex: hence, X* is
the unique global minimizer of f by theorem 5.6.30 and theorem
5.6.31
5.6.33 Example
The Linier Programming Problem
min Co + CtX
S.t.
AX ≤ b , A is m x n , X  Rn
f is strictly conveX on R3. The stationary point f’(X) = 0
The objective function of the Linier Programming (LP) is both
concave and convex. The feasible region
X = {X  Rn|AX ≤ b, X ≥ 0}
is convex since it is the intersection of convex set
X = H  R +n
Dimana H = {X  Rn|AX ≤ b} and
R+n = {X  Rn|X ≥ 0}
Consequently, any optimal solution of the LP is a global optimal
solution. Obviously, This is also true of the maximization problem.
5.7 The Optima of Quasiconcave and
Quasiconvex Function
5.7.1 Definition
Function f : C  R defined on convex subset C of Rn is said to
be quasiconcave on C if and only if
5.7.2 Definition
Function f : C  R defined on convex subset C of Rn is said to be
quasiconvex on C if and only if
5.7.3 Definition
Function f : C  R defined on convex subset C of Rn is said to be
strictly quasiconcave on C if and only if
5.7.4 Definition
Function f : C  R defined on convex subset C of Rn is said to be
strictly quasiconvex on C if and only if
5.7.8 Corollary
(i) Every concave function is quasiconcave
(ii) Every convex function is quasiconvex
5.7.9 Corollary
A linear function is both quasiconcave and quasiconvex.
5.7.10 Theorem
(i) A strictly concave function is strictly quasiconcave
(ii) A strictly convex function is strictly quasiconvex.
5.7.11 Theorem
Let : f : C  R be continuous on a strictly convex subset C of Rn. If is
strictly quasiconcave on C, then the upper contour set UCf () is
strictly convex for every   R.
5.7.12 Example
The Normal Distribution Function
5.7.13 Theorem
(i) A differentiable function f : C  R defined on open convex set C C
Rn is quasiconcave on C if and only if
X,Y  C, f(X) ≥ f(Y)
[f’(Y)]t (X-Y) ≥ 0
(ii) A differentiable function f : C  R defined on open convex set C C
Rn is quasiconvex on C if and only if
X,Y  C, f(X) ≤ f(Y)
[f’(Y)]t (X-Y) ≤ 0
5.8 Constrained Optimization
The general form of the constrained optimization problem may
be expressed as follows :
Max (Min)
f (X)
S.t.
Gk (X) ≥ 0 ,
k = 1, 2, 3, …, m
Where X  Rn. The function f is objective function and the in-equalities
called the constraints. A vector X satisfies the constraints called a
feasible solution and the set of feasible solutions is called feasible set or
feasible region. A feasible solution that maximizes (minimizes) the value
of the objective function on the feasible region is called maximizer
(minimizer). The term optimizer or optimal solution refers to either
maximizer or minimizer.
The distinction between a constrained optimization problem and an
un-constrained problem can be seen fram the geometry of two-variable
problem with a single equality constraint.
Suppose that the problem is :
Max
f(X1,X2)
S.t.
P1 X1 + P2 X2 = Y
5.9 Optimization With One Equality Constraint :
The Two-Variable Case
Solution by Direct Substitution
Consider the problem
Max (Min) f (X1, X2)
S.t.
g (X1, X2) = 0
Dimana f dan g adalah diferensiabel. If, from the constraint
g(X1, X2) = 0, it is possible to express one variable, say X2 in terms of
the other variable then write X2 = h(X1) and substitute this in
f (X1, X2). The problem reduces to the un-constrained problem of
optimizing f [ X1, h(X1)].
5.9.1 Example
Min (X1 – 1)2 + (X2)2
S.t.
2 X 1 + X2) = 4
Let
f (X1, X2) = (X1 – 1)2 + (X2)2
From the constraint, we get
X2 = h(X1) = 4 - 2(X1)
X1* = 9/5
X2* = 4 – 2(9/5) = 2/5
5.9.2 Remark
The solution by direct substitution is particularly useful if the
constraint is linear since it is easy to express one variable in terms
of the other variables. When the constraint is non-linear, it may be
difficult or impossible to obtain such an explicit function. This limits
the usefulness of this method. On the other hand, the graphical
method is limited to two-variable problems. These limitations are
not possessed by a third method of solution called the Lagrange
Multiplier Method.
5.9.3 Definition
Max (Min) f (X)
S.t.
g (X1) = b,
X  R2
The function L defined by
L(,X) = f (X) + [g(X) – b]
Is called the Lagrangean of the problem and the scalar  is called
the Lagrange multiplier
5.9.4 Remark
(1) The Lagrangean is also written as
L(,X) = f (X) – [g(X) – b]
(2) The Gradient of L is
L’ (,X) =
L’ (,X)
L1’ (,X)
L2’ (,X)
=
g (X) – b
f1’ (X)+g1’(X)
f2’ (X)+g2’(X)
and the Hessian matrix of L is
L’’ (,X) =
=
L’’ (,X)
L1’’ (,X)
L2’’ (,X)
L1’’ (,X)
L2’’ (,X)
L11’’ (,X)
L12’’ (,X)
L21’’ (,X)
L22’’ (,X)
0
g1’(X)
g2’(X)
g1’(X)
L11’’(,X)
L21’’(,X)
g2’(X)
L12’’ (,X)
L22’’ (,X)
5.9.5 Theorem
(Necessary or First-Order Condition) Given the problem
Max (Min) f (X)
S.t.
g (X) = b,
X  R2
Where f and g have continuous partial derivatives. Let X* be an
optimizer of f on the feasible set and suppose that g’(X*)  0,
j=1,2. Then there exists a scalar * such that
L’(*, X*) = 0
Proof :
5.9.6 Theorem
(Sufficient or Second-Order Condition) Given the problem
Max (Min)
f (X)
s.t.
g(X) = b, X  R2
Where f and g have continuous second-order partial
derivatives. Let X* and * satisfy L’(*,X*) = 0 and suppose
that gj’(X*)  0, j = 1, 2
(i) If |L”(*,X*)| < 0 , Then X* is a local minimizer
(ii) If |L”(*,X*)| > 0 , Then X* is a local maximizer
Proof :
For notational convenience, we will suppress the arguments of each
function; e.g., g1(X) will be written simply is g1’. From the constraint,
the total differential is g1’dX1+ g1’ dX2 = 0, from which we get
dX2
g 1’
= –
dX1
g 2’
Let
Then
Hence,
Subtituting ….
Y = f(X1,X2)
dY = f1’dX1+ f2’dX2
dY
dX2
dX1 = f1’ + f2’ dX1
d2 Y
We know that if
< 0 at X*, then X* is a local maximizer. Hence,
2
dX1
if the determinant |L’’(*,X*)| > 0 , then X* is a local maximizer.
5.9.7 Example
Min
(X1 – 1)2 + (X2)2
s.t.
2X1 + X2 = 4
Lagrangean: L(,X) = (X1– 1)2 + (X2)2 + (2X1 + X2 – 4)
F.O.C.
L’(,X) = 2X1 + X2 – 4 = 0
L1’(,X) = 2(X1– 1) + 2 = 0
L2’(,X) = 2X2 +  = 0
X1* = 9/5 , X2* = 2/5 , * = – 4/5
0 2 1
S.O.C.
|L”(u*,X*)| = 2 2 0 = -10 < 0
1 0 2
By theorem 5.9.6 X* is local minimizer on feasible set. Hessian
matrix of the objective function is
2
f”(X) = 0
0
2
Which is positive definite for every XR2  f is convex. The
objective function is strictly convex. X* is the unique global
minimizer.
5.9.8 Example
Max
s.t.
Lagrangean: L(u,X)
F.O.C.
L’(,X)
L1’(,X)
L2’(,X)
X1* = Y/2P1
f(X1, X2) = X1X2
P1X1 + P2X2 = Y ,
P1,P2,Y > 0
= X1X2 + (P1X1 + P2X2 – Y)
= P1X1 + P2X2 – Y = 0
= X2 + P1 = 0
= X1 + P2 = 0
, X2* = Y/2P2 , * = – Y/2P1P2
0 P1 P2
S.O.C.
|L”(u*,X*)| = P1 0 1
= 2P1P2 > 0
P2 1 0
By theorem 5.9.6 X* is a local maximizer on the feasible set. X*
the unique global maximizer on the feasible set. They can not
opposite signs since their objective function value would be
negative which can not be optimal since
2
VOF (X*) = X1*X2* = y
>0
4P1P2
X* is the unique global maximizer on the feasible set.
5.10 Optimization With One Equality Constraint :
The n-Variable Case
5.10.1 Definition
Given the problem
Lagrangean :
Max(min)
s.t.
f(X)
g(X) = b ,
L(,X) = f(X) +  [(g(X) – b]
5.10.2 Remark
1) The gradient of L is
L’(,X) =
X  Rn
L’(,X)
L1’(,X)
.
.
.
Ln’(,X)
g(X) – b
f ’(X) + g’1(X)
= 1
.
.
.
fn’(X) + gn’(X)
2) Hessian matriz of L is
L’’(,X) =
0
g1’(X)
g2’(X)
.
.
.
gn’(X)
g1’(X)
L11”(,X)
L21”(,X)
g2’(X)
…
L12”(,X) …
L22”(,X) …
Ln1”(,X)
Ln2”(,X) …
gn’(X)
L1n”(,X)
L2n”(,X)
.
.
.
Lnn”(,X)
5.10.3 Notation
Leading principal submatrices of order k of the Hessian matrix
of L will be denoted by Lk”(,X)
L1’’ (,X) = 0
L2” (,X) =
0
g1’(X)
g1’(X)
L 11”(,X)
L3”(,X) =
0
g1’(X)
g2’(X)
g1’(X) L11”(,X) L12 ”(,X)
g2’(X) L21”(,X) L22”(,X)
.
.
.
etc
5.10.4 Theorem
(Necessary of First-Order Condition). given the problem
Max(min)
f(X)
s.t.
g(X) = b ,
X  Rn
Dimana f dan g have continuous FOC partial derivatives. Let X* be
an optimizer over the feasible set such that g’j(X*)  0, j= 1,2, …,n.
Then there exists a scalar * such that
L’ (*,X*) = 0
5.10.5 Theorem
(Sufficient or Second-Order Condition). Given the problem
Max (min)
f(X)
s.t.
g(X) = b , X  Rn
Where f and g have continuous second-order partial derivatives.
Suppose that X* and * satisfy FOC, i.e. L’(*,X*) = 0.
(i) If |L3”(*,X*)| < 0 , |L4”(*,X*)| < 0, …, |Ln”(*,X*)| < 0 ,
then X* is a local minimizer of f on the feasible set.
(ii) If |L3”(*,X*)| > 0 , |L4”(*,X*)| < 0 ,
|L”5(*,X*)| > 0, …, (-1)n|Ln”(*,X*)| > 0 , then X* is a local
maximizer of f on the feasible set.
Proof : Gue and Thomas (1968)
5.10.6 Example
Max (min)
s.t.
X12 + X1X2 + 2X22 + X32
X1 – 3X2 – 4X3 = 16
Lagrangean :
L(,X) = X12 + X1X2 + 2X22 + X32 +  (X1 – 3X2 – 4X3 – 16)
First Order Condition :
L’ (,X) =
X1 – 3X2 – 4X3 – 16
2X1 + X2 + 
X1 + 4X2 + 3
2X3 – 4
=
0
0
0
0
This system of equations yields the solution
X1* = 4, X2* = – 4, X3* = – 8, * = – 4
Hessian Matrix of L :
L”(,X) =
0
1
-3
-4
1 -3
2 1
1 4
0 0
|L3”(*,X*)| =
0
1
-3
1 -3
2 1
1 4
|L4”(*,X*)| =
0
1
-3
-4
1 -3
2 1
1 4
0 0
S.O.C. :
Hence, X* is a local minimizer on
The Hessian Matrix of f is :
2 1
f”(X) =
1 4
0 0
-4
0
0
2
= -28 < 0
-4
0
0
2
= -168 < 0
the feasible set.
0
0
2
whose eigenvalues are :
1 = 2 ,
2 = 3 + 2 ,
3 = 3 – 2
Which are all positive. Hence, f”(X) is positive definite for all X R3.
That f is strictly convex on R3. This implies X* is the unique global
minimizer of f on the feasible set.
5.11 Optimization With Several Equality
Constraints : The n-Variable Case
5.11.1 Definition
Given The Problem
Max (min) f(X)
s.t.
gk(X) = bk k = 1,2 …,m; m < n , X  Rn
The function L defined by n
L(,X) = f(X) +  k (gk(X) – bk)
k=1
is called the Lagrangean of the problem. The variables 1, 2, …, n
are called the lagrange multipliers.
5.11.2 Remark
The gradient of L, with the arguments of the functions
suppressed, is :
L’ =
g1 – b1
g2 –
.. b2
.
gn – bm
f
gk
+ k
X1
X1
..
.
gk
f
+ k
Xn
Xn
The Hessian Matrix of L is :
L’ =
0
.
.
.
0
…
…
0
.
.
.
0
g1 … gm
X1
X1
.
.
.
.
.
.
g1 … gm
Xn
Xn
g1
Xi
.
.
.
gm
X1
…
…
L11” …
.
.
.
Ln1” …
g1
X
.n
.
.
gm
Xn
L1n”
.
.
.
Lnn”
Note that the matrix L’’ has dimension (m+n) x (m+n).
The Leading Principal Sub Matrix (LPSM) of order k will be
denoted by Lk”.
5.11.3 Theorem
(Necessary or F.O.C.). Given the problem
Max (min)
f(X)
s.t.
gk(X) = bk , k = 1,2, …, m
m < n , X  Rn
Where f and gk (k = 1,2, …,m) have continuous first-Order Partial
Derivatives. Let X* be an optimizer of f over the feasible set and
suppose that the Jacobian determinant
g1(X*) … g1(X*)
 X1
Xn
..
..
.
.
0
gm(X*)
gm(X*)
…
X1
Xn
Then there exist scalars k* (k=1,2,…,m) such that
L’(*, X*) = 0
Proof : Panik (1976).
5.11.4 Theorem
(Sufficient or S.O.C.). Given the problem
MaX (min)
f (X)
s.t.
gk(X) = bk , k = 1,2, …,m , m < n
X  Rn
Where f and gk (k = 1,2,…,m) have continuous Second-Order
Partial Derivatives. Suppose X* and  satisfy the F.O.C., i.e.
L’ (*, X*) = 0
(i) If |L2m+1”(*,X*)| , |L2m+2”(*,X*)| ,…, |Lm+n”(*,X*)|
have the same sign as (-1)m, then X* is a local minimizer of f
on the feasible set.
(ii) If |L2m+1”(*,X*)| , |L2m+2”(*,X*)| ,…, |Lm+n”(*,X*)|
Alternate in sign with the sign of |L2m+1(*,X*)| being that of
(-1)m+1 , then X* is a local maximizer of f on the feasible set.
Proof :