OR II
GSLM 52800
1
Outline
optimization – unconstrained
optimization
 classical
 dimensions
 feasible
of optimization
direction
2
Classical Optimization Results
 Unconstrained Optimization


different dimensions of optimization conditions
nature of conditions



necessary conditions (必要條件): satisfied by any minimum (and
possibly by some non-minimum points)
sufficient conditions (充分條件): if satisfied by a point, implying
that the point is a minimum (though some minima may not satisfy
the conditions)
order of conditions



first-order conditions: in terms of the first derivatives of f & gj
second-order conditions: in terms of the second derivatives of f & gj
general assumptions: f, g, gj  C1 (i.e., once continuously
differentiable) or C2 (i.e., twice continuously differentiable) as
required by the conditions
3
Feasible Direction

S  n: the feasible region
x
 S: a feasible point
feasible direction d of x: if there exists  >
0 such that x+d  S for 0 <  < 
a
4
Two Key Concepts
for Classical Results
 f:
the direction of steepest accent
 gradient
of f at x0 being orthogonal to
the tangent of the contour f(x) = c at x0
5
The Direction of Steepest Accent f



contours of f(x1, x2) = x12  x22
f: direction of steepest accent
in some sense, increment of unit
move depending on the angle with
f

within 90 of f: increasing


closer to 0: increasing more
beyond 90 of f: decreasing


x2
x1
closer to 180: decreasing more
above results generally true for
any f
6
Gradient of f at x0 Being Orthogonal to
the Tangent of the Contour f(x) = c at x0
=
2
x1
 f(x10, x20)
=c
 f(x1, x2)
d

2
x2
on the tangent plane at x0
 f(x0+d)
 c for small 
 roughly
speaking, for f(x0) = c, f(x0+d) = c for
small  when d is on the tangent plane at x0
7
First-Order Necessary Condition
(FONC)

f  C1 on S and x* a local minimum of f

then for any feasible direction d at x*, Tf(x*)d
0
 increasing
of f at any feasible direction
f(x) = x2 for 2  x  5
f(x, y) = x2 + y2
for 0  x, y  2
f(x, y) = x2 + y2 for
x  3, y  3
8
FONC for Unconstrained NLP
 C1 on S & x* an interior local minimum
(i.e., without touching any boundary) 
Tf(x*) = 0
f
9
FONC Not Sufficient
 Example
 Tf((0,
 (0,
0))d = 0 for all feasible direction d
0): a maximum point
 Example
 f(0)
x
3.2.2: f(x, y) = -(x2 + y2) for 0  x, y
3.2.3: f(x) = x3
=0
= 0 a stationary point
10
Feasible Region with
Non-negativity Constraints
f (x* )
 0, if x*j  0;
x j
or, equivalently
f (x* )
 0,
x j
f (x* )
 0, if x*j  0.
x j
x*j
f (x* )
0
x j
 Example
3.2.4. (Example 10.8 of JB) Find
candidates of the minimum points by the
FONC.
2
2
2
3
x

x

x
 min f(x) =
1
2
3  2 x1x2  2 x1x3  2 x1
 subject
to x1  0, x2  0, x2  0
11
Second-Order Conditions
 another
 f(x)
form of Taylor’s Theorem
= f(x*)+Tf(x*)(x-x*)
+0.5(x- x*)TH(x*)(x - x*)+ ,
 where
 if
 being small, dominated by other terms
Tf(x*)(x-x*) = 0,
 f(x)
 f(x*) (x- x*)TH(x*)(x - x*)  0
12
Second-Order Necessary Condition
f
 C2 on S
x* is a local minimum of f, then for any
feasible direction d  n at x*,
 if
 (i). Tf(x*)d
 (ii).
 0, and
if Tf(x*)d = 0, then dTH(x*)d  0
13
Example 3.3.1(a)
 SONC
satisfied
f(x) = x2 for 2  x  5
f(x, y) = x2 + y2
for 0  x, y  2
f(x, y) = x2 + y2 for
x  3, y  3
14
Example 3.3.1(b)
 SONC:
more discriminative than FONC
y) = -(x2 + y2) for 0  x, y in Example
3.2.2
 f(x,
 (0,
0), a maximum point, failing the SONC
15
SONC for Unconstrained NLP
f
 C2 in S
 x*
an interior local minimum of f, then
 (i). Tf(x*)
 (ii).
= 0, and
for all d, dTH(x*)d  0
 (ii)
 H(x*) being positive semi-definite
 convex
f satisfying (ii) (and actually more)
16
Example 3.3.2
 identity
candidates of minimum points for
the f(x) = x13  x22  3x1  2 x2
 Tf(x*)
x
= (3x12  3x1, 2 x2  2)
= (1, -1) or (-1, -1)
6 x1 0 
 H(x) = 

0
2


 (1,
-1) satisfying SONC but not (-1, -1)
17
SONC Not Sufficient
 f(x,
y) = -(x4 + y4)
 Tf((0,
 (0,
0))d = 0 for all d
0) a maximum
18
SOSC for Unconstrained NLP
f
 C2 on S  n and x* an interior point
 if
 (i).
Tf(x*) = 0, and
 (ii).
 x*
H(x*) is positive definite
a strict local minimum of f
19
SOSC Not Necessary
 Example
x
3.3.4.
= 0 a minimum of f(x) = x4
 SOSC
not satisfied
20
Example 3.3.5
f ( x1, x2 , x3 )  3x12  x22  x32  2 x1x2  2 x1x3  2 x1
 In
Example 3.2.4, is (1, 1, 1) a minimum?
 6  2 2 
 2 2 0 
H
(
x
)

.


 2 0
2 
6
> 0;
6
2
2
2
 positive
6
2 2
2
2
0  (6)(2)(2)  (2)(2)(2)  (2)(2)(2)  8
2
0
2
 12  (2)(2)  8;
definite, i.e., SOSC satisfied
21
Effect of Convexity
for all y in the neighborhood of x*  S,
Tf(x*)(y-x*)  0
 If
 convexity
 f(y)
of f implies
 f(x*) + Tf(x*)(y-x*)  f(x*)
 x*
a local min of f in the neighborhood of x*
 x*
a global minimum of f
22
Effect of Convexity
 C2 convex  H positive semi-definite
everywhere
f

Taylor's Theorem, when Tf(x*)(x-x*) = 0,
 f(x)
= f(x*) + Tf(x*)(x-x*)
+ (x- x*)TH(x* + (1-)x)(x - x*)
= f(x*) + (x- x*)TH(x* + (1-)x)(x - x*)
 f(x*)

x* a local min  a global min
23
Effect of Convexity

facts of convex functions

(i). a local min = a global min
 (ii).
H(x) positive semi-definite everywhere
 (iii).
strictly convex function, H(x) positive definite
everywhere

implications
f  C2 convex function, the FONC Tf(x*) = 0 is
sufficient for x* to be a global minimum
 for
 if
f strictly convex, x* the unique global min
24