L6 Optimal Design concepts pt B
•
•
•
•
•
•
•
Homework
Review
Single variable minimization
Multiple variable minimization
Quadratic form
Positive definite tests
Summary
1
Global/local optima
Global Maximum?
f(x*)≤ f(x)
Anywhere in S
Local Maximum?
f(x*)≤ f(x)
In small neighborhood N
Closed & Bounded
Weierstrass Theorem
2
Taylor Series Expansion
Assume f(x) is:
1. Continuous function of a single variable x
2. Differentiable n times
3. x ∈ S, where S is non-empty, closed, and bounded
4. therefore x* is a possible optima
df ( x*)
1 d 2 f ( x*)
2
f ( x )  f ( x*) 
( x  x*) 
(
x

x
*)
 ...
2
dx
2 dx
1 d 3 f ( x*)
1 d 4 f ( x*)
3
4
(
x

x
*)

(
x

x
*)
 h.o.t.
3
4
3! dx
4! dx
3
Taylor Series Approximations
let x  x*  d , a small change or vicinit y of x * ...then
f ( x )  f ( x * d )
df ( x*)
1 d 2 f ( x*) 2 1 d 3 f ( x*) 3
f ( x )  f ( x*) 
d
d 
d  h.o.t.
2
3
dx
2 dx
3! dx
df ( x*)
d
Firs t Order Appr oximation
dx
df ( x*)
1 d 2 f ( x*) 2
f ( x )  f ( x*) 
d
d
Second Order Appr oximation
2
dx
2 dx
f ( x )  f ( x*) 
4
Single variable minimization
Given that x* is the minimum of f(x), then any
movement away from x* is “uphill”, therefore to
guarantee that a move goes uphill
f ( x )  0
1
f ( x*)d 2  ...
2
1
1
f ( x*)d 3  f iv ( x*)d 4  h.o.t.
3!
4!
f ( x )  f ( x )  f ( x*)  f ( x*)d 
First-order necessary condition
f ( x*)  0
5
Stationary point=max,min,neither
Any points satisfying f ( x*)  0
Are called “stationary” points.
Those points are a:
1. Min pt, or
2. Max pt, or
3. Neither (i.e. an inflection pt)
We need another test!
6
Second-order sufficient condition
Look at second order term
1
f ( x*)d 2  ...
2
1
1
f ( x*)d 3  f iv ( x*)d 4  h.o.t.
3!
4!
f ( x )  f ( x )  f ( x*)  f ( x*)d 
Second-order sufficient condition for a minimum
f ( x*)  0
7
Second-order condition?
What if f ( x*)  0
Then f(x*) is not a minimum of f(x*).
It is a maximum of f(x*).
8
Single variable optimization
First-order necessary condition
f ( x*)  0
f ( x*)  0
Second-order sufficient
condition for a minimum
f ( x*)  0
Second-order sufficient
condition for a maximum
? ?
9
Higher—order tests?
What if f ( x*)  0
Then the second order test fails. We need
higher order derivatives…
Min
f even ( x*)  0
Max
f even ( x*)  0
10
Second-order necessary conditions
Note the “= 0” possibility
f ( x*)  0
A pt not satisfying this
test is not a min!
f ( x*)  0
A pt not satisfying this
test is not a max!
11
Multiple variable optimization
If x* is the minimum of f(x), then any movement
away from x* is “uphill”.
f  f (x )  f (x*)  0
1
f  f ( x * d )  f ( x*)  f T ( x*)d  dT H d  R
2
1
f  f T ( x*)d  dT H d (second order)
2
dot produc t tri ple product
scalar
scalar
f  scalar
How can we guarantee that for a move in any d, we
make away from x*, we go “uphill”?
12
First-order necessary Condition
For x* to be a local minimum: f  f (x )  f (x*)  0
1 T
T
f  f ( x*)d  d H d
2
 f 
0
 x 
1rst order term
0
 1
 
 f 
0


T
f ( x*)  x2
 0
f (x*)  0


0

  
 
 f 
 


0

 x n  x*
13
Second-order sufficient condition
For x* to be local minimum:
dT H d  0
That is H(x*) must be positive definite
Remember that
dT H d
has “quadratic form”
14
Quadratic form of a matrix
1 T
Given F (x )  x A x where A is symmetric, for example :
2
1 0 0
 x1 
x   x2 
x T  x1 x2 x3 
A  0 1 0


 x3 
0 0 1
1 0 0  x   x 
1
1
T
T


x A x  x  (Ax ) Ax  0 1 0  x2    x2 

x  x 
0 0 1  3   3 
x A x  x  ( Ax )  x1
T
T
x2
x3 
T
 x1 
 x2   x12  x22  x32
 x3 
1 2
F ( x )  x1  x22  x32 
2
15
F (x )  2 x12  2 x1 x2  4 x1 x3  6 x22  4 x2 x3  5 x32
Wuad From Ex
16
Positive definiteness?
• By inspection
• Leading principal minors
• Eigenvalues
1 T
Given F (x )  x A x where A is symmetric
2
1 2

F
(
x
)

x1  x22  x32 
e.g. by inspection
2
17
Find leading principal minors to
check PD of A(x)
 a11 a12
a
a22
21
A  a31 a32



an1 an 2
a13  a1n 
a23  a2 n 

a33  a3n 
   
an 3  ann 
M 1  a11
M2 
a11
a12
a21 a22
a11
a12
a13
M 3  a21 a22
a23
a31 a32
a33
18
Principal Minors Test for PD
A matrix is positive definite if:
1.No two consecutive minors can be zero AND
2. All minors are positive, i.e.
Mk  0
If two consecutive minors are zero
The test cannot be used.
19
Principal Minors Test for ND
A matrix is negative definite if:
1.No two consecutive minors can be zero AND
2. Mk<0 for k=odd
3. Mk>0 for k=even
If two consecutive minors are zero
The test cannot be used.
20
Eigenvalues
Ax  x
Ax  x  0
( A  I )x  0
Since x should not be zero… we should find
values for lambda such that
A  I  0
21
Eigenvalue test
A  I  0
Form
Positive Definite (PD)
xT A x  0 for all x ,
other than x  0
Positive Semi-def (PSD)
xT A x  0 for all x , and
Eigenvalue Test
i  0
i  0
xT A x  0 for at least one x  0
Indefinite
xT A x  0 for some x
xT A x  0 for other x
i  0
i  0
22
Eigenvalue example
0
 1 1
A   1 1 0 


0  1
 0
A  I 
0   0 0 
 1 1
A  I   1  1 0    0  0   0

 

0  1  0 0  
 0
1 
1
0
1
1 
0
0
0
1 
0
A  I  ( 1   )[( 1   )2  1]  0
1  1, 2  0, 3  2
( 1   )2  1  0
(1  2  2 )  1  0
 2  2  0
( 2   )  0
Therefore A is NSD
23
Summary
•
•
•
•
Single variable minimization
Multiple variable minimization
Quadratic form
Positive definite tests
24