CS B553: ALGORITHMS FOR
OPTIMIZATION AND LEARNING
Univariate optimization
f(x)
x
KEY IDEAS
Critical points
 Direct methods

Exhaustive search
 Golden section search


Root finding algorithms
Bisection
 [More next time]

Local vs. global optimization
 Analyzing errors, convergence rates

Figure 1
f(x)
Local maxima
Inflection point
Local minima
x
Figure 2a
f(x)
a
b
x
Figure 2b
Find critical points, apply 2nd derivative test
a
b
f(x)
x
Figure 2b
f(x)
a
b
x
Figure 2c
f(x)
a
b
Global minimum must be one
of these points
x
Figure 3
Exhaustive grid search
f(x)
a
b
x
Exhaustive grid search
f(x)
a
b
x
Figure 4
Two types of errors
Analytical error
f(x)
f(xt)
f(x*)
x* xt
Geometric error
x
Does exhaustive grid search
achieve e/2 geometric error?
f(x)
x*
b
a
e
x
Does exhaustive grid search
achieve e/2 geometric error?
Not necessarily for multi-modal
objective functions
f(x)
x*
b
a
Error
x
Figure 5
LIPSCHITZ CONTINUITY
Slope +K
|f(x)-f(y)|  K|x-y|
Slope -K
Figure 6
Exhaustive grid search achieves Ke/2
analytical error in worst case
f(x)
b
a
e
x
Figure 7a
Golden section search
f(x)
a
m
Bracket [a,b]
Intermediate point m with f(m) < f(a),f(b)
b
x
Figure 7b
Golden section search
f(x)
a
c
m
b
Candidate bracket 1 [a,m]
Candidate bracket 2 [c,b]
x
Figure 7b
Golden section search
f(x)
a
m
b
x
Figure 7b
Golden section search
f(x)
a
m
c
b
x
Figure 7b
Golden section search
f(x)
a
m
b
x
Optimal choice: based on golden ratio
f(x)
a
c
m
b
Choose c so that (c-a)/(m-c) = , where  is the golden ratio
=> Bracket reduced by a factor of -1 at each step
x
NOTES
Exhaustive search is a global optimization:
error bound is for finding the true optimum
 GSS is a local optimization: error bound holds
only for finding a local minimum
 Convergence rate is linear:

|𝑥𝑛+1 −𝑥 ∗ |
lim
𝑛→∞ |𝑥𝑛 −𝑥 ∗ |
= 𝜇 with 0 < 𝜇 < 1
xn = sequence of bracket midpoints
Figure 8
Root finding: find x-value where f’(x) crosses 0
f(x)
f’(x)
x
Figure 9a
Bisection
g(x)
a
b
Bracket [a,b]
Invariant: sign(f(a)) != sign(f(b))
Figure 9
Bisection
g(x)
a
m
b
Bracket [a,b]
Invariant: sign(f(a)) != sign(f(b))
Figure 9
Bisection
g(x)
a
b
Bracket [a,b]
Invariant: sign(f(a)) != sign(f(b))
Figure 9
Bisection
g(x)
a
m
b
Bracket [a,b]
Invariant: sign(f(a)) != sign(f(b))
Figure 9
Bisection
g(x)
a
b
Bracket [a,b]
Invariant: sign(f(a)) != sign(f(b))
Figure 9
Bisection
g(x)
a
m
b
Bracket [a,b]
Invariant: sign(f(a)) != sign(f(b))
Figure 9
Bisection
g(x)
a
b
Bracket [a,b]
Invariant: sign(f(a)) != sign(f(b))
Linear convergence: Bracket size is reduced by
factor of 0.5 at each iteration
NEXT TIME
Root finding methods with superlinear
convergence
 Practical issues
