Institute for Mathematics
Karl-Franzens-Universität Graz
Christian Clason
problem set 03
PS 621.221
winter 2009/10
Optimization I
Homework problems (to be completed by 29.10.2009)
H 3.1 Missing descent directions
Consider f : R2 → R, f ( x1 , x2 ) = x22 − 3x2 x12 + 2x14 .
(a) Compute all stationary points of f .
(b) Show that in x = (0, 0), every direction d is an ascent direction.
(c) Is x = (0, 0) a local minimizer of f ?
H 3.2 Line searches are necessary
Consider the steepest descent method for the minimization of the function f : R2 → R,
f ( x1 , x2 ) = x12 + x22 with constant step length α = 1, i.e.
x k +1 = x k − ∇ f ( x k )
Beginning with x0 = ( 12 , 0)T , compute several steps of the method.
H 3.3 Line search for quadratic functions
Let f : Rn → R,
1 T
x Qx + c T x + γ,
2
be a quadratic function with Q ∈ Rn×n symmetric positive definite, c ∈ Rn and γ ∈ R. Let
x ∈ Rn and d ∈ Rn \ {0} with ∇ f ( x )T d < 0 be given. Show that:
f (x) =
(a) the step length
tmin = −
∇ f ( x)T d
d T Qd
gives the steepest descent of f along d, i.e., f ( x + tmin d) ≤ f ( x + td) holds for all t ∈ R,
with strict inequality for t 6= tmin ,
(b) there is a constant θ > 0 independent of x und d, such that
f ( x + tmin d) ≤ f ( x ) − θ
∇ f ( x)T d
k d k2
2
.
please turn over!
Programming assignment (send to [email protected] by 5.11.2009)
P 3.1 Descent method with line search
Implement the steepest descent method with line search according to the Armijo rule. Proceed
as follows:
(a) Write a function which computes a step length by the Armijo rule. In Matlab, this could
be defined as follows:
function alpha = armijo(f,gradf,x,d,sigma,beta)
Here, f and gradf are anonymous functions which return for given x the functional value
f ( x ) and the gradient ∇ f ( x ), respectively. Furthermore, x is the starting point and d the
search direction; sigma and beta are the parameters introduced in the lecture. The return
value alpha is a step length which satisfies the Armijo condition.
(b) Write a function which implements the steepest descent method and calls the function
armijo. The iteration should be terminated if k∇ f ( x k )k < ε for a given tolerance ε > 0,
or a maximal number of iterations has been reached. This function must be defined as
follows:
function X = steepestdesc(f,gradf,x0,eps,maxit)
with anonymous functions f and gradf as described above, starting value x0, tolerance
eps and maximum number of iterations maxit. The return value X should contain all
iterates x k as a matrix X=[x0;x1;x2;...].
(c) Test your implementation with the Rosenbrock function f ( x1 , x2 ) = 100( x2 − x12 )2 + (1 −
x1 )2 . A corresponding anonymous function is:
f = @(x) 100*(x(2)-x(1)^2)^2+(1-x(1))^2;
Use as starting value x0 = (1, −0.5)T and as parameters e = 10−2 , maxit = 10000,
σ = 10−2 and β = 21 . Compare your results with those obtained using the search direction
∇ f (xk )
dk = − k∇ f (xk )k .
(d) Modify your program such that it uses the Armijo-Goldstein conditions instead of the
Armijo condition. Test this method with Rosenbrock function as well.
Comment your program suitably! (What does this variable stand for? What is this loop doing? What
do the following lines compute?)