Numerical Optimization in MATLAB
Andrii Parkhomenko
Universitat Autònoma de Barcelona and Barcelona GSE
Spring 2017
Andrii Parkhomenko (UAB & Barcelona GSE)
Numerical Optimization in MATLAB
0 / 24
Optimization
Typical problem in economics: find an optimum
Case 1:
max f (x) = 0.5x − 0.02x 2
x
Easy: just solve for x that satisfies f 0 (x) = 0
Case 2:
1
max f (x) = 0.5 ln x − 0.02x 2 +
x
5
(1 + x) 3
Can’t solve analytically ⇒ need to use numerical methods
Andrii Parkhomenko (UAB & Barcelona GSE)
Numerical Optimization in MATLAB
1 / 24
Numerical Optimization
Continuous differentiable objective function: derivative-based methods
Arbitrary objective function: derivative-free methods
Andrii Parkhomenko (UAB & Barcelona GSE)
Numerical Optimization in MATLAB
2 / 24
Numerical Optimization in MATLAB
These slides cover three functions contained in MATLAB’s Optimization
Toolbox:
fminunc: unconstrained optimization, derivative-based
fmincon: constrained optimization, derivative-based
fminsearch: unconstrained optimization, derivative-free
Andrii Parkhomenko (UAB & Barcelona GSE)
Numerical Optimization in MATLAB
3 / 24
fminunc
Finds parameters that minimize a given function
Syntax:
x = fminunc(objfun,x0)
I
I
I
x0: starting guess
objfun: objective function specified in a different script
x: solution
Andrii Parkhomenko (UAB & Barcelona GSE)
Numerical Optimization in MATLAB
4 / 24
fminunc
Example 1:
1
max f (x) = 0.5 ln x − 0.02x 2 +
x
5
(1 + x) 3
0.6
0.4
0.2
0
f(x)
−0.2
−0.4
−0.6
−0.8
−1
−1.2
−1.4
0
1
2
3
4
5
x
Andrii Parkhomenko (UAB & Barcelona GSE)
Numerical Optimization in MATLAB
5 / 24
fminunc
Example 1:
1
max f (x) = 0.5 ln x − 0.02x 2 +
x
5
(1 + x) 3
Step 1. Main code
x0 = 1;
x = fminunc(@objfun,x0)
Step 2. Function
function y = objfun(x)
y = 0.5*log(x) - 0.02*x^2 + (1+x)^(-5/3);
y = -y;
end
Andrii Parkhomenko (UAB & Barcelona GSE)
Numerical Optimization in MATLAB
6 / 24
fminunc
fminunc uses the BFGS Quasi-Newton method:
xk+1 = xk + αk pk
1
Compute direction pk
pk = −Bk−1 ∇fk
Bk+1 = Bk −
Bk sk sk0 Bk
yk yk0
+
sk0 Bk sk
yk0 sk
where sk ≡ xk+1 − xk and yk ≡ fk+1 − fk
2
Compute stepsize αk . Use line search to approximately solve
min f (xk + αk pk )
αk
Andrii Parkhomenko (UAB & Barcelona GSE)
Numerical Optimization in MATLAB
7 / 24
fminunc
Introduce options:
options = optimset(’Opt1’,Opt1Val,’Opt2’,Opt2Val,...);
x = fminunc(@objfun,x0, options)
Important algorithm options:
Display:
optimset(’Display’, ’off’)
Tolerance level:
optimset(’TolFun’, 1e-6)
optimset(’TolX’, 1e-6)
Maximum number of iterations or evaluations
optimset(’MaxIter’, 400)
optimset(’MaxFunEvals’, 100)
Andrii Parkhomenko (UAB & Barcelona GSE)
Numerical Optimization in MATLAB
8 / 24
fminunc
Caution:
Objective function must be continuous
Local minima and flat regions are big issues
Andrii Parkhomenko (UAB & Barcelona GSE)
Numerical Optimization in MATLAB
9 / 24
fminunc
Example 2. Discontinuous objective function:
(
0.5 ln x − 0.02x 2 + 1 5 ,
0<x ≤4
(1+x) 3
max f (x) =
x
0.5 ln x − 0.02x 2 + 1 5 + 1,
4<x
(1+x) 3
1.5
1
f(x)
0.5
0
−0.5
−1
−1.5
0
1
2
3
4
5
x
Andrii Parkhomenko (UAB & Barcelona GSE)
Numerical Optimization in MATLAB
10 / 24
fminunc
Example 3. Objective function with flat regions:
(
0.5 ln x − 0.02x 2 + 1 5 , 0 < x ≤ 4
(1+x) 3
max f (x) =
x
0.441546,
4<x
0.6
0.4
0.2
0
f(x)
−0.2
−0.4
−0.6
−0.8
−1
−1.2
−1.4
0
1
2
3
4
5
x
Andrii Parkhomenko (UAB & Barcelona GSE)
Numerical Optimization in MATLAB
11 / 24
fminunc
Example 4. Objective function with local maxima:
max f (x) = sin(3x) + x − 0.25x 2
x
2
1.5
f(x)
1
0.5
0
−0.5
−1
0
1
2
3
4
5
x
Andrii Parkhomenko (UAB & Barcelona GSE)
Numerical Optimization in MATLAB
12 / 24
fminunc
To obtain more details on the workings of fminunc:
Read documentation: Help → Type ’fminunc’
Read the code: Type ’open fminunc’
Andrii Parkhomenko (UAB & Barcelona GSE)
Numerical Optimization in MATLAB
13 / 24
fmincon
Finds parameters that minimize a given function, subject to a
constraint
Same algorithm as fminunc
Syntax:
x = fmincon(objfun,x0,A,b,Aeq,beq,lb,ub,nlcon,options)
I
I
I
I
I
I
I
x0: starting guess
objfun: objective function specified in a different script
A, b: linear inequality constraint: Ax ≤ b
Aeq, beq: linear equality constraint: Ax = b
lb, ub: lower and upper bounds: x ∈ [lb, ub]
nlcon: nonlinear constraint
x: solution
Andrii Parkhomenko (UAB & Barcelona GSE)
Numerical Optimization in MATLAB
14 / 24
fmincon
Example 1:
max f (x) = 0.5 ln x − 0.02x 2 +
x
1
5
s.t. x ≤ a
(1 + x) 3
0.6
0.4
0.2
0
f(x)
−0.2
−0.4
−0.6
−0.8
−1
−1.2
−1.4
0
1
2
3
4
5
x
Andrii Parkhomenko (UAB & Barcelona GSE)
Numerical Optimization in MATLAB
15 / 24
fmincon
Sometimes it’s better to reformulate a constrained problem as an
unconstrained
I
Can use a formulation with Lagrange multipliers
Andrii Parkhomenko (UAB & Barcelona GSE)
Numerical Optimization in MATLAB
16 / 24
fminsearch
Finds parameters that minimize a given function
Derivative-free algorithm (Nelder-Mead)
Syntax:
x = fminsearch(objfun,x0,options)
I
I
I
x0: starting guess
objfun: objective function specified in a different script
x: solution
Andrii Parkhomenko (UAB & Barcelona GSE)
Numerical Optimization in MATLAB
17 / 24
fminsearch
Nelder-Mead algorithm:
1
select 2 points: x1 and x2
2
order s.t. x̂ = x1 and x̄ = x2 if f (x̂) ≥ f (x̄); and x̂ = x2 and x̄ = x1
otherwise
3
4
5
6
reflection: x R = x̂ + (x̂ − x̄)
if f (x R ) > f (x̂) and f (x R ) > f (x̄), expansion: x E = x̂ + α(x R − x̂), α > 1
if f (x R ) ≤ f (x̂) and f (x R ) > f (x̄), update x̄ = x R and go to step 2
if f (x R ) ≤ f (x̂) and f (x R ) ≤ f (x̄), contraction: x C = x̂ + β(x̄ − x̂), β < 21
if f (x E ) > f (x R ), update x̄ = x E and go to step 2
if f (x E ) < f (x R ), update x̄ = x R and go to step 2
if f (x C ) > f (x̄), update x̄ = x C and go to step 2
if f (x C ) < f (x̄), shrink: replace x̄ = x̂ + γ(x̄ − x̂) and go to step 2, γ < 1
Iterate until convergence: x̂ ≈ x̄ and/or f (x̂) ≈ f (x̄)
Andrii Parkhomenko (UAB & Barcelona GSE)
Numerical Optimization in MATLAB
18 / 24
fminsearch
Example 1:
1
max f (x) = 0.5 ln x − 0.02x 2 +
x
5
(1 + x) 3
0.6
0.4
0.2
0
f(x)
−0.2
−0.4
−0.6
−0.8
−1
−1.2
−1.4
0
1
2
3
4
5
x
Andrii Parkhomenko (UAB & Barcelona GSE)
Numerical Optimization in MATLAB
19 / 24
fminsearch
Example 2. Discontinuous objective function:
(
0.5 ln x − 0.02x 2 + 1 5 ,
0<x ≤4
(1+x) 3
max f (x) =
x
0.5 ln x − 0.02x 2 + 1 5 + 1,
4<x
(1+x) 3
1.5
1
f(x)
0.5
0
−0.5
−1
−1.5
0
1
2
3
4
5
x
Andrii Parkhomenko (UAB & Barcelona GSE)
Numerical Optimization in MATLAB
20 / 24
fminsearch
Example 3. Objective function with flat regions:
(
0.5 ln x − 0.02x 2 + 1 5 , 0 < x ≤ 4
(1+x) 3
max f (x) =
x
0.441546,
4<x
0.6
0.4
0.2
0
f(x)
−0.2
−0.4
−0.6
−0.8
−1
−1.2
−1.4
0
1
2
3
4
5
x
Andrii Parkhomenko (UAB & Barcelona GSE)
Numerical Optimization in MATLAB
21 / 24
fminsearch
Example 4. Objective function with local maxima:
max f (x) = sin(3x) + x − 0.25x 2
x
2
1.5
f(x)
1
0.5
0
−0.5
−1
0
1
2
3
4
5
x
Andrii Parkhomenko (UAB & Barcelona GSE)
Numerical Optimization in MATLAB
22 / 24
fminsearch
Example 5. Objective function with many discontinuities:
max f (x) = bxc + 0.2x − 0.04x 2
x
9
8
7
6
f(x)
5
4
3
2
1
0
0
Andrii Parkhomenko (UAB & Barcelona GSE)
5
10
x
15
Numerical Optimization in MATLAB
20
23 / 24
fminunc vs fminsearch
Rule of thumb: use fminunc for continuous differentiable functions,
fminsearch otherwise
No option guarantees that you find a (global) optimum
For complicated cases:
I
I
I
understand the problem well
try different algorithms
try different starting points
Andrii Parkhomenko (UAB & Barcelona GSE)
Numerical Optimization in MATLAB
24 / 24