S. Boyd
EE364
Lecture 4
Convex optimization problems
• optimization problem in standard form
• convex optimization problem
• standard form with generalized inequalities
• multicriterion optimization
4–1
Optimization problem: standard form
minimize f0(x)
subject to fi(x) ≤ 0, i = 1, . . . , m
hi(x) = 0, i = 1, . . . , p
where fi, hi : Rn → R
• x is optimization variable
• f0 is objective or cost function;
fi(x) ≤ 0 are the inequality constraints;
hi(x) = 0 are the equality constraints
• x is feasible if it satisfies the constraints;
the feasible set C is the set of all feasible points;
problem is feasible if there are feasible points
• problem is unconstrained if m = p = 0
• optimal
(can be
optimal
optimal
value is f ? = inf x∈C f0(x)
−∞; convention: f ? = +∞ if infeasible);
point: x ∈ C s.t. f (x) = f ?;
set: Xopt = {x ∈ C | f (x) = f ?}
Convex optimization problems
4–2
example:
minimize x1 + x2
subject to −x1 ≤ 0
−x2 ≤ 0
1 − x1x2 ≤ 0
(1)
• feasible set C is half-hyperboloid
• optimal value is f ? = 2
• only optimal point is x? = (1, 1)
5
4
C
x2
3
2
1
0
0
Convex optimization problems
1
2
x1
3
4
5
4–3
Implicit and explicit constraints
explicit constraints: fi(x) ≤ 0, hi(x) = 0
implicit constraint: x ∈ dom fi, x ∈ dom hi
D = dom f0 ∩ · · · ∩ dom fm ∩ dom h1 ∩ · · · ∩ dom hp
is called domain of the problem
example
minimize − log x1 − log x2
subject to x1 + x2 − 1 ≤ 0
has an implicit constraint
x ∈ D = {x ∈ R2 | x1 > 0, x2 > 0}
Convex optimization problems
4–4
Feasibility problem
suppose objective f0 = 0, so
f? =
½
0
if C =
6 ∅
+∞ if C = ∅
thus, problem is really to
• either find x ∈ C
• or determine that C = ∅
i.e., solve the inequality / equality system
fi(x) ≤ 0, i = 1, . . . , m
hi(x) = 0, i = 1, . . . , p
or determine that it is inconsistent
Convex optimization problems
4–5
Convex optimization problem
convex optimization problem in standard form:
minimize f0(x)
subject to fi(x) ≤ 0, i = 1, . . . , m
aTi x − bi = 0, i = 1, . . . , p
• f0, f1, . . . , fm convex
• affine equality constraints
• feasible set is convex
often written as
minimize f0(x)
subject to fi(x) ≤ 0, i = 1, . . . , m
Ax = b
where A ∈ Rp×n
Convex optimization problems
4–6
example. problem (1)
• has convex objective and feasible set
• is not a standard form convex optimization problem
since f3(x) = 1 − x1x2 is not convex
can easily be cast as standard form convex optimization
problem:
minimize x1 + x2
subject to −x1 ≤ 0
−x2 ≤ 0
√
1 − x1x2 ≤ 0
(1 −
√
x1x2 is convex on R2+)
many different ways, e.g.,
minimize x1 + x2
subject to −x1 ≤ 0
−x2 ≤ 0
− log x1 − log x2 ≤ 0
Convex optimization problems
4–7
example. fi all affine yields linear program
minimize cT0 x + d0
subject to cTi x + di ≤ 0, i = 1, . . . , m
Ax = b
which is a convex optimization problem
example. minimum norm approximation with limits
on variables
minimize kAx − bk
subject to li ≤ xi ≤ ui, i = 1, . . . , n
is convex
example. maximum entropy with linear equality
constraints
P
minimize
i xi log xi
subject to P
xi ≥ 0, i = 1, . . . , n
i xi = 1
Ax = b
is convex
(more on these later)
Convex optimization problems
4–8
Local and global optimality
x ∈ C is locally optimal if it satisfies
y ∈ C, ky − xk ≤ R =⇒ f0(y) ≥ f0(x)
for some R > 0
c.f. (globally) optimal, which means x ∈ C,
y ∈ C =⇒ f0(y) ≥ f0(x)
for cvx opt problems, any local solution is also global
proof:
• suppose x is locally optimal, but y ∈ C, f0(y) <
f0(x)
• take small step from x towards y, i.e., z = λy +
(1 − λ)x with λ > 0 small
• z is near x, with f0(z) < f0(x); contradicts local
optimality
Convex optimization problems
4–9
An optimality criterion
suppose f0 is differentiable in convex problem
then x ∈ C is optimal iff
y ∈ C =⇒ ∇f0(x)T (y − x) ≥ 0
• −∇f0(x) defines supporting hyperplane for C at x
• if you move from x towards any feasible y, f0 does
not decrease
contour lines of f0
C
x
−∇f0(x)
• hence x ∈ C, ∇f0(x) = 0 implies x optimal
• for unconstr. problems, x is optimal iff ∇f0(x) = 0
Convex optimization problems
4–10
Quasiconvex optimization problem
quasiconvex optimization problem in standard form:
minimize f0(x)
subject to fi(x) ≤ 0, i = 1, . . . , m
Ax = b
• f0 is quasiconvex
• f1, . . . , fm are convex
• affine equality constraints
• feasible set, all sublevel sets are convex
example. linear-fractional programming
minimize (aT x + b)/(cT x + d)
subject to Ax = b, F x ¹ g, cT x + d > 0
Convex optimization problems
4–11
Solving quasiconvex problems via
bisection
minimize f0(x)
subject to fi(x) ≤ 0, i = 1, . . . , m
Ax = b
fi convex, f0 quasiconvex
idea: express sublevel set f0(x) ≤ t as sublevel set of
convex function:
f0(x) ≤ t ⇔ φt(x) ≤ 0
where φt : Rn → R is convex in x for each t
now solve quasiconvex problem by bisection on t,
solving convex feasibility problem
φt(x) ≤ 0,
fi(x) ≤ 0, i = 1, . . . , m,
Ax = b
(with variable x) at each iteration
Convex optimization problems
4–12
bisection method for quasiconvex problem:
given l < p∗; feasible x; ² > 0
u := f0(x)
repeat
t := (u + l)/2
solve convex feasibility problem
φt(x) ≤ 0, fi(x) ≤ 0, Ax = b
if feasible,
u := t
x := any solution of feas. problem
else l := t
until u − l ≤ ²
• reduces quasiconvex problem to sequence of convex
feasibility problems
• finds ²-suboptimal solution in log2(1/²) iterations
Convex optimization problems
4–13
Epigraph form
write standard form problem as
minimize t
subject to f0(x) − t ≤ 0,
fi(x) ≤ 0, i = 1, . . . , m
hi(x) = 0, i = 1, . . . , p
• variables are (x, t)
• m + 1 inequality constraints
• objective is linear : t = eTn+1(x, t)
• if original problem is cvx, so is epigraph form
f0(x)
−en+1
C
linear objective is ‘universal’ for convex optimization
Convex optimization problems
4–14
Standard form with generalized
inequalities
convex optimization problem in standard form with
generalized inequalities:
minimize f0(x)
subject to fi(x) ¹Ki 0, i = 1, . . . , L
Ax = b
where:
• f0 : Rn → R convex
• ¹Ki are generalized inequalities on Rmi
• fi : Rn → Rmi are Ki-convex
example. semidefinite programming
minimize cT x
subject to A0 + x1A1 + · · · + xnAn ¹ 0
where Ai = ATi ∈ Rp×p
Convex optimization problems
4–15
How fi, hi are described
analytical form
functions can have analytical form, e.g.,
f (x) = xT P x + 2q T x + r
f is specified by giving the problem data, coefficients,
or parameters, e.g.
P = P T ∈ Rn×n,
q ∈ Rn ,
r∈R
oracle form
functions can be given by oracle or subroutine
that, given x, computes f (x) (and maybe ∇f (x),
∇2f (x), . . . )
• oracle model can be useful even if f has analytic
form, e.g., linear but sparse
• how f given affects choice of algorithm, storage
required to specify problem, etc.
Convex optimization problems
4–16
Some hard problems
‘slight’ modification of convex problem can be very
hard
• convex maximization, concave minimization, e.g.
maximize kxk
subject to Ax ¹ b
• nonlinear equality constraints, e.g.
minimize cT x
subject to xT Pix + qiT x + ri = 0, i = 1, . . . , K
• minimizing over non-convex sets, e.g., Boolean
variables
find
x
such that Ax ¹ b,
xi ∈ {0, 1}
Convex optimization problems
4–17
Multicriterion optimization
vector objective
F (x) = (F1(x), . . . , FN (x))
F1, . . . , FN : Rn → R
(can include equality, inequality constraints)
Fi called objective functions: roughly speaking, want
all Fi small
family of specifications indexed by t ∈ RN :
F (x) ¹ t
i.e., Fi(x) ≤ ti, i = 1, . . . , N .
achievable specification: t s.t. F (x) ¹ t feasible
Convex optimization problems
4–18
Achievable specifications
set of achievable objectives:
A = {t ∈ RN | ∃x s.t. F (x) ¹ t}
t2
A (achievable specs)
t1
if Fi are convex then A is convex
boundary of A is called (optimal) tradeoff surface
Convex optimization problems
4–19
Pareto optimality
x dominates (is better than) x̃ if F (x) 6= F (x̃)
F (x) ¹ F (x̃)
i.e., x is no worse than x̃ in any objective, and better
in at least one
x0 is Pareto optimal if no x dominates it
t2
achievable specs
F (x0)
t1
specs tighter than F (x0)
roughly, x0 Pareto optimal means F (x0) is on tradeoff
surface (F (x0) ∈ ∂A)
Convex optimization problems
4–20
Pareto problem: find Pareto-optimal x
real (but more vague) engineering problem:
search/explore/characterize tradeoff surface, e.g.:
• ‘can reduce F5 below 0.1, but only at huge cost in
F4 and F2’
• ‘can pretty much minimize F3 independently of
other objectives’
• ‘F1 and F2 tradeoff strongly for F1 ≤ 1, F2 ≤ 2’
t2
strong tradeoff
weak tradeoff
t1
Convex optimization problems
4–21
Scalarization
multicriterion problem with F1, . . . , FN
minimize weighted sum of objectives: choose weights
wi > 0, and solve
X
minimize
wiFi(x)
i
which is the same as
minimize wT t
subject to F (x) ¹ t
(i.e., t ∈ A)
t2
achievable specs
P
i
w
witi = constant
F (x0)
t1
• solution x0 is Pareto optimal
• for many cvx problems, all Pareto optimal points
can be found this way, as weights vary
Convex optimization problems
4–22
interpretation
• hyperplane wT t = wT F (x0) supports A at F (x0)
• specifications in halfspace
{t | wT t < wT F (x0)}
are unachievable
t2
achievable specs
F (x0)
t1
guaranteed unachievable specs
Convex optimization problems
4–23