Online convex optimization
Gradient descent without a gradient
Abie Flaxman CMU
Adam Tauman Kalai TTI
Brendan McMahan CMU
Standard convex optimization
Convex feasible set S ½ <d
Concave function f : S ! <
Goal: find x
f(x) ¸ maxz2Sf(z) – 
}
Steepest ascent
•
•
•
•
Move in the direction of steepest ascent
Compute f’(x) (rf(x) in higher dimensions)
Works for convex optimization
(and many other problems)
x1 x2x3x4
Typical application
• Toyota produces certain numbers of cars
per month
• Vector x 2 <d (#Corollas, #Camrys, …)
• Profit of Toyota is concave function of
production vector
• Maximize total (eq. average) profit
PROBLEMS
Problem definition and results
• Sequence of unknown (concave) functions
• On tth month, choose production xt 2 S ½ <d
8 z, z’ 2 S
t
t
• Receive profit f (x )
|| z – z’ || · D
• Maximize avg. profit
||rf(z)|| · G
• No assumption on distribution of ft
First try
Zinkevich ’03:
PROFIT
f4(x4)
f3(x3)
f2(x2)
f4
f1(x1) f3
f2 If we could only compute gradients…
f1
x4 x3 x2
x1
#CAMRYS
Idea: 1-point gradient estimate
With probability ½, estimate = f(x + )/
PROFIT
With probability ½, estimate = –f(x – )/
E[ estimate ] ¼ f’(x)
x-x x+
#CAMRYS
PROFIT
Analysis
In expectation, gradient ascent on
x- x+
#CAMRYS
For online optimization, use Zinkevich’s
analysis of online gradient ascent [Z03]
d-dimensional online algorithm
• Choose u 2 <d, ||u||=1
• Choose xt+1 = xt +  u ft(xt+u)/
• Repeat
x3
x4
x1
S
x2
Hidden complication…
Dealing with steps outside set is difficult!
S
Hidden complication…
Dealing with steps outside set is difficult!
S’
…reshape into
Isotropic position
Related work & conclusions
• Related work
– Online convex gradient descent [Zinkevich03]
– One point gradient estimates [Spall97,Granichin89]
– Same exact problem [Kleinberg04] (different solution)
• Conclusions
– Can estimate gradient of a function from single
evaluation (using randomness)
– Adaptive adversary ft(xt)=ft(x1,x2,…,xt)
– Useful for situations with a sequence of different
functions, no gradient info, one evaluation per
function, and others