Distributed Optimization
Yen-Ling Kuo
Der-Yeuan Yu
May 27, 2010
Outline [Yu]
• Optimized Sensing: From Water to the Web
• Distributed Dynamic Programming
• Distributed Solutions to Markov Decision
Problems
Optimized Sensing
• Problem Statement
• Greedy Algorithms and Submodularity
• Robust Sensing Optimization with Saturate
Algorithm
• Application in Blogs
Problem Statement
• How do we detect contamination in drinking
water distribution networks?
• Which blogs should we read to learn about
the biggest, newest stories on the Web?
• Fundamental Question: How can we get the
most useful information at minimum cost
(limited resources)?
Solutions to Optimized Sensing
• Covers fields of statistics, machine learning,
sensor networks, and robotics
• With partially observable Marko decision
processes, we can get optimal solutions
• But it is difficult to scale POMDP to large
problems
• Introducing a new algorithm based on
submodularity
Formulation
• Sensing quality function F(A)
– A: the set of sensor locations Si (i=1~k)
– V: the set of all locations
• We can also have cost constraints
– Total cost of sensor deployment no greater than
the budget
• Goal: Find A*
– This is NP-hard already
Greedy Algorithm
• Iteratively find Si
• This naïve algorithm actually performs pretty
well
– Why? Submodularity
– We get near-optimal solutions
• Submodularity: diminishing returns
Diminishing Returns
Cost-Effective Lazy Forward-Selection
(CELP)
• Greedy algorithm
• Lazy evaluations
– Delaying computation until the result is required
– A computational technique
Robust Sensing Optimization
• Idea: Protect system against adversaries that know of
our deployment of sensors
• Goal: Maximize the worst-case detection
performance
• Approach
• Unfortunately, this naïve extension can fail
Failure of Greedy Algorithm on WorstCase Scenarios
• I1, I2: two contamination events
• S1, S2, S3: three possible sensor locations
– S1: detect I1 immediately, but never I2
– S2: detect I2 immediately, but never I1
– S3: detect both I1 and I2, but only after a long time
• We can only place two sensors
• Greedy would pick S3 first and then either S1 or S2
• But we know the optimal solution should be S1 and
S2
• Solution? Saturate algorithm
Saturate Algorithm
• Idea: reduce the non-submodular worst-case
objective to a submodular optimization
problem
– Transform non-submodular to submodular
• Transformation
– Guess optimal solution value C using binary search
– Try to find A such that F(A) is no less than C
Performance of Saturate
From Water to the Web
Blog Reading
• Problem: Information cascading
Improvements
• Number-of-posts (NP) model
– Reading a big blog can be time-consuming, so they define
the cost to be the number of posts
• CELP tends to choose blogs with many posts
• NP model tends to choose summarizer blogs
– But stories appear in summarizer blogs a little late
Other Thoughts
• What if we are looking for stories to read
instead of blogs to read?
– We can reverse our information management goal
– Find posts instead of blogs
– Ref. 10
• End of Paper
Distributed Dynamic Programming
for Path Planning
• Asynchronous Dynamic Programming
• Learning Real-Time A*
Asynchronous Dynamic Programming
• Propagate costs from target to start locations
Learning Real-Time A* (LRTA*)
LRTA*(n)
• LRTA with n agents
• Faster
– Agents break ties differently
– They can share the same h-value table
LRTA*(2)
Distributed Solutions to Markov
Decision Problems
• As previously mentioned in the Water to Web
paper, MDPs can be difficult to scale to big
problems
• Solution: Exploit independence properties
• We address the modularity of actions
Action Selection in multiagent MDPs
Implementation
Subtask Distribution
Problem
• A global problem is broken
down into subtasks
• Subtasks are distributed
among agents
• Each agent has different
capabilities
25
Contract Net
• Stages
–
–
–
–
Recognition
Announce
Bidding
Awarding & Expediting
• Initial assignment: Not optimal
• Anytime property
– Improve assignment in negotiation process
26
Assignment problem
• Problem definition
– A set N of n agents
– A set X of n objects
– A set M ⊆ N × X of possible assignment pairs, and
– A function v : M → R
X
N
M
• Find optimal assignment
27
Corresponding Linear Program
• Linear program (LP) formulation
Profit maximization
Resource constraint
Optimal solution
• Any LP can be solved in
polynomial time O(n3)
28
Competitive Equilibrium
• Consider a price vector p = (p1, …, pn)
– The utility from an assignment j to agent i is
u(i, j) = v(I, j) - pj
• A feasible assignment S and a price vector p are in
competitive equilibrium when for every pairing (i, j) ∈
S it is the case that ∀ k, u(i, j) ≥ u(i, k)
Every agent will not change its selection
S is a optimal solution
29
Naïve Auction Algorithm
• Round-robin style
• Bid increment is the difference between the
utility to i of the best and second-best object
The agent will not overbid
30
Problem in Naïve Auction
• When more than one object offers maximal
utility for an agent
– Bid increment is zero
31
Terminating Auction Algorithm
• Modify the bid increment
–
ε-competitive equilibrium: u(i, j) + ε ≥ u(i, k)
Agents may overbid some objects
32
Scheduling Problem
• Problem definition
–
–
–
–
N is a set of n agents
X is a set of m discrete and consecutive time slots
q = (q1, . . . , qm) is a reserve price vector
v = (v1, . . . , vn), where vi is the valuation function of agent I
F
• Find optimal allocation
33
Corresponding Integer Program
• Integer program (IP) formulation
• IPs are not solvable polynomial time
34
Competitive Equilibrium – General Form
• Definition
– For all i ∈ N it is the case that
Fi = argmaxT ⊆ X (vi(T) − ∑j|xj∈T pj)
– For all j such that xj ∈ F∅ it is the case that pj = qj
– For all j such that xj ∈ F∅ it is the case that pj ≥ qj
• May not exist competitive equilibrium
Has a competitive equilibrium solution
↕
The LP relaxation of the associated integer program has a integer solution.
35
Ascending Auction Algorithm
• Center advertise an ask price
• Bid increment is constant
36
Problem in Ascending Auction
• If the increment is too large
• May not converge to optimal solution
37
Social Laws and Conventions
• Social law
– A restriction on the given strategies of the agents
– Induce a sub-game
• Social convention
– The sub-game consists of a single strategy for all agent
• Other topics
– Social goal negotiation
– Social norm negotiation
– ….
38