A Unified Distributed Algorithm for Non-Games
Non-cooperative, Non-convex, and Non-differentiable
Jong-Shi Pang∗ and Meisam Razaviyayn†
presented at
Workshop on Optimization for Modern Computation
Peking University, Beijing, China
10:10 – 10:45 PM, Thursday September 04, 2014
∗ Department
of Industrial and Systems Engineering, University of Southern
California, Los Angeles
† Visiting
Research Assistant, Department of Electrical and Computer Engineering, University of Minnesota, Minneapolis
1
The Non-cooperative Game G
• An n-player non-cooperative game G wherein each player i = 1, · · · , n, ann
Y
n
X j , solves the
ticipating the rivals’ strategy tuple x−i , xj i6=j=1 ∈ X −i ,
i6=j=1
optimization problem:
i
−i
minimize
θ
(x
,
x
)
i
i
i
x ∈X
• X i ⊆ Rni is a closed convex set
• θi : Ω → R is a locally Lipschitz continuous and directionally differentiable
n
Y
function defined on Ω ,
Ωi where each Ω i is an open convex set containing
i=1
X
i
• A key structural assumption for convergence of distributed algorithm:
each θi (x) = fi (x)+gi (xi ), with fi (x), dependent on all players’ strategy profile
n
i
x , x i=1 , being twice continuously differentiable but not necessarily convex,
and gi (xi ), dependent on player i’s strategy profile xi only is convex but not
necessarily differentiable.
2
Quasi-Nash equilibrium: Definition and existence
Definition. A player profile
x∗
,
n
∗,i
x i=1
is a QNE if for every i = 1, · · · , n:
θi (•, x∗,−i ) 0 (x∗,i ; xi − x∗,i ) ≥ 0, ∀ xi ∈ X i .
Existence. Suppose each θi (x) = fi (x) + gi (x) with ∇xi fi continuously differentiable on Ω and gi (•, x−i ) convex on X i that is compact and convex.
Proof by a fixed-point argument applied to the map:
n
Y
n
i
X i 7→ Φ(x) , (Φi (x))ni=1 ∈ X , where, for i = 1, · · · , n,
Φ : x , x i=1 ∈ X ,
i=1
Φi (x) , argmin
z i ∈X i
h
i
α i
i 2
fi (z , x ) + gi (z , x ) + k z − x k ,
2
i
−i
i
−i
with α > 0 such that the minimand is strongly convex in z i for fixed x−i .
Remark. For existence, gi (x) can be fully dependent on the player profile x;
but for convergence of distributed algorithm, gi (xi ) is only player dependent.
3
The unified algorithm
The main idea:
• Employing player-convex surrogate objective functions and the information
from the most current iterate, non-overlapping groups of players update, in
parallel, their strategies from the solution of sub-games.
• Thus the algorithm is a mixture of the classical block Gauss-Seidel and
Jacobi iterations, applied in a way consistent with the game-theoretic setting
of the problem.
Two key families:
• The player groups: σ ν , σ1ν , · · · , σκνν consists of κν pairwise disjoint subsets
of the players’ labels, for some integer κν > 0. Players in each group σkν solve
a sub-game; all such sub-games in iteration ν are solved in parallel.
Nν ,
κν
[
σkν not necessarily equal to {1, · · · , n};
k=1
i.e., some players may not update in an iteration.
4
xν
∈ X , the bivariate surrogate objectives:
κν
.
in lieu of the original objectives {θi }i∈σkν k=1
• Given
n
ν
σkν
b
θi (xσk ; xν )
: i∈
σkν
oκν
k=1
σkν
Gν
for k = 1, · · · κν : the optimization problems of
• The subgames, denoted
ν
the players in σk are



















ν
ν


σ
i
σ
;−i
ν
k
x ,x k
minimize
θbi 
;
x
.

|{z}
i
i
|
{z
}
x
∈X







 subgame variables

input
to
subgame




ν


σ
at iteration ν
x k
i∈σ ν
k
• The new iterate for a step size τσkν ∈ (0, 1]


ν;σkν
ν;σkν 
−
x
x
b
.
| {z }
solution to subgame
• Need directional derivative consistency at limit x∞ of generated sequence:
ν
ν
xν+1;σk , xν;σk + τσkν 
t
t
σ
θi (•, x∞,−i ) 0 (x∞,i ; xi − x∞,i ) ≥ θbi k (•, x∞,σk ;−i ; x∞ ) 0 (x∞,i ; xi − x∞,i ), ∀ xi ∈ X i .
5
An illustration. A 10-player game with the grouping:
σ ν = { {1, 2}, {3, 4, 5}, {6, 7, 8, 9} }
so that κν = 3 and N ν = {1, · · · , 9}, leaving out the 10th-player.
{1,2}
Players 1 and 2 update their strategies by solving a subgame Gν
{1,2}
{1,2}
by the surrogate objective functions θb1
(•; xν ) and θb2
(•; xν ).
defined
In parallel, players 3, 4, and 5 update their strategies by solving a subgame
{3,4,5}
Gν
using the surrogate objective functions
n
o
{3,4,5}
{3,4,5}
{3,4,5}
ν
ν
ν
θb3
(•; x ), θb4
(•; x ), θb5
(•; x ) ;
similarly for players 6 through 9.
The 10th player is not performing an update in the current iteration ν according to the given grouping.
6
Special cases: player groups
• Block Jacobi N ν = {1, · · · , n} and σkν may contains multiple elements.
• Point Jacobi κν = n; thus σkν = {k} for k = 1, · · · n: each player i solves an
optimization problem:
minimize θbi (xi ; xν ).
xi ∈X i
• Block Gauss-Seidel κν = 1 for all ν: only the players in the block σ1ν update
their strategies that immediately become the inputs to the new iterate xν+1
while all other players j 6∈ σ1ν keep their strategies at the current iterate xν,j .
• Point Gauss-Seidel κν = 1 and σ1ν is a singleton.
• Above are deterministic player groups; also consider randomized player
groups: Let {σ1 , · · · σK } be a partition of {1, · · · , n}. At iteration ν, the subset
σ ν ⊆ {σ1, . . . , σK } of player groups is chosen randomly and independently from
the previous iterations, so that
Pr(σi ∈ σ ν ) = pσi > 0,
There is a positive probability pσi , same at all iterations ν, for the subset σi
of players to be chosen to update their strategies.
7
Special cases: surrogate objectives
• Standard convex case Suppose θi (•, x−i ) is convex. For i ∈ σkν , let
αi i
ν
ν
ν
σν
θbi k (xσk ; z) , θi (xσk , z −σk ) +
k x − z i k2 , for some positive scalar αi
|2
{z
}
regularization
• Mixed convexity and differentiability Suppose θi (•, x−i ) = gi (•, x−i )+fi (•, x−i ),
where gi (•, x−i ) is convex and fi (•, x−i ) is differentiable. Let
X
αi
σkν
σkν
σkν
−σkν
b
θi (x ; z) , gi (x , z ) + fi (z) +
∇z j fi (z)T ( xj − z j ) + k xi − z i k2
2
j∈σkν
{z
}
|
partial linearization
|
{z
}
ν
convex in xσk for fixed z
• Newton-type quadratic approximation Suppose ∇xi θi (•, x−i ) exists. Let
X
X
0
0
ν
0
σkν
σkν
T
j
j
1
b
( xj − z j )T B σk ;j,j ( xj − z j ),
θi (x ; z) , θi (z) +
∇xj θi (z) ( x − z ) + 2
j∈σkν
|
ν
0
j,j 0 ∈σkν
{z
ν
quadratic in xσk for fixed z
}
0
ν
B σk ;j,j approximates mixed partial derivatives of θi (•, z −σk ) w.r.t. xj and xj .
8
Convergence analysis
Two approaches
• Contraction — showing that the sequence {xν }∞
ν=1 contracts in the vector
sense by means of the assumption of a spectral radius condition of a key
matrix
• Potential — relying on the existence of a potential function that decreases
at each iteration.
Think about a system of linear equations: Ax = b
• (Generalized) diagonal dominance yields convergence under contraction.
• Symmetry of A yields the potential function: P (x) , 12 xT Ax − bT x.
9
Contraction approach
T
t t
b , σ1, · · · , σκt t t=1 of index subsets
• An integer T > 0 and a fixed family σ
of the players’ labels that partitions {1, · · · , n}.
• Families of bivariate surrogate functions
n t
oκt
t
σ
t
b = θb k : i ∈ σ
θ
, for t = 1, · · · , T ,
k
i
k=1
t
t
t
σ
such that for every pair (xσk ;−i ; z), the function θbi k (•, xσk ;−i ; z) is convex.
σt
• For each set σkt , let Gt k denote the subgame consisting of the players i ∈ σkt
σt
with objective functions θbi k (•; z) for certain (known) iterate z to be specified.
• Let κν = κt and σkν = σkt for ν ≡ t modulo T and for all k = 1, · · · , κν ; thus,
σν
σt
for each i = 1, · · · , n, θbi k = θbi k where ν ≡ t modulo T and σkt is the unique
index set containing i.
• Thus, each player i and the members in σkt will update their strategy tuple
σt
exactly once every T iterations through the solution of the subgame Gt k .
• Finally, we take each step size τσkν = 1.
10
A further illustration
b 1 = {{1, 2}, {3, 5, 6}},
Consider a 12-player game with T = 3 and with σ
b 2 = {{4, 7, 8}}, and σ
b 3 = {{9}, {10, 11, 12}}.
σ
12
0
0;i
Starting with x = x i=1 = x(0) , we obtain after one iteration
1
1;{1,2}
1;{3,5,6}
0;4
0;{7thru12}
x = x
,x
,x ,x
.
The sub-vectors x1;{1,2} and x1;{3,5,6} mean that the players 1 and 2 update
their strategies by solving a 2-player subgame and simultaneously the players
3, 5, and 6 update their strategies by solving a 3-player subgame. The
remaining players 4, 7 through 12 do not update their strategy in this first
iteration.
The next two iterations yield, respectively,
x2 =
x1;{1,2} , x1;{3,5,6} , x2;{4,7,8} , x0;{9thru12}
x3
x1;{1,2} , x1;{3,5,6} , x2;{4,7,8} , x3;{9} , x3;{10,11,12}
=
.
{4,7,8}
The update of x2 employs x1 in defining the player objectives θb4
(•; x1 ),
{4,7,8}
{4,7,8}
θb7
(•; x1 ), and θb8
(•; x1 ). Similarly, the update of x3 employs x2 .
11
After three iterations, we have completed a full cycle where all players have
updated their strategies exactly once, obtaining the new iterate x(1) = x3 .
The
next cycle
of updates is then initiated according to the same partition
o
n
b 1, σ
b 2, σ
b 3 and employs the same family of bivariate surrogate functions.
σ
• group 1:
{1,2} b{1,2}
{3,5,6} b{3,5,6} b{3,5,6}
θb1
, θ2
, θ6
, θ5
;
, θb3
|
|
{z
}
{z
}
2-person subgame 3-person subgame
{z
}
|
2 subgames solve in parallel
{4,7,8} b{4,7,8} b{4,7,8}
, θ8
, θ7
• group 2: θb4
;
|
{z
}
3-person subgame
• group 3:
parallel
single game
{10,11,12} b{10,11,12} b{10,11,12} parallel:
{9}
, θb10
, θ11
, θ12
θb9
|
|{z}
{z
} single opt. + game
single-player opt
3-person subgame
|
{z
}
2 subgames solved in parallel
sequential
sequential
group 1 − − − − − − − − − > group 2: − − − − − − − − − > group 3
12
Set-up for assumptions
t
t
t
t
σ
σ
• Assume θbi k (xσk ; z) = gi (xi )+ fbi k (xσk ; z), where gi is convex and the (surrogate
σt
objective)fbi k (•; z) is twice continuously differentiable.
σt
t
t
t
> 0 such that
• fbi k (xσk ; z) is strongly convex in xσk uniformly in z; i.e., ∃ γk;ii
t
t
for all xi ∈ X i , all uσk ∈ X σk and all z ∈ X ,
t
i
i T
2 bσk
σkt
i
i
t
x −u
∇ui ui fi (u ; z) x − u ≥ γk;ii
k xi − ui k2 .
t
σ
• Further assume that each function ∇ui fbi k is continuously differentiable in
both arguments with bounded derivatives. Let
t
t
σ
σ
2
t
,
sup
γk;ij
∇uj ui fbi k (u k ; z) < ∞, ∀ i 6= j in σkt
σt
σt
u k ∈X k ; z∈X 2 bσkt σkt
t
γ
ek;i` ,
sup
∇z`ui fi (u ; z) < ∞, ∀ i ∈ σkt and ` = 1, · · · , n.
σt
σt
u k ∈X k ; z∈X
Let Γ , blkdiag Γ
h
e, Γ
e ts
Let Γ
iT
t,s=1
t T
t=1
t κt
, where each Γ t , blkdiag Γk
h
e ts , Γ
e ts 0
, where each Γ
k,k
i(κt,κs)
0
(k,k )=(1,1)
k=1
h
t
and Γkt , γk;ij
e ts 0 ,
with Γ
k,k
h
t
γ
ek;ij
i
.
i,j∈σkt
ij∈σks 0
i∈σkt
13
.
h
i
t T
t
h
, where each Γ , blkdiag
The comparison matrix: Γ , blkdiag Γ
t=1
t
i
h
if i = j
γk;ii
t
t
t
t.
,
where
Γ
for
i,
j
∈
σ
,
and Γk , γ k;ij
t
k
k
−γk;ij otherwise
i,j∈σkt
ij
i
t κt
Γk
k=1
e , which has all off-diagonal entries nonKey assumption: The matrix Γ − Γ
positive (thus a Z-matrix), is also a P-matrix (thus a Minkowski matrix).
e=L
e+D
e +U
e as the sum of the strictly lower triangular, diagonal,
Writing Γ
and strictly upper triangular parts, respectively, we have
e is invertible and has a nonnegative inverse,
• Γ−L
h
i −1 e
e
e
• the spectral radius of the (nonnegative) matrix Γ − L
D + U is less
than unity, or equivalently,
t
t
• ∃ positive scalars dk;ij
and dek;i`
such that
t
t
γk;ii
dk;ii
>
X
i6=j∈σkt
t
t
γk;ij
dk;ij
+
n
X
t
t
γ
ek;i`
dek;i`
, ∀ t = 1, · · · , T, k = 1, · · · , κt , i ∈ σkt .
`=1
14
Potential Games
Definition. A family of functions {θi (x)}ni=1 on the set X admits
• an exact potential function P : Ω → R if P is continuous such that for all
i, all x−i ∈ Ω−i , and all y i and z i ∈ Ωi ,
P (y i , x−i ) − P (z i , x−i ) = θi (y i , x−i ) − θi (z i , x−i );
• a generalized potential function P : Ω → R if P is continuous such that
for all i, all x−i ∈ Ω−i , and all y i and z i ∈ Ωi ,
θi (y i , x−i ) > θi (z i , x−i ) ⇒ P (y i , x−i ) − P (z i , x−i ) ≥ ξi (θi (y i , x−i ) − θi (z i , x−i )),
for some forcing functions ξi : R+ → R+ , i.e., lim ξi (tν ) = 0 ⇒ lim tν = 0.
ν→∞
ν→∞
Example Generalized ; exact:
minimize
x1 ∈R
θ1 (x1 , x2 ) , x1 | minimize
subject to −2 ≤ x1 ≤ 2
x2 ∈R
θ2 (x1 , x2 ) , x1 x2 + x2
| subject to 1 ≤ x2 ≤ 3.
Generalized potential function: P (x1 , x2 ) = x1 x2 + x2 .
The potential function, if it exists, is employed to gauge the progress of the
algorithm.
15
How to recognize the existence of a potential?
The convex case. Suppose that θi (•, x−i ) is convex. Recalling its subdifferential, ∂xi θi (•, x−i ), we define the multifunction
Θ(x) ,
n
Y
∂xi θi (x), x ∈ X .
i=1
Among the following four statements, it holds that (a) ⇔ (b) ⇒ (c) ⇔ (d):
(a) Θ(x) is maximally cyclically monotone on Ω ,
n
Y
Ω i;
i=1
(b) ∃ a convex function ψ(x) such that ∂ψ(x) = Θ(x) for all x ∈ Ω;
(c) ∃ a convex function ψ(x) on Ω and continuous functions Ai (x−i ) on Ω −i
such that θi (x) = ψ(x) + Ai (x−i ) for all x ∈ Ω and all i = 1, · · · , n;
(d) the family {θi (x)}ni=1 admits a convex exact potential function P (x).
If ∇xi θi (x) is differentiable, the existence of a (differentiable) potential is
related to the symmetry of the Jacobian of the vector function (∇xi θi (x))ni=1 .
16
Player selection rule is
• Essentially covering if ∃ an integer T ≥ 1 such that
N ν ∪ N ν+1 ∪ . . . ∪ N ν+T −1 = { 1, 2, . . . , n }, ∀ ν = 1, 2, . . . ,
so that within every T iterations, all players will have updated their strategies
at least once.
[Unlike partitioning, the above index sets may overlap, resulting in some players updating their strategies more than once during these T iterations. ]
• Randomized if the players are chosen randomly, identically, and independently from the previous iterations so that
Pr(j ∈ N ν ) = pj ≥ pmin > 0, ∀ j = 1, 2, . . . , n, ∀ν = 1, 2, . . . .
Postulates on objectives and their surrogates:
• Each θi (x) = fi (x) + gi (xi ) for some differentiable function fi and convex
function gi .
n ν
o
ν
ν
σkν
σkν
σk
σ
i
σ
ν
b
b
b
• Correspondingly, θi (x k ; z) = gi (x )+fi (x k ; z), where the family fi (•; x )
i∈σkν
admits an exact potential function fbσkν (•; xν ) satisfying
17
• Strong convexity: there exists a constant η > 0 such that
η
ν
ν
ν ν
ν
ν
ν
e σk − xσk + k x
e σk ; y) ≥ fbσkν (xσk ; y) + ∇xσkν fbσkν (xσk ; y)T x
fbσkν (x
e σk − xσk k2
2
ν
ν
for all x, x
eσk ∈ X σk , and y in X .
T
ν
σkν
σ
;−i
T
i
i
b
(ui − xi )
• Gradient consistency: ∇xi fi (x) (u − x ) = ∇xi fi (•, x k ; x)|xi
for all ui , xi ∈ X i , x−i ∈ X −i and i ∈ σkν .
18
Convergence with constant step-size. Assume
• an exact potential function P exists;
• a scalar L > 0 exists such that k∇fi (x) − ∇fi (x0 )k ≤ Lkx − x0 k for all x, x0 ∈ X
and all i = 1, · · · , n;
• a constant step-size τ ∈ (0, 2η/L) is employed.
Then, for an essentially covering player selection rule, every limit point of the
iterates generated by the unified algorithm is a QNE of the game G. Same
holds with probability one for the randomized player selection rule.
Generalized potential games: 2 more restrictions:
• Point Gauss-Seidel, i.e., each σkν is a singleton;
• Tight upper-bound assumption:
ν
ν
ν
ν
ν
ν
θbσν (xσ ; y) ≥ θσν (xσ ; y −σ ) and θbσν (xσ ; x) = θσν (xσ ; x−σ ),
∀x, y ∈ X .
19
Concluding remarks
• We have introduced and analyzed the convergence of a unified distributed
algorithm for computing a QNE of a multi-player game with non-smooth,
non-convex player objective functions and with decoupled convex constraints.
• The algorithm employs a family of surrogate objective functions to deal with
the non-convexity and non-differentiability of the original objective functions
and solves subgames in parallel involving deterministic or randomized choice
of non-overlapping groups of players.
• The convergence analysis is based on two approaches: contraction and
potential; the former relies on a spectral condition while the latter assumes
the existence of a potential function.
• Extension of the algorithm and analysis to games with coupled convex constraints can be done by introducing multipliers (or prices) of such constraints
that are updated in an outer iteration.
• Non-convex constraints are presently being researched.
Thank you!
20