Team Optimality Conditions of Differential Decision
Systems with Nonclassical Information Structures
Charalambos D. Charalambous, Themistoklis Charalambous and
Christoforos Hadjicostis
University of Cyprus and Royal Institute of Technology (KTH)
.
.
Presented at 2014 ECC
Strasbourg, France
June 27, 2014
Outline
1
Motivation + Objectives
2
Team Optimality
Stochastic Differential Equations
3
Team & PbP Optimality Conditions
Necessary Conditions
Sufficient Conditions
4
Example
5
Conclusions & Future Work
6
References
Outline
1
Motivation + Objectives
2
Team Optimality
3
Team & PbP Optimality Conditions
4
Example
5
Conclusions & Future Work
6
References
Motivation + Objectives: I
Motivation
Classical theory of optimization: often developed based on centralized
information-classical information structures.
Decentralized decision systems: often consist of multiple observation
posts and decision/control stations-nonclassical information structures.
Information structures: information available as arguments to the
strategies at control stations to implement their actions.
–Applications in
1
Transportation systems, smart grid energy systems, social network
systems;
2
General large scale distributed systems with local decisions.
Motivation + Objectives: II
References
Team Problems+Informations Structures+Dynamic Optimization
Team Theory (Cooperative)
[MR-72]: Marschak and R. Radner, Economic Theory of Teams,
1972.
[KSM82]: J. Krainak, J.L. Speyer, and S.I. Marcus, Static Team
Problems-Part I: Sufficient Conditions and the Exponential Cost
Criterion, IEEE Transactions on Automatic Control, pp. 839–848,
1982.
[WS]: P.R. Wall and J.H. van Schuppen, A class of Team Problems
with Discrete Action Spaces: Optimality Conditions Based on
Multimodularity, SIAM Journal on Control and Optimization, pp.
875-892, 2000.
Motivation + Objectives: III
Information Structures
[W68]: H.S. Witsenhausen, A Counterexample in Stochastic
Optimum Control, SIAM Journal on Control and Optimization,
pp.131-147, 1968.
[W71]: H.S. Witsenhausen, Separation of Estimation and Control for
Discrete Time Systems, Proceedings of the IEEE, pp.1557-1566,
1971.
Recent Esseys
Jan H. van Schuppen and Tiziano Villa (editors), Coordination
Control of Distributed Systems, Springer, 2014.
Motivation + Objectives: IV
Distributed System + Decentralized Decisions
1
Interconnected Systems
2
Decentralized decisions + Nonclassical information structures
Example of distributed system with decentralized control
SYS1
SYS2
C1
C2
…
SYSn
Cn
Motivation + Objectives: V
Objectives
How do we optimize?
1
Team optimality versus person-by-person optimality
2
Necessary and sufficient conditions of optimality
3
Existence of regular or randomized strategies
Do our methods apply to other game criteria?
1
Non-cooperative games (Nash-equilibrium)
2
Minimax games
Outline
1
Motivation + Objectives
2
Team Optimality
3
Team & PbP Optimality Conditions
4
Example
5
Conclusions & Future Work
6
References
Team Optimality: Formulation I
Differential System- x i ∈ Rni , u i ∈ Rdi
ẋ i (t) = f i (t, x i (t), uti )
X
j|N |
f ij (t, x j1 (t), . . . , x j|Ni | (t), utj1 , . . . , ut i ),
+
(1)
j∈Ni
4
x i (0) = x0i , t ∈ (0, T ], i ∈ ZN = {1, 2, . . . , N}
4
Ni = {j1 , j2 , . . . , j|Ni | } : neighbors of subsystem i
Observations- y i ∈ Rki
4
y i (t) =hi (t, x j1 , . . . , x j|Ki | )
≡hi (t, {x ji (s), . . . , x j|Ki | (s) : 0 ≤ s ≤ t}),
i ∈ ZN
(2)
Team Optimality: Formulation II
Team Pay-Off
J(u) ≡J(u 1 , u 2 , . . . , u N )
Z T
4
=
`(t, x 1 (t), . . . , x N (t), ut1 , . . . , utN )dt + Φ(x 1 (T ), . . . , x N (T )).
0
(3)
Compact Representation
4
u = Vector {u 1 , . . . , u N },
4
h = Vector {h1 , . . . , hN },
x = Vector {x 1 , . . . , x N },
y = Vector {y 1 , . . . , u N },
4
4
ẋ(t) =f (t, x(t), ut ), t ∈ (0, T ],
y (t) =h(t, x),
t ∈ [0, T ].
(4)
(5)
Team Optimality: Formulation III
Preliminaries
L ⊥
H=M
M : direct sum representation of a Hilbert space H;
PM (x) : orthogonal projection of a Hilbert space element x ∈ H onto
the closed subspace M ⊂ H.
n
4
C ([0, T ], Rn ) = continuous functions φ : [0, T ] −→ Rn :
o
n
supt∈[0,T ] |φ(t)|R < ∞ ;
n
4
4
B ∞ ([0, T ], Rn ) = measurable functions φ : [0, T ] −→ Rn : ||φ||2 =
o
supt∈[0,T ] |φ(t)|2Rn < ∞ ;
n
o
R
4
L2 ([0, T ], Rn ) = φ : [0, T ] −→ Rn : [0,T ] |φ(t)|2Rn dt < ∞ ;
n
4
L2 ([0, T ], L(Rm , Rn )) = Σ : [0, T ] −→ Rn×m :
o
R
4 R
2
∗ (t)Σ(t))dt < ∞ .
|Σ(t)|
dt
=
tr
(Σ
L(Rm ,Rn )
[0,T ]
[0,T ]
Team Optimality: Formulation IV
Decentralized Strategies
4
i
U [0, T ] = u i ∈ L2 ([0, T ], Rdi ) : uti ∈ Ai ⊂ Rdi , t ∈ [0, T ],
i
i
ut is nonanticipative measurable w.r.t.{y (s) : 0 ≤ s ≤ t} , ∀i ∈ ZN
Ai : closed, bounded and convex, ∀i ∈ ZN
4
i
U(N) [0, T ] = ×N
i=1 U [0, T ]
1
Open Loop, if uti = µi (t), t ∈ [0, T ], where µi : [0, T ] −→ Ai ;
2
Closed Loop Feedback, if uti = µi (t, y i ) are nonanticipative
functionals of the observation trajectory y i (·), for t ∈ [0, T ];
3
Closed Loop Memoryless, if uti = µi (t, y i (t)), for t ∈ [0, T ].
Team Optimality: Formulation V
Problem
Team Optimality. Find a u o ∈ U(N) [0, T ] which satisfies
J(u 1,o , u 2,o , . . . , u N,o ) ≤ J(u 1 , u 2 , . . . , u N ),
∀u ∈ U(N) [0, T ]
(6)
Person-by-Person (PbP) Optimality. Find a u o ∈ U(N) [0, T ] which
satisfies
J̃(u i,o , u −i,o ) ≤ J̃(u i , u −i,o ),
4
∀u i ∈ Ui [0, T ], i ∈ ZN
J̃(v , u −i ) = J(u 1 , u 2 , . . . , u i−1 , v , u i+1 , . . . , u N )
(7)
Outline
1
Motivation + Objectives
2
Team Optimality
3
Team & PbP Optimality Conditions
4
Example
5
Conclusions & Future Work
6
References
Team Optimality Conditions I
Methodology
weak variations;
convexity condition (can be replaced by randomized strategies).
Assumptions (A)
f is a Borel measurable map f : [0, T ] × Rn × A(N) −→ Rn ;
There exists a K ∈ L2,+ ([0, T ], R) such that
(A1) |f (t, x, u) − f (t, y , u)|Rn ≤ K (t)|x − y |Rn uniformly in u ∈ A(N) ;
(A2) |f (t, x, u) − f (t, x, v )|Rn ≤ K (t)|u − v |Rd uniformly in x ∈ Rn ;
(A3) |f (t, x, u)|Rn ≤ K (t)(1 + |x|Rn ) uniformly in u ∈ A(N) ;
(A4) For any x, x̃ ∈ C ([0, T ], Rn ),
|hi (t, x) − hi (t, x̃)|Rki ≤ K |x − x̃|C ([0,T ],Rn ) , K > 0, i = 1, . . . , N.
Team Optimality Conditions II
Lemma 1
Suppose Assumptions (A) hold. Then for any u ∈ U(N) [0, T ], the
following hold.
1) The differential system has a unique solution x ∈ B ∞ ([0, T ], Rn ) which
is continuous x ∈ C ([0, T ], Rn );
2) The solutions are continuously dependent on th strategies, in the sense
that, as u i,α −→ u i,o in Ui [0, T ], ∀i ∈ ZN , x α −→ x o in B ∞ ([0, T ], Rn ).
Team Optimality Conditions III
Assumptions (B)
(B1) The map f : [0, T ] × Rn × A(N) −→ Rn is continuous in
(t, x, u) and continously differentiable with respect to x, u;
(B2) {fx , fu } are bounded uniformly on [0, T ] × Rn × A(N) ;
(B3) The maps ` : [0, T ] × Rn × A(N) −→ (−∞, ∞] is Borel
measurable, continuously differentiable with respect to (x, u), the
map ϕ : [0, T ] × Rn −→ (−∞, ∞] is continuously differentiable with
respect to x, `(t, 0, 0) is bounded, and there exist K1 , K2 > 0 such
that
|`x (t, x, u)|Rn + |`u (t, x, u)|Rd ≤ K1 1 + |x|Rn + |u|Rd ,
|ϕx (x)|Rn ≤ K2 1 + |x|Rn ;
(B4) |hi (t, x)|Rki ≤ K sup0≤s≤t 1 + |x(s)|2Rn , ∀t ∈ [0, T ], x ∈
C ([0, T ], Rn ), i = 1, . . . , N.
Team Optimality Conditions IV
Hamiltonian System
Hamiltonian:
H :[0, T ] × Rn × Rn × A(N) −→ R,
4
H(t, x, ψ, u) = hf (t, x, u), ψi + `(t, x, u).
(8)
Adjoint ψ ∈ L2 ([0, T ], Rn ):
ψ̇(t) = − fx∗ (t, x(t), ut )ψ(t) − `x (t, x(t), ut )
= − Hx (t, x(t), ψ(t), ut ), t ∈ [0, T ),
ψ(T ) =ϕx (x(T )).
(9)
(10)
State:
ẋ(t) =f (t, x(t), ut ) = Hψ (t, x(t), ψ(t), ut ), t ∈ (0, T ],
(11)
x(0) =x0 .
(12)
Team Optimality Conditions V
Theorem 1. (Necessary conditions of team optimality, ECC2014)
Suppose Assumptions (B) hold, Ai ⊂ Rdi are closed, bounded and
yi
convex, and {y i (s) : 0 ≤ s ≤ t} generates H0,t
-a closed subspace of a
Hilbert space for i = 1, . . . , N.
For u o ∈ U(N) [0, T ] to be team optimal, it is necessary that
(1) There exists a process ψ o ∈ L2 ([0, T ], Rn );
(2) The triple {u o , x o , ψ o } satisfy the inequality:
N Z
X
i=1
T
hHui (t, x o (t), ψ o (t), uto ), uti − uti,o idt ≥ 0, ∀u ∈ U(N) [0, T ]; (13)
0
(3) u o ∈ U(N) [0, T ] satisfies
hP y i Hui (t, x o (t), ψ o (t), uto ) , v i − uti,o i ≥ 0,
H0,t
∀v i ∈ Ai , t ∈ [0, T ], i = 1, 2, . . . , N.
(14)
Team Optimality Conditions VI
Theorem: Sufficient Conditions for Team Optimality
Suppose the conditions of Theorem 1 hold.
Let (u o (·), x o (·)) denote any control-state pair and let ψ o (·) the
corresponding adjoint variable.
Suppose the following conditions hold.
(C1) H(t, ·, x, u), t ∈ [0, T ] is convex in (x, u) ∈ Rn × A(N) ;
(C2) ϕ(·) is convex in x ∈ Rn .
Then
(x o (·), u o (·)) is a team optimal pair if it satisfies the conditional
Hamiltonian;
PbP optimality implies team optimality.
Outline
1
Motivation + Objectives
2
Team Optimality
3
Team & PbP Optimality Conditions
4
Example
5
Conclusions & Future Work
6
References
Example-Linear Quadratic Form I
f (t, x, u) =A(t)x + b(t) + B(t)u,
1
1
`(t, x) = hu, R(t)ui + hx, H(t)xi + hx, F (t)i + hu, E (t)xi + hu, m(t)i,
2
2
1
ϕ(x) = hx, M(T )xi + hx, N(T )i,
2
The projected Hamiltonians give optimal strategies:
N
n
X
uti,o = −Rii−1 (t) mi (t) +
Eij (t)P
j=1
+
N
X
j=1,j6=i
Rij (t)P
i
y
H0,t
yi
H0,t
x j,o (t)
utj,o + B (i),∗ (t)P
i
y
H0,t
o
ψ o (t) , i = 1, 2, . . . , N.
Outline
1
Motivation + Objectives
2
Team Optimality
3
Team & PbP Optimality Conditions
4
Example
5
Conclusions & Future Work
6
References
Conclusions & Future Work I
Conclusions.
Maximum principle + orthogonal projections apply to deterministic
decentralized optimization.
General constraints, i.e., state, control, integral, etc., can be handled.
Future Work
Compute examples.
Develop discrete-time & minimax-noncooperative games.
Recent work on decentralized stochastic decision systems
Charalambous-Ahmed-CDC: 2013,
Charalambous-Ahmed-MTNS:2014, Arxiv.
Outline
1
Motivation + Objectives
2
Team Optimality
3
Team & PbP Optimality Conditions
4
Example
5
Conclusions & Future Work
6
References
References I
Strong formulation + Examples and Applications
1
2
3
N. U. Ahmed and C. D. Charalambous, Stochastic Minimum Principle
for Partially Observed Systems Subject to Continuous and Jump
Diffusion Processes and Driven by Relaxed Controls, SIAM Journal on
Control and Optimization, pp.3235-3257, 2013.
C. D. Charalambous and N. U. Ahmed, Centralized Versus
Decentralized Team Optimality of Distributed Stochastic Differential
Decision Systems with Noiseless Information Structures-Part II:
Applications, IEEE Transactions on Automatic Control (submitted),
2013, http://arxiv.org/abs/1302.3416.
C. D. Charalambous and N. U. Ahmed , Team Optimality Conditions of
Distributed Stochastic Differential Decision Systems with Decentralized
Noisy Information Structures, IEEE Transactions on Automatic Control
(submitted, 2013), http://arxiv.org/abs/1304.3246.
References II
Weak Girsanov’s formulation + examples and applications
1
2
C. D. Charalambous, Dynamic Team Theory of Stochastic Differential
Decision Systems with Decentralized Noisy Information Structures via
Girsanov’s Measure Transformation, MCSS (submitted), 2013,
http://arxiv.org/abs/1309.1913.
C. D. Charalambous, Dynamic Team Theory of Stochastic Differential
Decision Systems with Decentralized Noiseless Feedback Information
Structures via Girsanov’s Measure Transformation, MCSS (submitted),
2013, http://arxiv.org/abs/1310.1488.