Interacting
Urns
Neeraja
Sahasrabudhe
Introduction
Background
Problem setup
Synchronization and Fluctuation
Theorems for Interacting Friedman Urns
Synchronization
L2 synchronization
Almost sure
synchronization
Fluctuation
Theorems
Neeraja Sahasrabudhe
Stable
Convergence
Stochastic
Approximation
Indian Institute of Technology, Bombay
Other models
and Future
Directions
Complex Networks and Emerging Applications
28th March 2016
Outline
Interacting
Urns
Neeraja
Sahasrabudhe
Introduction
Background
Problem setup
Synchronization
L2 synchronization
Almost sure
synchronization
Fluctuation
Theorems
Stable
Convergence
Stochastic
Approximation
Other models
and Future
Directions
1 Introduction
Background
Problem setup
2 Synchronization
L2 -synchronization
Almost sure synchronization
3 Fluctuation Theorems
Stable Convergence
Stochastic Approximation
4 Other models and Future Directions
Introduction: Pólya Urn Model
Interacting
Urns
Neeraja
Sahasrabudhe
Introduction
Background
Problem setup
Synchronization
L2 synchronization
Almost sure
synchronization
Fluctuation
Theorems
Stable
Convergence
Stochastic
Approximation
Other models
and Future
Directions
The classical Pólya urn scheme consists of an urn containing x
balls of one colour and y balls of another colour. At time t, one
ball is drawn randomly from the urn and its color observed; it is
then replaced in the urn, along with a ball of the same color.
Interacting
Urns
Neeraja
Sahasrabudhe
Introduction
Background
Problem setup
Synchronization
L2 synchronization
Almost sure
synchronization
Fluctuation
Theorems
Stable
Convergence
Stochastic
Approximation
Other models
and Future
Directions
Initial composition of the urn: (W0 , B0 ). Let Zt denote
the fraction of white balls at time t ≥ 0.
Interacting
Urns
Neeraja
Sahasrabudhe
Introduction
Background
Problem setup
Synchronization
L2 synchronization
Almost sure
synchronization
Fluctuation
Theorems
Stable
Convergence
Stochastic
Approximation
Other models
and Future
Directions
Initial composition of the urn: (W0 , B0 ). Let Zt denote
the fraction of white balls at time t ≥ 0.
Wt+1 = Wt + Yt+1 , where Yt+1 denotes the number of
white balls added to the urn at time t + 1.
Interacting
Urns
Neeraja
Sahasrabudhe
Introduction
Background
Problem setup
Synchronization
L2 synchronization
Almost sure
synchronization
Fluctuation
Theorems
Stable
Convergence
Stochastic
Approximation
Other models
and Future
Directions
Initial composition of the urn: (W0 , B0 ). Let Zt denote
the fraction of white balls at time t ≥ 0.
Wt+1 = Wt + Yt+1 , where Yt+1 denotes the number of
white balls added to the urn at time t + 1.
We have,
Yt+1 =
1 w. p. Zt
0 w. p. 1 − Zt
Interacting
Urns
Neeraja
Sahasrabudhe
Introduction
Background
Problem setup
Synchronization
L2 synchronization
Almost sure
synchronization
Fluctuation
Theorems
Stable
Convergence
Stochastic
Approximation
Other models
and Future
Directions
Initial composition of the urn: (W0 , B0 ). Let Zt denote
the fraction of white balls at time t ≥ 0.
Wt+1 = Wt + Yt+1 , where Yt+1 denotes the number of
white balls added to the urn at time t + 1.
We have,
Yt+1 =
Zt is a Martingale.
1 w. p. Zt
0 w. p. 1 − Zt
Friedman urns
Interacting
Urns
Neeraja
Sahasrabudhe
Introduction
Background
Problem setup
Synchronization
L2 synchronization
Almost sure
synchronization
Fluctuation
Theorems
Stable
Convergence
Stochastic
Approximation
Other models
and Future
Directions
Friedman urn model (proposed by Bernard Friedman in 1949)
is a generalization of a Pólya urn, where the chosen ball is
replaced with α balls of same colour and β balls of the other
colour.
Interacting
Urns
Neeraja
Sahasrabudhe
Introduction
Background
Problem setup
Synchronization
L2 synchronization
Almost sure
synchronization
Fluctuation
Theorems
Stable
Convergence
Stochastic
Approximation
Other models
and Future
Directions
If nth draw is a white ball. Then,
Add: α white and β black balls.
Interacting
Urns
Neeraja
Sahasrabudhe
Introduction
Background
Problem setup
Synchronization
L2 synchronization
Almost sure
synchronization
Fluctuation
Theorems
Stable
Convergence
Stochastic
Approximation
Other models
and Future
Directions
If nth draw is a white ball. Then,
Add: α white and β black balls.
If
nth
draw is a black ball. Then,
Add: α black and β white balls.
What do we know?
Interacting
Urns
Neeraja
Sahasrabudhe
Introduction
Background
Problem setup
Synchronization
L2 synchronization
Almost sure
synchronization
Fluctuation
Theorems
Stable
Convergence
Stochastic
Approximation
Other models
and Future
Directions
In case of Pólya urns, the fraction of balls of either colour
approaches, with probability 1, a random limit that is
distributed according to a beta distribution.
What do we know?
Interacting
Urns
Neeraja
Sahasrabudhe
Introduction
Background
Problem setup
Synchronization
L2 synchronization
Almost sure
synchronization
Fluctuation
Theorems
Stable
Convergence
Stochastic
Approximation
Other models
and Future
Directions
In case of Pólya urns, the fraction of balls of either colour
approaches, with probability 1, a random limit that is
distributed according to a beta distribution.
In case of Friedman urns, this limit is deterministic and is
equal to 1/2 with probability 1.
Interacting Urns
Interacting
Urns
Neeraja
Sahasrabudhe
Introduction
Background
Problem setup
Synchronization
L2 synchronization
Almost sure
synchronization
Fluctuation
Theorems
Stable
Convergence
Stochastic
Approximation
Other models
and Future
Directions
More than one urn. The reinforcement scheme depends on all
urns or on a non-trivial subset of the given set of urns.
Interacting
Urns
Neeraja
Sahasrabudhe
Introduction
Background
Problem setup
Synchronization
L2 synchronization
Almost sure
synchronization
Fluctuation
Theorems
Stable
Convergence
Stochastic
Approximation
Other models
and Future
Directions
Consider N urns denoted by U(1), . . . , U(N) such that at
time t = 0 each urn contains W0 (i) > 0 white and
B0 (i) > 0 black balls.
Interacting
Urns
Neeraja
Sahasrabudhe
Introduction
Background
Problem setup
Synchronization
L2 synchronization
Almost sure
synchronization
Fluctuation
Theorems
Stable
Convergence
Stochastic
Approximation
Other models
and Future
Directions
Consider N urns denoted by U(1), . . . , U(N) such that at
time t = 0 each urn contains W0 (i) > 0 white and
B0 (i) > 0 black balls.
Let N0 (i) = W0 (i) + B0 (i) denote the total number of
balls in each urn at the beginning and Wt (i) (resp. Bt (i))
denote the number of white balls (resp. black balls) in
U(i) at time t.
Interacting
Urns
Neeraja
Sahasrabudhe
Introduction
Background
Problem setup
Synchronization
L2 synchronization
Almost sure
synchronization
Fluctuation
Theorems
Stable
Convergence
Stochastic
Approximation
Other models
and Future
Directions
Consider N urns denoted by U(1), . . . , U(N) such that at
time t = 0 each urn contains W0 (i) > 0 white and
B0 (i) > 0 black balls.
Let N0 (i) = W0 (i) + B0 (i) denote the total number of
balls in each urn at the beginning and Wt (i) (resp. Bt (i))
denote the number of white balls (resp. black balls) in
U(i) at time t.
At time t + 1,
Wt+1 (i) = Wt (i) + Yt+1 (i)
(1)
where Yt+1 (i) denotes the number of white balls added to
urn U(i) at time t + 1.
Interacting
Urns
Neeraja
Sahasrabudhe
Introduction
Background
Problem setup
Synchronization
L2 synchronization
Almost sure
synchronization
Fluctuation
Theorems
Stable
Convergence
Stochastic
Approximation
Other models
and Future
Directions
Consider N urns denoted by U(1), . . . , U(N) such that at
time t = 0 each urn contains W0 (i) > 0 white and
B0 (i) > 0 black balls.
Let N0 (i) = W0 (i) + B0 (i) denote the total number of
balls in each urn at the beginning and Wt (i) (resp. Bt (i))
denote the number of white balls (resp. black balls) in
U(i) at time t.
At time t + 1,
Wt+1 (i) = Wt (i) + Yt+1 (i)
(1)
where Yt+1 (i) denotes the number of white balls added to
urn U(i) at time t + 1.
We assume that Yt (i) for i = 1, . . . , N are conditionally
independent given the past. We denote the total number
of balls in each urn at time t by Nt (i).
Interacting
Urns
Neeraja
Sahasrabudhe
Introduction
Background
Problem setup
Let Zt (i) denote the fraction of white balls in U(i) at time t
and Zt denote the overall fraction of white balls. That is,
Synchronization
L2 synchronization
Almost sure
synchronization
Fluctuation
Theorems
Stable
Convergence
Stochastic
Approximation
Other models
and Future
Directions
N
Zt
=
=
1 X
Wt (i)
NNt
i=1
1 X
Zt (i)
N
i
Reinforcement scheme:
Interacting
Urns
Neeraja
Sahasrabudhe
Introduction
Background
Problem setup
Synchronization
L2 synchronization
Almost sure
synchronization
Fluctuation
Theorems
Stable
Convergence
Stochastic
Approximation
Other models
and Future
Directions
We consider the following reinforcement model:
pZt + (1 − p)Zt (i)
for w = α
P(Yt+1 (i) = w |Ft ) =
1 − pZt − (1 − p)Zt (i) for w = β
for some fixed p ∈ [0, 1]. The parameter p is called the
interaction parameter.
Interacting
Urns
Neeraja
Sahasrabudhe
Introduction
Background
Problem setup
Synchronization
L2 synchronization
Almost sure
synchronization
Fluctuation
Theorems
Stable
Convergence
Stochastic
Approximation
Other models
and Future
Directions
E [Zt+1 (i)|Ft ] = E [Wt+1 (i)/Nt+1 |Ft ]
Wt (i) + Yt+1 (i)
= E
|Ft
Nt+1
Nt
1
=
Zt (i) +
[αai + β(1 − ai )]
Nt+1
Nt+1
Nt + (α − β)(1 − p)
(α − β)p
=
Zt (i) +
Zt
Nt+1
Nt+1
β
+
Nt+1
Interacting
Urns
Neeraja
Sahasrabudhe
Introduction
Background
Problem setup
Synchronization
L2 synchronization
Almost sure
synchronization
Fluctuation
Theorems
Stable
Convergence
Stochastic
Approximation
Other models
and Future
Directions
"
1 X
Zt+1 (i)|Ft
E [Zt+1 |Ft ] = E
N
i
1 X
=
E [Zt+1 (i)|Ft ]
N
#
i
=
β
Nt + (α − β)
Zt +
Nt+1
Nt+1
Synchronization
Interacting
Urns
Neeraja
Sahasrabudhe
Introduction
Background
Problem setup
Synchronization
L2 synchronization
Almost sure
synchronization
Fluctuation
Theorems
Stable
Convergence
Stochastic
Approximation
Other models
and Future
Directions
Fraction of balls of a given color converges almost surely, as
the time goes to infinity, to the same limit for all urns.
L2 -synchronization
Interacting
Urns
Neeraja
Sahasrabudhe
Introduction
Background
Problem setup
Synchronization
L2 synchronization
Almost sure
synchronization
Fluctuation
Theorems
Stable
Convergence
Stochastic
Approximation
Other models
and Future
Directions
Theorem (S.)
Set ρ =
α−β
α+β
> 0. The following asymptotic estimates hold:
 2ρ−2
for ρ > 1/2
 t
t −1 log t for ρ = 1/2
Var (Zt ), Var (Zt (i)) ∼
 −1
t
for ρ < 1/2
for every i ∈ {1, . . . , N}.
 2ρ−2ρp−2
for ρ > 1/[2(1 − p)]
 t
t −1 log t
for ρ = 1/[2(1 − p)]
Var (Zt (i) − Zt ) ∼
 −1
t
for ρ < 1/[2(1 − p)]
Analysis
Interacting
Urns
Neeraja
Sahasrabudhe
Introduction
Background
Problem setup
Synchronization
L2 synchronization
Almost sure
synchronization
Fluctuation
Theorems
Stable
Convergence
Stochastic
Approximation
Other models
and Future
Directions
For a given i, for ρ < 1/2, the L2 rate of convergence of
var (Zt ), Var (Zt (i)) and Var (Zt − Zt (i)) are the same.
Analysis
Interacting
Urns
Neeraja
Sahasrabudhe
Introduction
Background
Problem setup
Synchronization
L2 synchronization
Almost sure
synchronization
Fluctuation
Theorems
Stable
Convergence
Stochastic
Approximation
Other models
and Future
Directions
For a given i, for ρ < 1/2, the L2 rate of convergence of
var (Zt ), Var (Zt (i)) and Var (Zt − Zt (i)) are the same.
In the interval 1/2 < ρ < 1/2(1 − p), the Var (Zt − Zt (i))
converges to zero faster than both Var (Zt ) and
Var (Zt (i)).
Analysis
Interacting
Urns
Neeraja
Sahasrabudhe
Introduction
Background
Problem setup
Synchronization
L2 synchronization
Almost sure
synchronization
Fluctuation
Theorems
Stable
Convergence
Stochastic
Approximation
Other models
and Future
Directions
For a given i, for ρ < 1/2, the L2 rate of convergence of
var (Zt ), Var (Zt (i)) and Var (Zt − Zt (i)) are the same.
In the interval 1/2 < ρ < 1/2(1 − p), the Var (Zt − Zt (i))
converges to zero faster than both Var (Zt ) and
Var (Zt (i)).
Again, for ρ > 1/[2(1 − p)], Var (Zt (i) − Zt ) converges at
a faster rate.
Analysis
Interacting
Urns
Neeraja
Sahasrabudhe
Introduction
Background
Problem setup
Synchronization
L2 synchronization
Almost sure
synchronization
Fluctuation
Theorems
Stable
Convergence
Stochastic
Approximation
Other models
and Future
Directions
For a given i, for ρ < 1/2, the L2 rate of convergence of
var (Zt ), Var (Zt (i)) and Var (Zt − Zt (i)) are the same.
In the interval 1/2 < ρ < 1/2(1 − p), the Var (Zt − Zt (i))
converges to zero faster than both Var (Zt ) and
Var (Zt (i)).
Again, for ρ > 1/[2(1 − p)], Var (Zt (i) − Zt ) converges at
a faster rate.
This is a deviation from the behaviour observed in the
interacting Pólya urns model.
Almost sure synchronization
Interacting
Urns
Neeraja
Sahasrabudhe
Introduction
Background
Problem setup
Synchronization
L2 synchronization
Almost sure
synchronization
Fluctuation
Theorems
Stable
Convergence
Stochastic
Approximation
Other models
and Future
Directions
Theorem (S.)
Let Zt and Zt (i) be as defined above. Then,
lim (Zt − Zt (i)) = 0 a.s.
t→∞
for every i ∈ {1, 2, . . . N}.
Quasi-martingale
Interacting
Urns
Neeraja
Sahasrabudhe
Proposition
Introduction
Background
Problem setup
Zt (i) and Zt are quasi-martingales. That is,
Synchronization
∞
X
L2 synchronization
Almost sure
synchronization
Fluctuation
Theorems
Stable
Convergence
Stochastic
Approximation
Other models
and Future
Directions
E [|E (Zt+1 (i)|Ft ) − Zt (i)|] < ∞.
t=0
and
∞
X
t=0
E [|E (Zt+1 |Ft ) − Zt |] < ∞.
Fluctuations around the limit
Interacting
Urns
Neeraja
Sahasrabudhe
Introduction
Background
Problem setup
Synchronization
L2 synchronization
Almost sure
synchronization
Fluctuation
Theorems
Stable
Convergence
Stochastic
Approximation
Other models
and Future
Directions
In his paper titled ‘Bernard Freidman’s Urn’ (1965), David
Freedman proved some important results on the fluctuations of
fraction of a particular colour of balls in a Friedman urn around
the limit 1/2.
Results
Interacting
Urns
Neeraja
Sahasrabudhe
Theorem (S.)
Introduction
Let ρ =
Background
Problem setup
1
Synchronization
L2 synchronization
Almost sure
synchronization
α−β
α+β .
Then, we have the following:
For 0 < ρ < 12 ,
√
t(Zt − 1/2) −−−→ N 0,
Fluctuation
Theorems
Stable
Convergence
Stochastic
Approximation
Other models
and Future
Directions
2
stably
ρ2
4N(1 − 2ρ)
For ρ = 12 ,
√
t
ρ2
stably
√
(Zt − 1/2) −−−→ N 0,
4N
log t
Interacting
Urns
Neeraja
Sahasrabudhe
Theorem (S.)
Let ρ =
Introduction
Background
Problem setup
1
α−β
α+β .
Then, we have the following:
For 0 < ρ <
1
2(1−p) ,
Synchronization
L2 synchronization
Almost sure
synchronization
√
stably
t(Zt − Zt (i)) −−−→ N
Fluctuation
Theorems
Stable
Convergence
Stochastic
Approximation
Other models
and Future
Directions
2
For ρ =
!
1 − N1 ρ2
0,
4[1 − 2ρ(1 − p)]
1
2(1−p) ,
√
t
1 ρ2
stably
√
(Zt − Zt (i)) −−−→ N 0, 1 −
N 4
log t
Interacting
Urns
Neeraja
Sahasrabudhe
Theorem (S.)
Let ρ =
α−β
α+β .
Introduction
Background
Problem setup
1
For ρ >
Then,
1
2
and u = 1 − ρ,
Synchronization
1
a.s./L
t u (Zt − 1/2) −−−−→ Ve
L2 synchronization
Almost sure
synchronization
for some real random variable Ve such that P(Ve 6= 0) > 0.
Fluctuation
Theorems
Stable
Convergence
Stochastic
Approximation
Other models
and Future
Directions
2
For ρ >
1
2(1−p)
and u = 1 − (1 − p)ρ,
1
a.s./L
t u (Zt − Zt (i)) −−−−→ Xe
for some real random variable Xe such that P(Xe 6= 0) > 0.
Stable Convergence
Interacting
Urns
Neeraja
Sahasrabudhe
Introduction
Background
Problem setup
Synchronization
L2 synchronization
Almost sure
synchronization
Fluctuation
Theorems
Stable
Convergence
Stochastic
Approximation
Other models
and Future
Directions
The notion of stable Convergence was introduced by Rényi
(1963). It is, in a sense, stronger version of convergence in
distribution.
Stable Convergence
Interacting
Urns
Neeraja
Sahasrabudhe
Introduction
Background
Problem setup
Synchronization
L2 synchronization
Almost sure
synchronization
Fluctuation
Theorems
Stable
Convergence
Stochastic
Approximation
Other models
and Future
Directions
Let (Ω, A, P) be a probability space, and let S be a Polish
space, endowed with its Borel σ-field. A kernel on S is a
collection K = {K (ω) : ω ∈ Ω} of probability measures on the
Borel σ-field of S such that, for each bounded Borel real
function f on S, the map
Z
ω 7→ K (f )(ω) = f (x) K (ω)(dx)
is A-measurable.
On (Ω, A, P), let (Yt ) be a sequence of S-valued random
variables and let K be a kernel on S. Then we say that Yt
stably
converges stably to K , and we write Yt −→ K , if
weakly
P(Yt ∈ · | H) −→ E [K (·) | H]
for all H ∈ A with P(H) > 0.
Interacting
Urns
Neeraja
Sahasrabudhe
Introduction
Background
Problem setup
Synchronization
L2 synchronization
Almost sure
synchronization
Fluctuation
Theorems
Stable
Convergence
Stochastic
Approximation
Other models
and Future
Directions
stably
If Yt −→ K , then Yt converges in distribution to the
probability distribution E [K (·)].
The convergence in probability of Yt to a random variable
Y is equivalent to the stable convergence of Yt to a
special kernel, which is the Dirac kernel K = δY .
Stochastic Approximation
Interacting
Urns
Neeraja
Sahasrabudhe
Introduction
Background
Problem setup
Synchronization
L2 synchronization
Almost sure
synchronization
Fluctuation
Theorems
Stable
Convergence
Stochastic
Approximation
Other models
and Future
Directions
A stochastic approximation scheme in Rd is given by
x(k + 1) = x(k) − a(k)h(x(k)) + a(k)M(k + 1)
where:
h : Rd 7→ Rd is a locally Lipschitz continuous function.
{M(k)} is a square-integrable martingale difference
sequence with respect to increasing family of σ-fields
Fk = σ(x(m), M(m), m ≤ k) with
E [kM(k + 1)k2 |Fk ] ≤ ∞
{a(k)} are step-sizes satisfying
X
X
a(k) = ∞,
a(k)2 < ∞.
a(k) > 0 ∀k,
k
k
The idea is that the incremental adaptation due to the
slowly decreasing step-size a(k) averages out the noise
{M(k)}.
Interacting
Urns
Neeraja
Sahasrabudhe
Introduction
Background
Problem setup
Under some additional conditions, the solution of the random
difference equation tracks, with probability one, the trajectory
of the solution of a limiting o.d.e. given by
Synchronization
L2 synchronization
Almost sure
synchronization
Fluctuation
Theorems
Stable
Convergence
Stochastic
Approximation
Other models
and Future
Directions
ẋ(t) = −h(x(t)).
(2)
as long as the iterates remain bounded.
Under additional conditions on the noise, it will converge to a
(possibly random) stable equilibrium.
Interacting
Urns
Neeraja
Sahasrabudhe
Theorem
Let δ0 = (1/2, . . . , 1/2) be an N-dimensional vector and let A
denote the N × N matrix given by
Introduction
 1/2 − ρ(1 − p) − ρp
N

− ρp
N


.

.

.
Background
Problem setup
Synchronization
L2 synchronization
Almost sure
synchronization
Fluctuation
Theorems
− ρp
N
.
Then,
Stable
Convergence
Stochastic
Approximation
1
Other models
and Future
Directions
2
For ρ < 1/2,
√
...
...
− ρp
N
− ρp
N
.
.
.
.
.
.
...
.
.
.
1/2 − ρ(1 − p) − ρp
N
− ρp
N






d
A−1 ρ2
n(Zet − δ0 ) −
→ N 0, 8(1−2ρ)
√
3
− ρp
N
1/2 − ρ(1 − p) − ρp
N
d
For ρ = 1/2, √logn n (Zet − δ0 ) −
→ N (0, Σ), for some N × N
positive semi-definite matrix Σ.
For ρ > 1/2, n(1−ρ) (Zet − δ0 ) converges a.s. and in L1
towards a finite random variable.
Other interacting urn models
Interacting
Urns
Neeraja
Sahasrabudhe
Introduction
Background
Problem setup
Synchronization
L2 synchronization
Almost sure
synchronization
Fluctuation
Theorems
Stable
Convergence
Stochastic
Approximation
Other models
and Future
Directions
Michel Benaı̈m, Itai Benjamini, Jun Chen and Yuri Lima,
“A generalized Pólya’s urn with graph based interactions”
(2015):
Graph-based model, with urns at each vertex and pair-wise
interactions: Given a finite connected graph G , place a bin
at each vertex. At discrete times, a ball is added to each
pair of bins.
In a pair of bins, one of the bins gets the ball with
probability proportional to its current number of balls
raised by some fixed power α > 0.
“Interacting Urn Models”, Mickaël Launay (2012).
Future directions
Interacting
Urns
Neeraja
Sahasrabudhe
Introduction
Background
Problem setup
Synchronization
L2 synchronization
Almost sure
synchronization
Fluctuation
Theorems
Stable
Convergence
Stochastic
Approximation
Other models
and Future
Directions
Reinforcement Scheme:
pZt + (1 − p)Zt (i)
for w = α
P(Yt+1 (i) = w |Ft ) =
1 − pZt − (1 − p)Zt (i) for w = β
Interacting
Urns
Neeraja
Sahasrabudhe
Introduction
Background
Problem setup
Synchronization
L2 synchronization
Almost sure
synchronization
Fluctuation
Theorems
Stable
Convergence
Stochastic
Approximation
Other models
and Future
Directions
Thank you!