CMA 2007
Applications of free probability and random matrix
theory
Øyvind Ryan
January 2008
Øyvind Ryan
Applications of free probability and random matrix theory
CMA 2007
Applications of free probability and random matrix theory
Some important concepts from classical probability
I
Random variables are functions (i.e. they commute w.r.t.
multiplication) with a given p.d.f. (denoted f )
I
Expectation (denoted E ) is integration
I
Independence
I
Additive convolution (∗) and the logarithm of the Fourier
transform
I
Multiplicative convolution
I
Central limit law, with special role of the Gaussian law
¢
¢∗n
¡¡
Poisson distribution Pc : The limit of 1 − nc δ(0) + nc δ(1)
as n → ∞.
I
I
Divisibility: For a given a, nd i.i.d. b1 , ..., bn such that
fa = fb1 +···+bn .
Øyvind Ryan
Applications of free probability and random matrix theory
CMA 2007
Applications of free probability and random matrix theory
I
Can we nd a more general theory, where the random variables
are matrices (or more generally, operators), with their
eigenvalue distribution (or spectrum) taking the role as the
p.d.f.?
I
What are the analogues to the above mentioned concepts for
this theory? In particular, what plays the role as central limit?
I
What are the applications of such a theory?
Øyvind Ryan
Applications of free probability and random matrix theory
CMA 2007
Applications of free probability and random matrix theory
Free probability
Free probability was developed as a probability theory for random
variables which do not commute, like matrices
I The random variables are elements in a unital ∗-algebra
(denoted A), typically B (H), or Mn (C).
I Expectation (denoted φ) is a normalized linear functional on A.
The pair (A, φ) is called a noncommutative probability space.
I For matrices, φ will be the normalized trace trn , dened by
n
1X
trn (a) =
aii .
n
i =1
I
For random matrices, we set φ(a) = τn (a) = E (trn (a)) is
dened by.
Useful way to think about free probability: Two independent
random matrices, where the eigenvectors for one of them point
in each direction with equal probability, then the two matrices
are "free" (to be dened).
Øyvind Ryan
Applications of free probability and random matrix theory
CMA 2007
Applications of free probability and random matrix theory
The full circle law
Let Xn = √1n Yn where Yn is n × n and has i.i.d. complex standard
Gaussian entries. This is a "central limit candidate". What
happens when n is large? The eigenvalues converge to what is
called the full circle law. Here for n = 500.
1
0.8
0.6
0.4
0.2
0
−0.2
−0.4
−0.6
−0.8
−1
−1
−0.5
0
0.5
1
plot(eig( (1/sqrt(1000)) * (randn(500,500) +
j*randn(500,500)) ),'kx')
Øyvind Ryan
Applications of free probability and random matrix theory
CMA 2007
Applications of free probability and random matrix theory
The semicircle law
35
30
25
20
15
10
5
0
−3
−2
−1
0
1
2
3
A = (1/sqrt(2000)) * (randn(1000,1000) +
j*randn(1000,1000));
A = (sqrt(2)/2)*(A+A');
hist(eig(A),40)
Øyvind Ryan
Applications of free probability and random matrix theory
CMA 2007
Applications of free probability and random matrix theory
The Marchenko Pastur law
What happens with the eigenvalues of N1 Xn XH
n when Xn is an
n × N random matrix with standard complex Gaussian entries?
I One can show that
µµ
¶p ¶
X
X ak
1
H
Xn Xn
=
c l (π̂)−1 +
τn
.
N
N 2k
π̂∈NC2p
I
k
Convergence is "almost sure", i.e. we have very accurate
eigenvalue prediction when the matrices are large.
When Nn → c, the eigenvalue distribution converges to the
Marchenko Pastur law with parameter c, denoted µ Nn , with
density
p
(x − a)+ (b − x )+
1 +
µc
f (x ) = (1 − ) δ(x ) +
,
(1)
c
2π cx
√
√
where (z )+ = max(0, z ), a = (1 − c )2 and b = (1 + c )2 .
Øyvind Ryan
Applications of free probability and random matrix theory
CMA 2007
Applications of free probability and random matrix theory
Four dierent Marchenko Pastur laws µ Nn are drawn.
1.6
c=0.5
c=0.2
c=0.1
c=0.05
1.4
1.2
Density
1
0.8
0.6
0.4
0.2
0
0.5
1
Øyvind Ryan
1.5
2
2.5
3
Applications of free probability and random matrix theory
CMA 2007
Applications of free probability and random matrix theory
Motivation for free probability
One can show that for the Gaussian random matrices we
considered, the limits
¡
¢
¡
¢
φ X i1 Y j1 · · · X il Y jl = lim trn Xin1 Ynj1 · · · Xinl Ynjl
n→∞
exist. If we linearly extend the linear functional φ to all polynomials
in A and B, the following can be shown:
Theorem
If Pi , Qi are polynomials in X and Y respectively, with 1 ≤ i ≤ l ,
and φ(Pi (X )) = 0, φ(Qi (Y )) = 0 for all i, then
φ (P1 (X )Q1 (Y ) · · · Pl (X )Ql (Y )) = 0.
This motivates the denition of freeness, which is the analogy to
independence.
Øyvind Ryan
Applications of free probability and random matrix theory
CMA 2007
Applications of free probability and random matrix theory
Denition of freeness
Denition
A family of unital ∗-subalgebras (Ai )i ∈I is called a free family if


aj ∈ Aij


⇒ φ(a1 · · · an ) = 0. (2)
i1 6= i2 , i2 6= i3 , · · · , in−1 6= in


φ(a1 ) = φ(a2 ) = · · · = φ(an ) = 0
A family of random variables ai is called a free family if the algebras
they generate form a free family.
Øyvind Ryan
Applications of free probability and random matrix theory
CMA 2007
Applications of free probability and random matrix theory
The free central limit theorem
Theorem
If
I
a1 , ..., an are free and self-adjoint,
I
φ(ai ) = 0,
I
I
φ(ai2 = 1,
supi |φ(aik )| < ∞ for all k,
√
then the sequence (a1 + · · · + an )/ n converges in distribution to
the semicircle law.
In free probability, the semicircle law thus
√ has the role of the
1
Gaussian law. it's density is density 2π 4 − x 2
Øyvind Ryan
Applications of free probability and random matrix theory
CMA 2007
Applications of free probability and random matrix theory
Similarities between classical and free probability
1. Additive convolution ¢: The p.d.f. of the sum of free random
variables. The role of the logarithm of the Fourier transform is
now taken by the R-transform, which satises
Rµa ¢µb (z ) = Rµa (z ) + Rµb (z ).
2. The S-transform: Transform on probability distributions which
satises Sµa £µb (z ) = Sµa (z )Sµb (z )
3. Poisson distributions have their analogy in the free Poisson
distributions: These are given by the Marcenko Pastur laws µc
with parameter c, which also can be written as the limit of
¡¡
¢
¢¢n
1 − nc δ(0) + nc δ(1)
as n → ∞
4. Innite divisibility: There exists an analogy to the Lévy-Hin£in
formula for innite divisibility in classical probability.
Øyvind Ryan
Applications of free probability and random matrix theory
CMA 2007
Applications of free probability and random matrix theory
Main usage of free probability in my papers
I
Let X and Y be random matrices. How can we make a good
prediction of the eigenvalue distribution of X when one has the
eigenvalue distribution of XY and Y? Simplest case is when
one assumes that Y is Gaussian (Noise). What about the
eigenvectors?
I
Assume that we have the eigenvalue distribution of
1
H
N (R + X)(R + X) , where R and X are independent n × N
random matrices, with X Gaussian. If the columns of R are
realizations of some random vector r, what is the covariance
matrix E (ri rj∗ )?
I
Multiplicative free convolution with the Marchenko Pastur law
is particularly useful. This has an ecient implementation.
Øyvind Ryan
Applications of free probability and random matrix theory
CMA 2007
Applications of free probability and random matrix theory
Application of free probability
The capacity per receiving antenna of a channel with n × m
channel matrix H and signal to noise ratio ρ = σ12 is given by
µ
¶
n
1
1
1X
1
H
C = log2 det In +
HH
=
log2 (1 + 2 λl )
n
mσ 2
n
σ
(3)
l =1
where λl are the eigenvalues of m1 HHH . We would like to estimate
C.
To estimate C , we will use free probability tools to estimate the
eigenvalues of m1 HHH based on some observations Ĥi
Øyvind Ryan
Applications of free probability and random matrix theory
CMA 2007
Applications of free probability and random matrix theory
Observation model
The following is a much used observation model:
Ĥi = H + σ Xi
(4)
where
I
The matrices are n × m (n is the number of receiving
antennas, m is the number of transmitting antennas)
I
Ĥi is the measured MIMO matrix,
Xi is the noise matrix with i.i.d standard complex Gaussian
I
entries.
Øyvind Ryan
Applications of free probability and random matrix theory
CMA 2007
Applications of free probability and random matrix theory
Existing ways to estimate the channel capacity
Several channel capacity estimators have been used in the literature:
³
´
1 PL
1
H
Ĥ
Ĥ
C1 = nL
log
det
I
+
n
i =1
mσ 2 i i ´
³2
1
1 PL
C2 = n log2 det In + Lσ2 m i =1 Ĥi ĤH
(5)
i
³
´
P
P
C3 = n1 log2 det In + σ21m ( L1 Li=1 Ĥi )( L1 Li=1 Ĥi )H )
Why not try to formulate an estimator based on free probability
instead?
Øyvind Ryan
Applications of free probability and random matrix theory
CMA 2007
Applications of free probability and random matrix theory
2.6
2.4
2.2
Capacity
2
1.8
1.6
1.4
1.2
1
True capacity
C1
C2
C3
0.8
0
5
10
15
20
Number of observations
25
30
Comparison of the classical capacity estimators for various number
of observations. σ 2 = 0.1, n = 10 receive antennas, m = 10
transmit antennas. The rank of H was 3.
Øyvind Ryan
Applications of free probability and random matrix theory
CMA 2007
Applications of free probability and random matrix theory
Main free probability result we will use
Dene
Γn =
Wn =
1
Rn RHn
N
1
(Rn + σ Xn )(Rn + σ Xn )H ,
N
where Rn and Xn are independent n × N random matrices, Xn is
complex, standard, Gaussian.
Theorem
If e .e .d .(Γn ) → νΓ , then e .e .d .(Wn ) → νW where νW is uniquely
identied by
νW rµc = (νΓ rµc ) ¢ δσ2
(r = "the opposite of £").
Øyvind Ryan
Applications of free probability and random matrix theory
CMA 2007
Applications of free probability and random matrix theory
Realization of the theorem for the problem at hand
Form the compound observation matrix
σ
Ĥ1...L = H1...L + √ X1...L , where
L
i
1 h
Ĥ1...L = √ Ĥ1 , Ĥ2 , ..., ĤL ,
L
1
H1...L = √ [H, H, ..., H] ,
L
X1...L = [X1 , X2 , ..., XL ] .
For the problem at hand, the theorem takes the form
³
´
n ≈ ν1
n
ν 1 Ĥ1...L ĤH rµ mL
¢ δσ 2
rµ
H
H1...L H
mL
m
m
1...L
1...L
(6)
1
H
Since m1 H1...L HH
1...L = m HH , we can now estimate the moments
of m1 HHH from the moments of the observation matrix
1
H
m Ĥ1...L Ĥ1...L , and thereby estimate the eigenvalues, and hence the
channel capacity.
Øyvind Ryan
Applications of free probability and random matrix theory
CMA 2007
Applications of free probability and random matrix theory
Free probability based estimator for the moments of the
channel matrix
Can also be written in the following way for the rst four moments:
ĥ1 = h1 + σ 2
ĥ2 = h2 + 2σ 2 (1 + c )h1 + σ 4 (1 + c )
ĥ3 = h3 + 3¡σ 2 (1 + c )h2 ¢+ 3σ 2 ch12
+3σ 4¡ c 2 + 3c + 1¢ h1
+σ 6 c 2 + 3c + 1
ĥ4 = h4 + 4σ 2 (1 + c )h3 + 8σ 2 ch2 h1
+σ 4 (6c 2 + 16c + 6)h2
+14σ 4 c (1 + c )h12
+4σ 6¡(c 3 + 6c 2 + 6c + 1¢)h1
+σ 8 c 3 + 6c 2 + 6c + 1 ,
where ĥi are the moments of the observation matrix
hi are the moments of m1 HHH .
Øyvind Ryan
(7)
1
H
m Ĥ1...L Ĥ1...L ,
Applications of free probability and random matrix theory
CMA 2007
Applications of free probability and random matrix theory
2.6
2.4
2.2
Capacity
2
1.8
1.6
1.4
1.2
1
True capacity
C
f
Cu
0.8
0
5
10
15
20
Number of observations
25
30
Comparison of Cf and Cu for various number of observations.
σ 2 = 0.1, n = 10 receive antennas, m = 10 transmit antennas. The
rank of H was 3.
Øyvind Ryan
Applications of free probability and random matrix theory
CMA 2007
Applications of free probability and random matrix theory
4.5
4
Capacity
3.5
True capacity, rank 3
C , rank 3
f
True capacity, rank 5
C , rank 5
3
f
True capacity, rank 6
Cf, rank 6
2.5
2
1.5
0
10
20
30
40
50
60
Number of observations
70
80
90
100
Cf for various number of observations. No phase o-set/phase
drift. σ 2 = 0.1, n = 10 receive antennas, m = 10 transmit
antennas. The rank of H was 3, 5 and 6.
Øyvind Ryan
Applications of free probability and random matrix theory
CMA 2007
Applications of free probability and random matrix theory
List of papers
I
Free Deconvolution for Signal Processing Applications.
Submitted to IEEE Trans. Inform. Theory.
arxiv.org/cs.IT/0701025.
I
Multiplicative free Convolution and Information-Plus-Noise
Type Matrices. arxiv.org/math.PR/0702342.
I
Channel Capacity Estimation using Free Probability Theory.
Submitted to IEEE. Trans. Signal Process.
arxiv.org/abs/0707.3095.
I
Random Vandermonde Matrices-Part I: Fundamental results.
Work in progress.
I
Random Vandermonde Matrices-Part II: Applications to
wireless applications. Work in progress.
I
Vandermonde Frequency Division Multiplexing: An Approach
for Cognitive Radio. Work in progess.
Øyvind Ryan
Applications of free probability and random matrix theory
CMA 2007
Applications of free probability and random matrix theory
This talk is available at
http://heim.i.uio.no/∼oyvindry/talks.shtml.
My publications are listed at
http://heim.i.uio.no/∼oyvindry/publications.shtml
Øyvind Ryan
Applications of free probability and random matrix theory