III. Multi-Dimensional Random Variables
and Application in Vector Quantization
Matrix Notation for Representing Vectors

Assume X is a two-dimensional
vector, then in matrix notation it
is represented as:
X   x1 x 2 

The norm of any vector X
|| X || could be computed
using the dot product as
follows:
2
x2
X
UX
x1
X = XX T

For any vector X the unit vector
UX is defined as
X
UX =
X
© Tallal Elshabrawy
2
Projection and Dot Product
We would like to evaluate the projection of vector
Y over vector X using the matrix notation (i.e, we
would like to compute the value m),
m
x1 y1  x 2 y 2
x12  x 22
XY T
m
X
For proof see
lecture notes
Y=[y1 y2]
X=[x1 x2]
a
x12  x 22  m
m
© Tallal Elshabrawy
3
Principle Component Analysis
 Suppose we have a number of samples in a two
dimensional space and we would like to identify a unit
vector U that crosses the origin for which these samples
are the closest.
 This could be achieved by finding the vector for which the
sum of squares of distances to such vector is minimized
(i.e., find U that minimizes d12+d22+d32+d42+…)
U
X4=[x14 x24]
X1=[x11 x21]
d4 d3
d1
d2
X3=[x13 x23]
X2=[x12 x22]
© Tallal Elshabrawy
4
Principle Component Analysis
From the Pythagoras theorem, we could argue that the
closest vector to the observed samples is also equivalently
a problem of finding the vector over which the sum of
projection of the sample points square on it is maximized.
l
min  d  d  d  d  ...
2
1
2
2
2
3
2
4
d
is equivalent to
m
max  m12  m 22  m32  m 24  ...

max  X1 U
  X U   X U   X U 
T 2
© Tallal Elshabrawy
T 2
2
T 2
3
T 2
4
 ...

l2  d 2  m 2
For constant l
d  m 
5
Principle Component Analysis
Define
 X1   x11
X   x
 2   12
X  x
X   3    13
 X 4   x14
 .   .
  
 .   .
x 21 
x 22 
x 23 

x 24 
. 

. 
X4=[x14 x24]
X1=[x11 x21]
U
X3=[x13 x23]
X2=[x12 x22]
x12 x13 x14 . .
x
u 2   11

x
x
x
x
.
.
22
23
24
 21

UX T   u1 x11  u 2 x 21 u1 x12  u 2 x 22 u1x13  u 2 x 23
UX T   u1
u1x14  u 2 x 24 . .
2
2
2
2
  UX T  XU T    u1 x11  u 2 x 21    u1 x12  u 2 x 22    u1 x13  u 2 x 23    u1x14  u 2 x 24   ...


  UX T  XU T    X1 U T    X 2 U T    X 3 U T    X 4 U T 
2
© Tallal Elshabrawy
2
2
2
6
Principle Component Analysis
Therefore the unit vector U (i.e., UUT=1) that is
closest to sample points X1, X2, X3, X4 satisfies
max  UX XU
T
T

Suppose the maximum value is λ
X4=[x14 x24]
X1=[x11 x21]
U
X3=[x13 x23]
X2=[x12 x22]
 UX T XUT  λ  λUUT  U  λU T 
 U  X T XUT  λUT   0
  X T X  U T  λUT
The equation above is an eigen vector problem for the matrix XTX which means that
the maximum we are seeking λ must solve the equation above and it is therefore
one of the eigen values (maximum eigen value) for the SQUARE matrix XTX
© Tallal Elshabrawy
7
Two-Dimensional Random Variables
Assume X is a two-dimensional random variable, then in
matrix notation X=[X1 X2]
What does it mean that X is a two-dimensional random
variable?
It means that there is this experiment/phenomenon that
could be expressed in terms of 2 random variables X1 and
X2
Example: Weather (W) could be categorized as a twodimensional random variable if we characterize it by
Temperature (T) and Humidity (H). (i.e., W = [T, H])
© Tallal Elshabrawy
8
Parameters of Two-Dimensional R.V.
Mean of any of the Random Variables
E  Xi    k Pr  Xi  k 
if Xi depict discrete random variables
i=1,2
Xi
E  Xi  
 yf  y  dy
Xi
if Xi depict continuous random variables
i=1,2
Xi
Variance of any of the Random Variables
2
2


var  X i   E  Xi  E  Xi     k  E  X i  Pr  X i  k 

 Xi
i=1,2
if Xi depict discrete random variables
2
2
var  X i   E  X i  E  X i      y  E  X i  f Xi  y  dy

 X
i=1,2
i
if Xi depict continuous random variables
© Tallal Elshabrawy
9
Parameters for Relation Between R.V.s
Correlation
Corr  X1 , X 2   E  X1X 2 
Covariance
Cov  X1 , X 2   E  X1  E  X1   X 2  E  X 2  
© Tallal Elshabrawy
10
Covariance Matrix
cov  X1 , X 2  
 var  X1 
RX  

var  X 2  
cov  X 2 , X1 
E  X' 2  E X' X'  
 1 2
  1  
RX  

2
 E  X1' X '2  E  X '2   

 

where X1'  X1  E  X1  and X'2  X2  E  X2 
© Tallal Elshabrawy
11
2-D R.V. & Principle Component Analysis
Assume X is a two-dimensional random variable, then in
matrix notation X=[X1 X2]
Assume X’ is a two-dimensional random variable, then in
matrix notation X’=[X’1 X’2]
where X1’=X1 - E[X’1], X2’=X2 - E[X’2]
The vector U that is closest to sample points of the twodimensional random variable X is the eigen vector that
corresponds to the maximum eigen value for the
Covariance Matrix Rx’. i.e., U solves
R X ' U T  λU T
© Tallal Elshabrawy
(Proof in lecture notes)
12
© Tallal Elshabrawy
13
© Tallal Elshabrawy
14
© Tallal Elshabrawy
15
© Tallal Elshabrawy
16
© Tallal Elshabrawy
17