II. Multi-Dimensional Random Variables
and Application in Vector Quantization
Scalar Quantization
The Process of Analog-to-Digital in any
Communication System involves
1. Sampling
(invertible)
Transfer the signal to discrete-time analog-amplitude
2. Quantization
(non-invertible)
Transfer the signal to discrete-time discrete-amplitude
3. Bit Representation
(invertible)
Represent the quantized signal as coded bits
© Tallal Elshabrawy
2
Scalar Quantization
Assume some continuous random variable X
represents the output of an analog source
Quantization  Y=q(X)
Since X is a random variable, Y is also a random
variable. Y is a discrete random variable
Quantization ≡ Distortion
Define error (Quantizer noise)  X-q(X)
© Tallal Elshabrawy
3
Signal Distortion and Fidelity
Signal Distortion

2
2


d = E  X  q  X      x  q  x   f X  x  dx

 
Fidelity of a Quantized Signal
SQNR Signal-to-Quantization Noise Power
E  X 
2
SQNR =
2

E  X  q  X  


Signal
Power
Noise
Power
Note:
The distortion d alone is not a significant enough measure for quality of a Quantizer because it
lacks information about this distortion is relative to what. That is why the SQNR is used for
measuring the fidelity of any Quantizer
© Tallal Elshabrawy
4
Quantiztion Process
Continuous
Sample X
Quantizer
q(.)
Ik
Discrete
Sample Y
xk-2
xk-1
yk
xk
xk+1
 xk, k=1,2,…,L-1 are known as decision levels, x0=-∞, xL= ∞
 yk, k=1,2,…,L are known as representation levels
 yk-yk-1, k=2,…,L is known as step size
 Quantizer mapping function Y=q(X)
Quantizer output is yk if the input sample X belongs to the interval Ik
© Tallal Elshabrawy
5
Example: 3-bit Uniform Quantizer
fX(x)


Assume signal X uniformly
distributed over (-4, 4)
Divide sample space of X
into L=8 equally spaced
subintervals of width ∆=1
1/8
I1
I2
I3
I4
Quantizer
Output
I5
I6
X
I7
I8
3.5
2.5
1.5
0.5

For each interval the
quantization level is mid
point
-4
-3
-2
-1
0
-0.5
1
2
3
Quantizer
4 Input
-1.5
-2.5

Each representation level
could be represented by 3
bits
© Tallal Elshabrawy
-3.5
Mid-Rise uniform Quantizer
6
Distortion for 3-bit Uniform Quantizer

2


d = E  X  q  X      x  q  x   f X  x  dx

 
2
8
2

d =  E  X  q  X  X  Ik  P  X  Ik 


k 1
© Tallal Elshabrawy
7
Distortion for 3-bit Uniform Quantizer
Let Focus on I5= [0, 1]
2
2



E  X  q  X   X  I5 = E  X  0.5 X  0,1





2


E  X  0.5 X  0,1    x  0.5 f X X 0,1  x  dx



fX  x
18
f X X 0,1  x  =

1
P  X   0,1 1 8
2
2

 E  X  0.5  X   0,1 


2

 E  X  0.5  X   0,1 


© Tallal Elshabrawy

  x  0.5
2
dx


1
  x  0.5 dx  12
2
8
Distortion for 3-bit Uniform Quantizer
We could generalize that
2
1


E  X  q  X   X  Ik =

 12
k
Therefore
8
2

d =  E  X  q  X  X  Ik  P  X  Ik 


k 1
8
1
d    P  X  Ik 
k 1 12
1 8
1
d=
  P  X  Ik  
12 k 1
12
© Tallal Elshabrawy
9
Distortion for 3-bit Uniform Quantizer
Given that
4
1 2
16
E  X  =  x dx 
8 4
3
2
Therefore
SQNR =
© Tallal Elshabrawy
E  X 2 
d
16/ 3

 64  18.06 dB
1 12
10
Distortion for n-bit Uniform Quantizer
For any signal that is uniformly distributed over
(-a/2,a/2) and a subinterval size of ∆=a/L where L is
the number of quantization levels and n is the
number of bits used to represent each level (i.e., L=2n)
SQNR=L2  22 n
SQNR  dB  =2n log10  2   6.02n dB
SQNR is increased by approximately 6 dB for each extra bit added to
the Quantizer
© Tallal Elshabrawy
11
Non-Uniform Quantizers
 Variable separation
between representation
levels
 WHY?
fX(x)
 Large range of input signal
 It is desirable to decrease
the number of
representation levels as
much as possible
x
 Basic Concept of NonUniform Quantizers
 Concentrate quantization
levels within the range
where the PDF of the
message signal is large
© Tallal Elshabrawy
Representation Levels
12
Design of an Optimum Quantizer
Continuous
Sample X
Quantizer
q(.)
Ik
Discrete
Sample Y
xk-2
xk-1
yk
xk
xk+1
 xk, k=1, 2,…,L-1 are known as decision levels, x0=-∞, xL= ∞
 yk, k=1,2,…,L are known as representation levels
The design of an optimal Quantizer involves determining
2L-1 variables that reflect
1. What are the decision levels and therefore decision subintervals
2. Where to insert the representation levels within decision intervals
© Tallal Elshabrawy
13
Theorem
A random variable X with PDF fX(x) is to be quantized with
an L-level Quantizer
y=q(x) = yk for xk-1<x<xk and k=1,2, …, L and x0=-∞, xL= ∞
Minimal Distortion is achieved by
1. yk=E[X| xk-1<x<xk ], k=1, 2, …, L
2. xk=(yk+yk+1)/2, k=1, 2, …, L-1
These two criteria provide a system of 2L-1 equations
when solved for (x1, x2, …, xL-1, y1, y2, …, yL) specify the
L-level Optimal Quantizer
© Tallal Elshabrawy
14
Proof
L
d 
xk
 x  y  f x dx
2
k
k 1 xk 1
X
To minimize d with respect to yk
k
d
 2   x  yk f X  x dx  0
yk
xk 1
x

xk
xk
 xf x dx  y  f x dx
X
k
xk 1
X
xk 1
xk
 xf x dx
X
 yk 
xk 1
xk
 f x dx

xk

xk 1
x
f X x 
dx
Px  xk 1 , xk 
X
xk 1
 yk 
xk
 xf
X x xk 1 , xk 
x dx  EX xk 1  x  xk 
Intuitively this result should make sense
because the best representation level
within a certain decision interval should
be the location where the signal is most
likely expected to be within this interval
(i.e., the expectation of the signal inside
the decision interval)
xk 1
© Tallal Elshabrawy
15
Proof (Cont’d)
L
d 
xk
 x  y  f x dx
2
k
k 1 xk 1
X
To minimize d with respect to xk
d
0
xk


xk
xk 1
 xk

2
2
  x  yk  f X x dx    x  yk 1  f X  x dx   0
 xk 1

xk
x
Fundamental Theorem in Calculus
d
Gx    h t dt  Gx   hx 
dx
c
 xk  yk  f X  xk    xk  yk 1  f X  xk   0
2
 xk 
2
yk  yk 1
2
© Tallal Elshabrawy
16
Proof (Cont’d)
Suppose xk is closer to yk than yK+1
All points in the shaded region are closer to
yk than yk+1which means the total distortion
is NOT minimum and the Quantizer is not
optimal
yk xk
yk+1
Suppose xk is closer to yk+1 than yK
All points in the shaded region are closer to
yk+1 than ykwhich means the total distortion
is NOT minimum and the Quantizer is not
optimal
xk yk+1
yk
When xk is in the middle between yk+1 and yK
We could not find a region within the
decision interval of yk where yk+1 is closer or
vice versa
© Tallal Elshabrawy
yk
xk
yk+1
17
Lloyd-Max Algorithm Version 1
An iterative algorithm to design an optimal Quantizer
1. Start with some arbitrary initial set of representation levels (i.e.,
assume yk, k=1, 2,…,L are known)
2. Use the second criterion for minimal distortion from the theorem to
calculate the corresponding decision levels
(i.e., xk=(yk+yk+1)/2, k=1, 2, …, L-1)
3. Given the decision levels computed in step 2, calculate the
corresponding representation levels using the first criterion for
minimal distortion from the theorem
(i.e., yk=E[X| xk-1<x<xk ], k=1, 2, …, L)
4. Keep iterating between steps 2 and 3 until the a stopping criterion
is reached. Examples of stopping criteria are change in distortion
from one iteration to the other falls below some threshold
© Tallal Elshabrawy
18
Lloyd-Max Algorithm Version 2
An iterative algorithm to design an optimal Quantizer
1. Start with some arbitrary initial set of decision levels (i.e., assume
xk, k=1, 2,…,L-1 are known)
2. Use the first criterion for minimal distortion from the theorem to
calculate the corresponding representation levels
(i.e., yk=E[X| xk-1<x<xk ], k=1, 2, …, L)
3. Given the representation levels computed in step 2, calculate the
corresponding decision levels using the second criterion for
minimal distortion from the theorem
(i.e., xk=(yk+yk+1)/2, k=1, 2, …, L-1)
4. Keep iterating between steps 2 and 3 until the a stopping criterion
is reached. Examples of stopping criteria are change in distortion
from one iteration to the other falls below some threshold
© Tallal Elshabrawy
19