Information theory
경희대학교
MediaLab
서덕영
[email protected]
Digital Multimedia
2017-07-31
MediaLab , Kyunghee University
2
Information theory
Analog  digital conversion (ADC)
“Being digital” : integrated service
Coding(compression)
*.zip, *.mp3, *.avi ….
Channel capacity
LTE, “빠름, 빠름, 빠름”
Error correction
Information theory: Entropy
amount of information = ‘degree of surprise’
Entropy and average code length
I ( x)   log 2 p( x) [bits]  H ( X )  average( I ( x))
Information source and coding
Memory-less source : no correlation
∙∙∙∙∙
Red blue yellow yellow red black red ∙∙∙
H ( X )  2bits

l  4bytes
00011010001100 ∙∙∙
H ( X )  l  2bits
7/31/2017
Media Lab. Kyung Hee University
4
Analog-to-digital conversion
ADC: Quantization?
analog-to-digit-al quantization
In order to cook in binary computers
digital TV, digital comm., digital control…
ADC
Infinite numbers
finite numbers
fine-to-coarse digital quantization
7/31/2017
Media Lab. Kyung Hee University
6
ADC: Fine-to-coarse Quantization
Dice vs. coin
1/6
1/2
{1,2,3}  head
{4,5,6}  tail
1
3
5
2
2
3
1
4
5
5
H
6
quantization
4 ∙∙∙
H ( X )  2.5849
H
T
T
H
H
T
T ∙∙∙
H(X ) 1
Effects of quantization
Data compression
Information loss, but not all
7/31/2017
Media Lab. Kyung Hee University
7
pdf and information loss

pdf (probability density function)

The narrower pdf, the less error
at the same number of bits
The narrower pdf, the less number
of bits at the same error
Media signal
Non-uniform pdf

Fixed step size
 More error
H ( X )  1.811bits
Variable step size
 Less error
H ( X )  2bits
Media signal
Correlation in text
memory-less and memory
I(x) = log2 (1/px) = “degree of surprise”
qu-, re-, th-, -tion, less uncertain
Of course, there are exceptions... Qatar, Qantas
Conditional probability
p(u|q) >> p(u)
Then, I(u|q) << I(u)
accordingly, I(n|tio) << I(n)
7/31/2017
Media Lab. Kyung Hee University
10
Differential Pulse-Coded Modulation (DPCM)
Quantize not x[n] but d[n].
Principle :
Pdf of d[n] is narrower than that of x[n].
Less error at the same number of bits.
Less amount of data, at the same error.
-2o
d [n]
x[n ]
-20o
30o
3
Quantize
xˆ[ n ]
d [n]
Prediction
Media signal
d [n]
Coding
*.zip
Coding
Series of symbols  bits
Requirements
Uniquely decodable
Less number of bits:
l  H (X )
∙∙∙∙∙
Red blue yellow yellow red black red ∙∙∙
H ( X )  2bits

00011010001100
l  4bytes
∙∙∙
H ( X )  l  2bits
Huffman code
Average code length ∼ Entropy?
A solution is Huffman code.
Used in *.mp3, *.avi, *.mp4
Ex) Encode/decode
Media signal
AADHBEA
Other codes
Arithmetic code: HDTV
Zip code
winzip, compress, etc.
Channel Capacity
Digital communications
주파수 경매???
Digital communications
Claude Shannon(1916~2001) - American
electrical engineer who founded information
theory with his 1948 paper "A Mathematical
Theory of Communication"
Channel model
Additive Gaussian noise : SNR = P/No  n(t)
x(t)
y(t)
1/2
1/2
n(t)
-5V
5V
2-D Gaussian noise
B
A
σ
Multi-dimensional Gaussian channel
Noise power σ2 = NoW
(P   2 )
P



1
log 2 
N



(P   )
2

2
N

N
2

 1
P 


log
1

2

2 
2





Realcomplex
W trials per sec over W Hz band
Number of small spheres

(P   )
2
 
2
N

N
cf )
4 3
r in 3D
3
P 

log 2 1  2 
  

P 
bits / s
Cawgn ( P,W )  W log 1 
 N oW 
Digital communications in AWGN channel
Shannon equation
C
[bps/Hz]
CATV (5MHz, 80Mbps)
위성 TV (5MHz, 30Mbps)
지상파 TV (5MHz, 20Mbps)
 1.59
Eb
No
[dB]
Conclusion
Information amount
I ( x)   log 2 p [bits]
Entropy
H ( X )   p( x)I ( x)
all x
Shannon’s channel capacity
S
C  W log 2 (1  )
N
 Digital communications, digital multimedia!!
Q&A