資訊理論
Ch4: Channel
授課老師: 陳建源
Email:[email protected]
研究室:法401
網站 http://www.csie.nuk.edu.tw/~cychen/
Ch4: Channel
4. 1 Introduction
Communication system
Source
Coder
Channel
Decoder
Recipient
radio, optical fibre
The input alphabet of the channel is the output alphabet of the coder
The output alphabet of the channel is the input alphabet of the decoder
The output alphabet of the channel may not the same as its the input alphabet
Ch4: Channel
4. 1 Introduction
noiseless
noisy
a
b
Channel
a
Channel
A noisy channel is characterized by the probability that a given
output letter stems from an input letter
Memory : the output letter depends upon a sequence of input letters
b1, b2,…
Ch4: Channel
4. 2 Capacity of a memoryless channel
A
ai
B
bi
Channel
A memoryless channel is completely specified by giving P(bs|ak),
s=1,…,r, k=1,…,n.
r
 P(b s | ak )  1
transition probability
s 1
The probability of the output letter bs
n
P(b s )   P(b s | a k )P(a k )
k 1
n
P(b s )   P(b s  a k )
k 1
Ch4: Channel
4. 2 Capacity of a memoryless channel
Mutual information between the input and output
P(b s  a k )
I(A, B)   P(b s  a k )log
P(b s )P(a k )
k 1 s 1
n
r
n
r
  P(b s  a k )log
k 1 s 1
P(b s | a k )
P(b s )
If the transition probabilities are fixed，only the input probabilities can be
manipulated.
Ch4: Channel
4. 2 Capacity of a memoryless channel
Def: The capacity C of a memoryless channel I defined by
C  maxI(A, B)
The maximum being taken over all possible input probabilities
p1,p2, …pn while the transition probabilies P(bs|ak) are held fixed.
已知
maximum
n
 P(ak )  1
k 1
Lagrange’s multiplier
n
r
I(A, B)   P(b s  a k )log
k 1 s 1
P(b s | a k )
P(b s )
Ch4: Channel
4. 2 Capacity of a memoryless channel
Lagrange’s multiplier
已知
maximum
n
 P(ak )  1
k 1
n
I(A, B) -   P(a k )
k 1
偏微分得到
r
 P(b s | ak )log
s 1
if P(ak )  0 for all k
P(b s | a k )
   loge
P(b s )
C    loge
Ch4: Channel
4. 2 Capacity of a memoryless channel
Example 4.2a
1-ε
0
0
ε
A
a1=0
a2=1
ε
1
1
B
b1=0
b2=1
1-ε
令
p1  P(a1 ); p2  P(a2 )
n
對
I(A, B) -   P(a k )
之
P(ak )  pk
偏微分
k 1
2
2
2
P(b s | a k )
即對  P(b s | a k ) p k log
   pk
P(b s )
k 1 s 1
s 1
Ch4: Channel
4. 2 Capacity of a memoryless channel
Example 4.2a
2
2
P(b s | a k )
 P(b s | ak ) pk log P(b )    pk
s
k 1 s 1
s 1
之
P(ak )  pk
2
偏微分得到
2
 P(b s | ak )log
s 1
P(b s | a k )
 loge -  =0
P(b s )
得到
C    loge
Ch4: Channel
4. 2 Capacity of a memoryless channel
Example 4.2a
2
  P(b s | a k )log
C    loge
s 1
 1 -  log
當p1=1/2, p2=1/2
K=1
P(b s | a k )
P(b s )
1- 

  log
P(b 1 )
P(b 2 )
P(b1 )  1/2, P(b 2 )  1/2
1 -  log 1 -    log

1/2
1/2
 1 -  log 1 -    1 -    log   
 1   log   1 -  log 1 -  
C  1   log   1 -  log 1 -  
K=2 亦同
Ch4: Channel
4. 2 Capacity of a memoryless channel
Example 4.2b
1-ε
0
A
a1=0
a2=1
0
ε
ε
1
1
1-ε
B
b1=0
b2=1
b3=2
2
已知
得到
3
P(b s | a k )
 P(b s | ak )log P(b )  loge -  =0
s
s 1
C    loge 
3
 P(b s | ak )log
s 1
P(b s | a k )
P(b s )
Ch4: Channel
4. 2 Capacity of a memoryless channel
Example 4.2b
1-ε
0
A
a1=0
a2=1
ε
0
ε
1
1
1-ε
B
b1=0
b2=1
b3=2
2
1
1 -  , P(b 2 )   , P(b 3 )  1 1 -  
2
2
1   log 1      log 
K=2 亦同
1

1   
P(b1 ) 
K=1
2
得到

1  

C  1   log
  log
1

1   
2
 1 
Ch4: Channel
4. 2 Capacity of a memoryless channel
Example 4.2c
1
0
A
a1=0
a2=1
a3=2
0
B
b1=0
b2=1
1/2
1
1/2
2
1
1-ε
已知
得到
2
P(b s | a k )
 P(b s | ak )log P(b )  loge -  =0
s
s 1
C    loge 
2
 P(b s | ak )log
s 1
P(b s | a k )
P(b s )
Ch4: Channel
4. 2 Capacity of a memoryless channel
Example 4.2c
1
0
A
a1=0
a2=1
a3=2
0
B
b1=0
b2=1
1/2
1
1/2
2
1
1log  1
K=1
1
2
得到
C 1
2
1
1-ε
P(b 1 ) 
1
1
, P(b 2 ) 
2
2
 P(b s | ak )log
s 1
1
1/ 2 1
1/ 2
log
 log
0
K=2
2
1/ 2 2
1/ 2
K=3
P(b s | a k )
P(b s )
1log
1
1
1
2
Ch4: Channel
4. 3 Convexity
Theorem 4.3a The mutual information is a concave function of the
input probability, i.e.
I(p (0) )  (1 -  )I(p (1) )  I(p (0)  (1   )p (1) )
for 0    1.
Ch4: Channel
4. 3 Convexity
Theorem 4.3b I(p) is a maximum (equal to the channel capacity) if, and
only if, p is such that
(a)

I(p)   for every k for which p k  0.
p k
(b)

I(p)   for every k for which p k  0.
p k
Ch4: Channel
4. 5 Uniqueness
Theorem 4.5a The output probabilities which correspond to the
capacity of the channel are unique.
Theorem 4.5b In an input which achieves capacity with the largest
number of zero probabilies, the non-zero probabilities are determined
uniquely and their number does not exceed the number of output letters.
Ch4: Channel
練習
Noiseless binary channel
1
0
A
a1=0
a2=1
1
0
1
B
b1=0
b2=1
C=1 bit
1
Noise channel with nonoverlapping output
1/2
0
A
a1=0
a2=1
1/2
1
2
1/3
1
3
2/3
4
B
b1=1
b2=2
b3=3
b4=4
C=1 bit
Ch4: Channel
練習
Noisy typewriter
0
A
a1=0
a2=1
a3=2
a4=3
1
2
1/2
1/2
1/2
1/2
1/2
1/2
3
1
2
1/2
1/2
26 字母
0
C=log13 bits
3
B
b1=0
b2=1
b3=2
b4=3
C=1 bit
Ch4: Channel
練習
Binary symmetric channel
1-ε
0
A
a1=0
a2=1
ε
0
ε
1
1
B
b1=0
b2=1
1-ε
I(A,B)=H(B)-H(B|A)
≦1-H(B|A)
 1   log  1 -  log1 -  
Ch4: Channel
練習
Binary erasure channel
1-ε
0
A
a1=0
a2=1
ε
1
ε
1
2
1-ε
I(A,B)=H(B)-H(B|A)
p=1/2
0
B
b1=0
b2=1
b3=2
 p(1   ) log p(1   )   log   (1  p)(1   ) log(1  p)(1   )  H ( B | A)
I(A,B)=1-ε
Ch4: Channel
練習
Z channel
1
0
A
a1=0
a2=1
0
B
b1=0
b2=1
1/2
1
1
1/2
1
   log e
p(0)
1
1/ 2 1
1/ 2
log
 log
   log e
2
p(0) 2
p(1)
1log
1-
1
1
log p(0)  log p(1)  0
2
2
p(0)  4 p(1) and p(0)  p(1)  1
p(0)  4/5
C  log5 - 2
Ch4: Channel
4. 6 Transmission properties
I(A, B)=H(A)-H(A|B)
the equivocation: measure of the
uncertainty as to what was sent when
observations are made on the output and
so assesses the effect of noise during
transmission.
Shannon’s theorem I.
If H(A) ≦ C, there is a code such that transmission over the channel is
possible with an arbitrarily small number of errors, i.e. the equivocation
is arbitrarily
Shannon’s theorem II.
If H(A) > C, there is no code for which the equivocation is less than
H(A)-C but there is one for which the equivocation is less than H(A)C+ε where ε is an arbitrary positive quantity.
Ch4: Channel
4. 7 Channels in cascade
B
A
Channel 1
C
Channel 2
n
P(b s ) 
 P(b s | a k )P( a k )
k 1
P(b)  P(b | a )P(a)
P(c)  P(c | b )P(b)
P(c)  P(c | b )P(b | a)P(a)
Ch4: Channel
4. 7 Channels in cascade
A
C
B
3/4
0
3/4
0
1/4
1/4
1/4
1/4
1
1
3/4
1
3/4
3/4 1/4  3/4 1/4  10/16
1/4 3/4  1/4 3/4   6/16


 
A
6/16  5/8 3/8

10/16  3/8 5/8
C
5/8
0
0
3/8
3/8
1
1
5/8
0
Ch4: Channel
4. 7 Channels in cascade
A
C
5/8
0
0
3/8
3/8
1
1
5/8
5
5/8 3
3/8
log
 log
   log e
8
P(0) 8
P(1)
3
3/8 5
5/8
log
 log
   log e
8
P(0) 8
P(1)
C
P(0)  P(1)
3
3/8 5
5/8 3
5
log
 log
 log 3  log 5  2
8
1/ 2 8
1/ 2 8
8
5 3
5
5 3
3 5
3
C  log 2  H( , )  1  log  log  log 5  log 3  2
8 8
8
8 8
8 8
8
Ch4: Channel
4. 7 Channels in cascade
The transition probabilities pjk of an infinite cascade are given by

0

1
2

1
 2
1
2

1
0
2

1
0

2
1
2
1
3

1
3

1
 3
1 1
3 3

1 1
3 3

1 1
3 3 