Chapter 9
Fundamental limits in information theory
(資訊理論)
If the entropy (熵) of the source (資訊源) is less than the capacity (容量) of the
channel(通道), then error-free communication over the channel can be achieved.
9.1 Uncertainty (不確定性), information (資訊) and entropy
Assume that the symbols emitted by the source during successive signaling
intervals are statistically independent (統計獨立).
=> discrete memoryless sources (DMS) (離散無記憶資訊源)
The source output is modeled as a discrete random variable, S , which takes
on symbols from a fixed finite alphabet (符號系統),
A  {s 0 , s1 ,..., s K 1}
(9.1)
with probabilities
P( S  s k )  p k
k  0,1,..., K  1
(9.2)
Define the amount of information, which is related to the inverse of the
probability of occurrence, gained after observation (觀察) the event S  s k as the
logarithmic function
1
(9.3)
I ( s k )  log( )
pk
Important properties:
1. I ( s k )  0
for p k  1
2. I ( s k )  0
for 0  p k  1
3. I ( s k )  I ( s i )
for p k  pi
4. I ( s k sl )  I ( s k )  I ( sl ) if s k and s l are statistically independent (統
計獨立)
-357
Using a logarithm to base 2 is the standard practice today. The resulting unit
of information is called the bit (位元).
I ( s k )  log 2 (
1
)   log 2 p k
pk
(9.4)
When p k  1 / 2 , we have I ( s k )  1 bit
The I ( s k ) is a discrete random variable (離散隨機變數), the mean (平均值)
of I ( s k ) is given by
H ( A)  E[ I ( sk )]
K 1
  pk I ( s k )
(9.5)
k 0
K 1
  pk log 2 (
k 0
1
)
pk
The important quantity H ( A) is called the entropy (熵). It is a measure of
the average information content per source symbol (每個資訊源所含的平均
資訊量).
** I ( s k )  0
for 0  p k  1 => H ( A)  E[ I ( s k )]  0
Some properties of entropy
(1)
(2)
(3)
0  H ( A)  log 2 K
(9.6)
H ( A)  0 , if and only if p k  1
(9.7)
H ( A)  log 2 K , if and only if p k  1 / K
-358
for all k
(9.8)
proof:
x  0 , equality holds only at the point x  1 . Consider any two
probability distributions { p0 , p1 ,, pK 1} and {q 0 , q1 ,  , q K 1 }
ln x  x  1,
K 1
 p k log 2 (
k 0
qk
q
1 K 1
)
p k ln( k )

pk
ln 2 k  0
pk
q
1 K 1

p k ( k  1)

ln 2 k  0
pk
1 K 1

 (q k  p k )
ln 2 k  0
(9.9)
K 1
1  K 1


  qk   pk   0
ln 2  k  0
k 0

We thus have the fundamental inequality
K 1
 pk log 2 (
k 0
qk
)0
pk
(9.10)
where the equality holds only if p k  q k for all k.
Suppose we next put
qk 
K 1
q
k 0
k
log 2 (
1
, k  0,1,  K  1
K
1
)  log 2 K
qk
=> (9.8)得證
Also, the use of (9.11) and (9.10) yields
K 1
p
k 0
k
log 2 (
1
)0
Kpk
or
K 1
 pk log 2 (
k 0
K 1
1
)   p k log 2 ( K )  0
pk
k 0
K 1
=>
p
k 0
k
log 2 (
1
)  log 2 ( K ) (9.6) 得證
pk
-359
(9.11)
Ex9.1. Entropy of binary memoryless source
Consider p1  1  p 0 ,
1
H ( A)  E[ I ( sk )]   pk log 2 (
k 0
When p 0  1 or p1  1
1
)   p0 log 2 p0  (1  p0 ) log 2 (1  p0 ) bits
pk
=> H ( A)  0 ;
when p 0  p1  1 / 2 , the entropy attains its maximum value, H max  1 bit.
圖 9.1 Entropy function H ( p0 ) 。
-360