Applied Mathematical Sciences, Vol. 8, 2014, no. 138, 6881 - 6888
HIKARI Ltd, www.m-hikari.com
http://dx.doi.org/10.12988/ams.2014.47569
On Bounds of a Generalized ‘Useful’ Mean
Codeword Length
Sonali Saxena*
*Post Doctoral Fellow of National Board of Higher Mathematics(NBHM)of
Department of Atomic Energy(DAE),India
Keerti Upadhyay
Jaypee University of Engineering and Technology
A.B. Road, Raghogarh – 473226
Dist. Guna – M.P., India
D.S. Hooda
Jaypee University of Engineering and Technology
A.B. Road, Raghogarh – 473226
Dist. Guna – M.P., India
Copyright © 2014 Sonali Saxena, Keerti Upadhyay and D. S. Hooda. This is an open access
article distributed under the Creative Commons Attribution License, which permits unrestricted
use, distribution, and reproduction in any medium, provided the original work is properly cited.
Abstract
In the present paper a new generalized ‘useful’ mean codeword length is defined
and characterized a new generalized information measure by obtaining bounds ofa
new generalized ‘useful’ mean codeword length in terms of a new generalized
information measure using Lagrange’s Multiplier method. The Shannon’s
Noiseless coding theorem is verified by considering Huffman and Shannon -Fano
coding schemes on taking empirical data..
Mathematics Subject Classification: 94A15, 94A17
Keywords: Kraft’s inequality, Utility distribution, Lagrange’s Multiplier method,
Huffman Codes, Shannon- Fano Codes, Shannon’s Noiseless coding theorem
6882
Sonali Saxena, Keerti Upadhyay and D. S. Hooda
1. Introduction
Information theory has ideas that are widely applicable to situations remote
from its original inspiration. The applicability of ideas is not exact; however, they
are very useful. One of the best applications of information measure is in noiseless
coding theorem which gives the bounds for suitable encoding of information in
terms of information measure.
Let a finite set of n source symbols X  x1 , x 2 , , x n  be encoded using
alphabet of D  2 symbols; then it has shown Feinstein [4] that there is a uniquely
decipherable/ instantaneous code with lengths l1 , l 2 , , l n if and only if the
following Kraft’s inequality [9] is satisfied:
n
D
 li
1
(1.1)
i 1
n
If L   pi li be the average codeword length then for a code which satisfies (1.1),
i 1
it has been shown that
L  H  P ,
(1.2)
with equality if and only if li   log pi , for i  1,2,, n . This is Shannon’s
noiseless coding theorem for a noiseless channel.
Shannon-Fano coding is less efficient than Huffman coding, but we have an
advantage that we can go directly from the probability pi to the codeword length
li .
Belis and Guiasu [2] observed that a source is not completely specified by the
Probability distribution P over the source alphabet X, in the absence of qualitative
character. So it can be assumed that the source alphabet letters are weighted
according to their importance or utilities.
Let U  u1 , u 2 , , u n  be the set of positive real numbers, where u i is the
utility of the outcome x i having probability p i .The utility u i in general
independent of p i , the probability of encoding of source symbol x i .
The information source is thus given by
 x1 x2  xn 
S   p1 p 2  p n , ui  0, 0  pi  1
for each i and
u1 u 2 u n 
n
p
i 1
i
1
(1.3)
Belis and Guiasu [2] introduced the “quantitative – qualitative” measure of infor-
On bounds of a generalized ‘useful’ mean codeword length
6883
mation
n
H P; U    u i p i log p i ,
(1.4)
i 1
which can be taken as a measure for the average quantity of ‘valuable’ or ‘useful’
information provided by the information source (1.3).
Guiasu and Picard [5] considered the problem of encoding the letter output by the
source (1.3) by means of a single letter prefix code whose codeword
w1 , w2 , , wn are of lengths l1 , l 2 , , l n , respectively and satisfy the Kraft’s
inequality (1.1). They introduced the following ‘useful’ mean length of code by
attaching utility distribution with probability distribution of a set of source
symbols:
n
Lu 
u
i 1
n
i
u
j 1
pi li
.
j
(1.5)
pj
Further they derived a lower bound for (1.5). However Longo [10] interpreted
(1.5) as the average transmission cost of letters xi and derived the bounds for
this cost function.
Campbell [3] introduced the exponentiated mean codeword length given by
1
n

LUD  t   log  pi Dtli  .
(1.6)
t
 i 1

Where LUD average codeword length for the uniquely decodable code, D is
represents the size of the alphabet and li is the codeword length associated with
xi of X . He proved the following noiseless coding theorem
H  X   LUD  t   H  X   1 ,
Under the condition
n
D
 li
 1,
(1.7)
(1.8)
i 1
where H  X  is the Renyi entropy of order  
1
and li is the codeword
1 t
length corresponding to source symbol xi .
Taneja et al. [12] defined the ‘useful’ average code lengths of order t as given
below:
 n
tli 
  u i pi D 
1
 ,   1 or t  1   0  t   .
Lu t   log D  i 1 n
(1.9)



t
t 1
 u j p j 
 j 1

6884
Sonali Saxena, Keerti Upadhyay and D. S. Hooda
Evidently, when t  0 , (1.9) reduces to (1.5). They derived the bounds for the
cost function (1.9) in terms of a generalized ‘useful’ information measure.
Hooda and Bhaker [7] have studied one of its generalizations in terms of  and
 given as below:
 1 
 n
li 
 

  ui pi D    

 .
(1.10)
L  P;U  
log D  i 1 n


1

 uj pj

j 1


They have studied the lower and upper bounds of codeword length (1.7) in term
of generalized measure.
Aczel and Daroczy [1] generalized entropy of order  and type  is given by:
 n    1 
  pi

1

i 1

.
(1.11)
H  P  
log D
1
 n  
  pj 
 j 1

It may be noted (1.11) was also characterized by Kapur [8] following different
method.
In the present paper we define a new generalized ‘useful’ mean codeword
length andobtain the generalized ‘useful’ measure of information by using
Lagrange’s multiplier method in section 2. In section3 we verify the Shannon’s
noiseless coding theorem on average codeword lengthin cases of Huffman and
Shannon-Fano coding schemes on taking empirical data.
2. A Generalized ‘Useful’ Mean Code Word Length and its
Bounds
In this section we define a new generalized ‘useful’ mean codeword length as
follows:
   
 n
li 


  ui pi D    

 , 0    1, 0    1,    .
(2.1)
L  P;U  
log D  i 1 n


 
ui pi



i

1


where l i is the length of codeword xi and pi is the probability of occurrence
of codeword xi .
Theorem 2.1For all uniquely decipherable codes generalized ‘useful’mean

codeword length L  P;U  defined in (2.1) satisfies the following relation
On bounds of a generalized ‘useful’ mean codeword length
H  P;U   L  P;U   H  P;U   1,
6885
(2.2)
   2   2 


 


 

u
p

i i
where H  P;U  
log D 
 , 0    1, 0    1,    , (2.3)
 

 ui pi 


2
under the generalized Kraft’s inequality given by
 1
 u i p i D  li   u i p i .
(2.4)
u i p i 1 D li
, for each i  1,2, , n
 u i pi
Substituting (2.5) in (2.1) we have
  2         




  ui pi  xi    


L  P;U  
log D 


 



u
p
i i


Thus we have to minimize (2.6) subject to the following constraint:
 u i pi 1 D li  1
x

 i
 u i pi
Proof:Let us choose xi 


Since L  P;U  is a pseudo convex function for each
(2.5)
(2.6)
(2.7)
i  1,2,, n , therefore,
we can obtain the minimum value of L  P;U  by applying the Lagrange’s
multiplier method.
Let us consider the corresponding Lagrangian as given below:

 2         




  u i p i  xi   

 n

L
log 


  xi  1 .


 


 u i pi     i 1


Differentiating w.r.t. x i and equating to zero, we get
 dL 
u p x 1


  i i i   0
 u i pi
 dx i     1
It implies
up
1
 xi  c i i , where c   0
u p
i
ii
or xi  cq i , where q i 

u i pi
.(2.8) (2.8) together with (2.5) gives
 u i pi
6886
Sonali Saxena, Keerti Upadhyay and D. S. Hooda
u i p i 1 D li
u p
 i i
 u i pi  u i pi
It implies
D li  pi2 
Taking log of both sides, we have
 l i  log pi2 
or li  log D pi 2
(2.9)
   
Multiplying both sides of (2.9) by 
  0 as    we get
  
   

 
   2 
   

l i  log D p i 
  
(2.10)
From (2.1) and (2.10), we get the minimum value of
L ( P;U ) as follows:
  2   2 

  ui pi 



L ( P;U ) min 
log 
  H ( P;U ) .
 
u
p
i i 



again, since l i is always integral value in (2.9), so it must be equal to
li  ai   i
2
where ai  log D pi 2  and 0   i  1 .
Putting (2.12) in (2.1), we have
  
   

   2 
 i 
 

  
  
u
p
p
D




i i
i

L  P;U  
log D 

 
 ui pi




2


2


2



  u i pi 



log 
  i
 
 u i pi 



Since 0   i  1, therefore, (2.13) reduce to
  2   2 

u
p


 i i    1  H   P;U   1 .
L  P;U  
log 

 
 ui pi 



Hence from (2.11) and (2.14), we get
H  P;U   L  P;U   H  P;U   1, which is (2.2).
(2.11)
(2.12)
(2.13)
2
(2.14)
On bounds of a generalized ‘useful’ mean codeword length
6887
Hence by using optimization technique we get the new generalized‘useful’
information measure given by (2.3).
3. Illustration
In this section we illustrate the veracity of the theorem 2.1 by taking empirical
data as given in table (3.1) and (3.2) on the lines of Hooda et al.[6].
Table- 3.1
Huffman
codeword
s
li
ui
0.3846
0.1795
0.1538
0
100
101
1
3
3
1
3
2
0.1538
110
3
2
0.1282
111
3
4
pi


0.2
0.3
H  P;U 
H  P;U  L  P;U    L  P;U  100%

6.0157
6.3764
94.34%
Table -3.2
pi
Shannon
Fanocodew
ords
li
ui


0.3846
00
2
1
0.2
0.3
0.1795
01
2
3
0.1538
10
2
2
0.1538
110
3
2
0.1282
111
3
4
H


 P;U 
6.0157
L  P;U 


6.5516

H  P;U 
L  P;U 
91.82%
From table (3.1) and (3.2) we infer the following:
1. Theorem 2.1 holds in both cases of Shannon -Fano codes and Huffman codes.
2. Huffman mean codeword length is less than Shannon –Fano mean codeword
length.
3. Coefficient of efficiency of Huffman Codes is greater than Shannon -Fano
Codes i.e. it is concluded that Huffman coding Scheme is more efficient than
Shannon -Fano coding scheme
100%
6888
Sonali Saxena, Keerti Upadhyay and D. S. Hooda
Conclusion
The various authors have characterized the generalized information measures
by different methods, but we have introduced a new generalized ‘useful’ mean
codeword length and studied its bounds in terms of the new generalized measure of
information by optimization technique.
Further, we have established the Shannon’s Noiseless Coding theorem with
the help of two different coding techniques by taking experimental data and prove
that Huffman coding scheme is more efficient than Shannon-Fano coding scheme.
References
[1] J.Aczel and Z. Daroczy, Uber Veallegemeineste Quasilinear Mittelwerte, Die
Mitgrewincbtsfunktionen Gebilelet,Sind. Public Mathematics Debrecen,10(1963),
171-190.
[2] M.Belis and S. Guiasu, A Quantitative-Qualitative Measure of Information in
Cybernetic Systems,IEEE Transfer Information Theory,14(1968), 593-594.
[3]L.L. Campbell, A coding theorem and Renyi’s Entropy, Information and
control 8(1965) 423-429
[4] A.Feinstein, Foundation of Information Theory, Mc Grew-Hill, New
York(1958).
[5] S.Guiasu and C.F. Picard, Borne Inferieure De La Longueur De Certain
Codes,C.R. Academy Science Paris, 273(1971), 248-251.
[6]D.S.Hooda, Keerti Upadhyay and Sonali Saxena, Characterization of a
Generalized Information Measure by Optimization Technique, International
Journal of Engineering Research and Management Technology, I(2014),
302-311.
[7] D.S.Hooda and U.S.Bhaker, A Profile on Noiseless Coding Theorem,
International Journal of Management and Systems, 8(1992), 76-85.
[8] J.Kapur, Generalized Entropy of Order  and Type  , Math.Seminar
(Delhi),4(1967).
[9] L.G.Kraft, A Device for Quantizing, Grouping and Coding Amplitude
Modulated Pulses, M.S.Thesis, Electrical Engineering Department, MIT, (1949).
[10] G.Longo,A Noiseless Coding Theorem for Sources having Utilities,SIAM
Journal Applied Mathematics,30 (1972), 739-748.
[11]C.E.Shannon, A Mathematical Theory of Communication, Bell System
Technology Journal27 (1948), 379-423.
[12]H.C.Taneja, D.S. Hooda and R.K.Tuteja,Coding Theorem on Generalized
‘useful’ Information, Soochow Journal of mathematics, 11 (1985), 123-131.
Received: July 11, 2014