Solutions to Assignment-2
MOOC-Information Theory
1. Which of the following is a prefix-free code?
a) 01, 10, 101, 00, 11
b) 0, 11, 01
c) 01, 10, 11, 00
d) None of the above
Solution:- The codewords of (a) are not prefix-free as the codeword 10 is
clearly a prefix of 101. Similarly in (b), the codeword 0 is prefix of the codeword
01. The codewords of (c) are prefix-free as no codeword is a prefix of another
codeword. Hence the correct answer is (c).
2. Which of the following is a uniquely decodable code but not prefix-free?
a) 01, 10, 101, 00, 11
b) 0, 11, 01
c) 01, 10, 11, 00
d) None of the above
Solution:- As explained in solution (1), codes (a) and (b) are not prefix-free.
The unique decodability of the code can be tested as below
(a) 01, 10, 101, 00, 11
S0
S1 S2
01
1
0
10
01
101
1
00
11
S3
1
0
01
S4
0
01
1
1
Since, the codeword 01 is present in both S0 and S2 (S3 , S4 and so on). Hence,
it is not uniquely decodable.
(b) 0, 11, 01
S0 S1 S2
0
1
1
11
01
S3
1
1
Here, none of the codewords are present in other columns. Hence, it is a
uniquely decodable code. ∴ The correct answer is (b).
3. Given a binary variable length code consisting of codewords of following
length w1 = 1, w2 = 2, w3 = 2, w4 = 3. Which of the following statement is
correct?
a) There exist a prefix-free code of given lengths
b) There exist many prefix-free codes of given lengths
c) There exist a uniquely decodable code of given lengths
d) None of the above
Solution:- The criteria for existence of a prefix-free code is given by Kraft’s
inequality
K
X
D−wi ≤ 1
i=1
P4
Here, K = 4 and D = 2. Substituting the values of wi ’s, we get i=1 D−wi =
1.125 which is greater than 1 and hence no prefix-free or uniquely decodable
code exist. So, the correct answer is (d).
4. A discrete memoryless source emits a symbol U that takes five different values u1 , u2 , u3 , u4 , u5 with probabilities 0.25, 0.25, 0.25, 0.125, 0.125 respectively. A binary Shannon-Fano code consists of codewords of following lengths
a) 3, 3, 3, 4, 4
b) 2, 2, 2, 3, 3
c) 1, 1, 1, 2, 2
2
d) None of the above
Solution:- The length, wi of the codeword ui is given as
wi = d− logD (PU (ui ))e
Here, D = 2 (binary codes). So, substituting the respective probabilities in the
above equation we get the lengths of codewords as 2, 2, 2, 3, 3. Hence, correct
answer is (b).
5. Given a discrete memoryless source that emits a symbol U , that takes five
different values u1 , u2 , u3 , u4 , u5 with probabilities 0.25, 0.25, 0.25, 0.125, 0.125
respectively. An optimal ternary prefix-free code is designed to represent U .
What is the expected length of the code?
a) 1.4
b) 1.5
c) 2.25
d) None of the above
Solution:- Here, D = 3 (Ternary Code) and K = 5. So, the number of
) = 0. So, the tree diagram of the
the unused leaves = Remainder( (K−D)(D−2)
D−1
optimal ternary prefix-code is represented as below.
Hence, the expected length of the code using path length lemma is given as
E[W ] = 1 + 0.5 = 1.5 bits (Probabilities of the nodes is directed in red). Thus,
the correct answer is (b).
3
6. Given a discrete memoryless source that emits 6-ary random variable U ,
{u1 , u2 , u3 , u4 , u5 , u6 } with probabilities {0.25, 0.25, 0.125, 0.125, 0.125, 0.125} respectively. Which of the following is not an optimal binary prefix-free code for
this source? (Prefix-free code is represented by binary tree.)
Figure 1: 6(a)
Figure 2: 6(b)
Figure 3: 6(c)
Figure 4: 6(d)
Solution:- It can be easily checked that all the above codes are prefix-free.
Now, the optimal code is one in which only the least likely (i.e, less probable)
leafs/nodes are connected. In (a), we observe that at each depth only least likely
leafs/nodes are joined. So, (a) is an optimal code. Similarly, we can observe in
(c) and (d) that the least likely leafs/nodes are only connected at each depth of
the tree. Now, in (b) we observe that the code tree has connected a node with
probability 0.5 with another node with probability 0.25 at depth level 1, while
another node with probability 0.25 is left to be connected at the root. Hence,
the code in (b) is not optimal. Thus, the correct answer is (b).
4
7. Given a discrete memoryless source that emits 6-ary random variable U ,
{u1 , u2 , u3 , u4 , u5 , u6 } with probabilities {0.25, 0.25, 0.125, 0.125, 0.125, 0.125} respectively. Which of the following is an optimal ternary prefix-free code for this
source?
Figure 5: 7(a)
Figure 6: 7(b)
Figure 7: 7(c)
7(d) None of the above
Solution:- It can be easily checked that all the above codes are prefix-free.
Here, D = 3 (ternary codes) and K = 6. Now, the number of the unused leaves
= Remainder( (K−D)(D−2)
) = 1. Now, an optimal code should have any unused
D−1
leaves only at the maximum depth of the D-ary tree. Since, the code in (b) is
of depth 2 but has a missing leaf at depth 1. So, (b) can’t be an optimal code.
Similarly, code in (c) is of depth 3 but also has a missing leaf at depth 1 from
the root. So, (c) can’t be an optimal code. Now, the code in (a) has a missing
leaf at the maximum depth. Hence, the code shown in (a) is optimal. Thus, the
correct answer is (a).
5
8. Given a discrete memoryless source that emits a symbol U , that takes
four different values u1 , u2 , u3 , u4 with probabilities 0.2, 0.2, 0.3, 0.3 respectively.
Which of the following is an optimal binary block to variable length code for
this source but not Huffman code. Codewords of u1 , u2 , u3 , u4 are respectively?
a) 00, 01, 10, 11
b) 00, 11, 01, 10
c) 0, 10, 110, 111
d) 111, 110, 10, 0
Figure 8: 8(a)
Figure 9: 8(b)
Figure 10: 8(c)
Figure 11: 8(d)
Solution:- The expected length of the codes by path length lemma for (a)=2,
(b)=2, (c)=2.4 and (d)=2.1. Thus, the codewords of (c) and (d) are not optimal. The code in (a) is a Huffman code as the two least likely codewords of
(a) differs by 1 bit only (i.e 00, 01). Whereas the code in (b), although being
optimal has it’s two least likely codewords differ by 2 bits (i.e 00, 11). Hence,
the code in (b) is an optimal, non-Huffman code. Thus, the correct answer is
(b).
6
9. Consider a bag containing 100 balls of red and blue colors. Let, Xi = 1
or 0 depending upon whether the i-th ball is of red color or blue color. Let,
Xi be independent with Pr{Xi = 1} = 0.25. You are asked to find the set of
all red balls. Any yes/no question is admissible. What is the minimum average
number of questions required to identify the set of red balls?
a) 80
b) 81
c) 82
d) None of the above
Solution:- Since, it’s a binary based setup. So, Pr{Xi = 0} = 0.75. The
minimum average number of questions required to identify the set of red balls
is given as
N = H(X1 , X2 , . . . , X100 )
Since, the Xi ’s are independent so we have
N=
i=100
X
H(Xi )
i=1
where H(Xi ) = −Pr{Xi = 0} log2 (Pr{Xi = 0}) − Pr{Xi = 1} log2 (Pr{Xi =
1}) = − 41 log2 14 − 34 log2 34 = 0.8113. ∴ N = 81.13 ≈ 82. Hence, the correct
answer is (c).
10. Which of the following cannot be a valid Huffman code?
a) 01, 10
b) 0, 10, 11
c) 01, 10, 11, 00
d) None of the above
Solution:- The binary tree diagram is shown below. The unused leafs of
each tree is shown by an empty circle, while the used leafs are shown with dotted circle. The codewords of (a) is shown to have two unused leaves and thus
the codeword length can be shortened as shown in the tree diagram with the
7
Figure 12: 10(a)
Figure 13: 10(b)
Figure 14: 10(c)
10(d) None of the above
red arrow. Hence, the codewords of (a) cannot be of a Huffman code since, it
is not optimal. While, the codewords of (b) and (c) are shown to be using all
the leaves and thus are valid Huffman codewords. Hence, correct answer is (a).
8