EEM 465
FUNDAMENTALS of DATA COMMUNICATIONS
Lecture 2
Lecturer
Assist.Prof.Dr. Nuray At
Data Compression
2
Data compression can be achieved by assigning short descriptions to the most frequent outcomes of the data source, and necessarily longer descriptions to the less frequent outcomes.
Codes
Definition: A source code C for a random variable X is a mapping from to , the set of finite‐length strings of symbols from a D‐ary alphabet. Let
the codeword corresponding to x
the length of For example, C(red)=00, C(blue)=11 is a source code for with alphabet 1
3
Definition: The expected length L(C) of a source code C(x) for a random variable X with pmf p(x)
is the length of the codeword associated with x.
Example 1: Let X be a random variable with the following distribution and codeword assignment
The entropy H(X) of X is 1.75 bits
The expected length L(C) of this code is also 1.75 bits
4
Example 2: Consider
The entropy is 1.58 bits
The expected length L(C) of this code is 1.66 bits. Here
Let denote Definition: A code is said to be nonsingular if every element of the range of X maps into a different string in
We usually wish to send a sequence of values of X.
2
5
Definition: The extension C* of a code C is the mapping from finite length strings of X to finite‐length strings of D
where indicates concatenation of the corresponding codewords.
For example, if C(x1)=00 and C(x2)=11, then C(x1 x2)=0011
Definition: A code is called uniquely decodable if its extension is nonsingular.
Definition: A code is called a prefix code or an instantaneous code if no codeword
is a prefix of any other codeword.
An instantaneous code can be decoded without reference to future codewords
since the end of a codeword is immediately recognizable.
Classes of Codes
6
3
7
Example 3: Classify the codes uniquely decodable and/or prefix?
as nonsingular, 8
Example 4: Classify the codes decodable and/or prefix?
as nonsingular, uniquely 4
Kraft Inequality
9
We wish to construct instantaneous codes of minimum expected length to describe a given source. The set of codeword lengths possible for instantaneous codes is limited by the following inequality.
Theorem: For any prefix code over an alphabet size D, the codeword lengths must satisfy the inequality
Conversely, given a set of codeword lengths that satisfy this inequality, there exists
an instantaneous code with these word lengths.
Optimal Codes
10
Consider the problem of finding the prefix code with the minimum expected length. Minimize
over all integers satisfying
Neglect the integer constraint on and assume equality in the constraint. Hence, we can write the constrained minimization using Lagrange multipliers
Differentiating with respect to and setting the derivative to 0: 5
11
Substituting this in the constraint, we find , and hence
yielding optimal code lengths This noninteger choice of codeword lengths yields expected codeword length
Theorem: The expected length L of any instantaneous D‐ary code for a random variable X is greater than or equal to the entropy, i.e.,
with equality iff
Bounds on the Optimal Codelength
The choice of word lengths 12
yields L=H. Since may not equal
an integer, we round it up to give integer word‐length assignments,
where is the smallest integer greater than or equal to x. This choice of codeword lengths satisfies
Multiplying by pi and summing over i, we obtain 6