Convolutional Codes
ECEN 5682 Theory and Practice of Error Control
Codes
Convolutional Codes
Peter Mathys
University of Colorado
Spring 2007
Peter Mathys
ECEN 5682 Theory and Practice of Error Control Codes
Convolutional Codes
Basic Definitions, Convolutional Encoders
Encoder State Diagrams
Viterbi Decoding Algorithm
Basic Definitions, Convolutional Encoders
Linear (n, k) block codes take k data symbols at a time and encode
them into n code symbols. Long data sequences are broken up into
blocks of k symbols and each block is encoded independently of all
others. Convolutional encoders, on the other hand, convert an
entire data sequence, regardless of its length, into a single code
sequence by using convolution and multiplexing operations. In
general, it is convenient to assume that both the data sequences
(u0 , u1 , . . .) and the code sequences (c0 , c1 , . . .) are semi-infinite
sequences and to express them in the form of a power series.
Peter Mathys
ECEN 5682 Theory and Practice of Error Control Codes
Convolutional Codes
Basic Definitions, Convolutional Encoders
Encoder State Diagrams
Viterbi Decoding Algorithm
Definition: The power series associated with the data sequence
u = (u0 , u1 , u2 , . . .) is defined as
u(D) = u0 + u1 D + u2 D 2 + . . . =
∞
X
ui D i ,
i=0
where u(D) is called the data power series. Similarly, the code
power series c(D) associated with the code sequence
c = (c0 , c1 , c2 , . . .) is defined as
2
c(D) = c0 + c1 D + c2 D + . . . =
∞
X
ci D i .
i=0
The indeterminate D has the meaning of delay, similar to z −1 in
the z transform, and D is sometimes called the delay operator.
Peter Mathys
ECEN 5682 Theory and Practice of Error Control Codes
Convolutional Codes
Basic Definitions, Convolutional Encoders
Encoder State Diagrams
Viterbi Decoding Algorithm
A general rate R = k/n convolutional encoder converts k data
sequences into n code sequences using a k × n transfer function
matrix G(D) as shown in the following figure.
u(D)
D
e
m
u
x
u(1) (D)
..
.
(k)
u (D)
Convolutional
Encoder
G(D)
c(1) (D)
..
.
c
(n)
(D)
M
u
x
c(D)
Fig.1 Block Diagram of a k-Input, n-Output Convolutional Encoder
Peter Mathys
ECEN 5682 Theory and Practice of Error Control Codes
Convolutional Codes
Basic Definitions, Convolutional Encoders
Encoder State Diagrams
Viterbi Decoding Algorithm
The data power series u(D) is split up into k subsequences,
denoted u (1) (D), u (2) (D), . . . , u (k) (D) in power series notation,
using a demultiplexer whose details are shown in the figure below.
u(D) −→
(u0 , u1 , . . . , uk−1, uk , uk+1, . . . , u2k−1 , u2k , u2k+1 , . . . , u3k−1 , . . .)
(1)
(1)
...
u0
(2)
u0
(2)
u1
...
(k)
u0
(1)
...
u1
u2
(2)
u2
...
. . . u(k)
1
...
. . . −→ u(1) (D)
...
. . . −→ u(2) (D)
..
.
...
(k)
u2
. . . −→ u(k) (D)
Fig.2 Demultiplexing from u(D) into u(1) (D), . . . , u(k) (D)
Peter Mathys
ECEN 5682 Theory and Practice of Error Control Codes
Convolutional Codes
Basic Definitions, Convolutional Encoders
Encoder State Diagrams
Viterbi Decoding Algorithm
The code subsequences, denoted by c (1) (D), c (2) (D), . . . , c (n) (D)
in power series notation, at the output of the convolutional
encoder are multiplexed into a single power series c(D) for
transmission over a channel, as shown below.
c(1) (D) → c(1)
0
c(2) (D) →
..
.
c(n) (D) →
(1)
...
(2)
c0
(2)
c1
...
(n)
c0
(1)
...
c1
c2
(2)
...
c2
. . . c(n)
1
...
...
...
...
...
(n)
c2
...
(c0 , c1 , . . . , cn−1 , cn , cn+1 , . . . , c2n−1 , c2n , c2n+1 , . . . , c3n−1 , . . .)
→ c(D)
Fig.3 Multiplexing of c(1) (D), . . . , c(n) (D) into Single Output c(D)
Peter Mathys
ECEN 5682 Theory and Practice of Error Control Codes
Convolutional Codes
Basic Definitions, Convolutional Encoders
Encoder State Diagrams
Viterbi Decoding Algorithm
Definition: A q-ary generator polynomial of degree m is a
polynomial in D of the form
g (D) = g0 + g1 D + g2 D 2 + . . . + gm D m =
m
X
gi D i ,
i=0
with m + 1 q-ary coefficients gi . The degree m is also called the
memory order of g (x).
Peter Mathys
ECEN 5682 Theory and Practice of Error Control Codes
Convolutional Codes
Basic Definitions, Convolutional Encoders
Encoder State Diagrams
Viterbi Decoding Algorithm
Consider computing the product (using modulo q arithmetic)
c(D) = u(D) g (D) .
Written out, this looks as follows
c 0 + c1 D + c 2 D 2 + . . . =
= (g0 + g1 D + g2 D 2 + . . . + gm D m ) (u0 + u1 D + u2 D 2 + . . .) =
= g0 u0 + g0 u1 D + g0 u2 D 2 + . . . + g0 um D m + g0 um+1 D m+1 + g0 um+2 D m+2
+ g1 u0 D + g1 u1 D 2 + . . . + g1 um−1 D m + g1 um D m+1 + g1 um+1 D m+2
+ g2 u0 D 2 + . . . + g2 um−2 D m + g2 um−1 D m+1 + g2 um D m+2
..
..
..
.
.
.
+ gm u0 D m + gm u1 D m+1 + gm u2 D m+2
+ ...
+ ...
+ ...
+ ...
Thus, the coefficients of c(D) are
cj =
m
X
gi uj−i ,
j = 0, 1, 2, . . . ,
where u` = 0 for ` < 0 ,
i=0
i.e., the code sequence (c0 , c1 , c2 , . . .) is the convolution of the
data sequence (u0 , u1 , u2 , . . .) with the generator sequence
(g0 , g1 , . . . , gm ).
Peter Mathys
ECEN 5682 Theory and Practice of Error Control Codes
Convolutional Codes
Basic Definitions, Convolutional Encoders
Encoder State Diagrams
Viterbi Decoding Algorithm
A convenient way to implement the convolution
cj =
m
X
gi uj−i ,
j = 0, 1, 2, . . . ,
where u` = 0 for ` < 0 ,
i=0
is to use a shift register with m memory cells (cleared to zero at
time t = 0), as shown in following figure.
m memory cells
. . . , u2 , u1 , u0
•
g0
···
g1
g2
+
+
gm
···
+
. . . , c2 , c1 , c0
Fig.4 Block Diagram for Convolution of u(D) with g(D)
Peter Mathys
ECEN 5682 Theory and Practice of Error Control Codes
Convolutional Codes
Basic Definitions, Convolutional Encoders
Encoder State Diagrams
Viterbi Decoding Algorithm
A general k-input, n-output convolutional encoder consists of k
such shift registers, each of which is connected to the outputs via
n generator polynomials.
Definition: A q-ary linear and time-invariant convolutional encoder
with k inputs and n outputs is specified by a k × n matrix G(D),
called transfer function matrix, which consists of generator
(`)
polynomials gh (D), h = 1, 2, . . . , k, ` = 1, 2, . . . , n, as follows
 (1)

(2)
(n)
g1 (D) g1 (D) . . . g1 (D)
 (1)

g2 (D) g2(2) (D) . . . g2(n) (D)

G(D) =  .
..
.. 
.
.
. 
 ..
(1)
(2)
(n)
gk (D) gk (D) . . . gk (D)
The generator polynomials have q-ary coefficients, degree mh` , and
are of the form
(`)
(`)
(`)
(`)
(`)
gh (D) = g0h + g1h D + g2h D 2 + . . . + gmh` h D mh` .
Peter Mathys
ECEN 5682 Theory and Practice of Error Control Codes
Basic Definitions, Convolutional Encoders
Encoder State Diagrams
Viterbi Decoding Algorithm
Convolutional Codes
Define the power series vectors
u(D) = [u (1) (D), u (2) (D), . . . , u (k) (D)] ,
c(D) = [c (1) (D), c (2) (D), . . . , c (n) (D)] .
The operation of a k-input n-output convolutional encoder can
then be concisely expressed as c(D) = u(D) G(D). Each individual
output sequence is obtained as
c (`) (D) =
k
X
(`)
u (h) (D) gh (D) .
h=1
Note: By setting u (h) (D) = 1 in the above equation, it is easily
seen that the generator sequence
(`)
(`)
(`)
(`)
(g0h , g1h , g2h , . . . , gmh` h ) ,
is the unit impulse response from input h to output ` of the
convolutional encoder.
Peter Mathys
ECEN 5682 Theory and Practice of Error Control Codes
Convolutional Codes
Basic Definitions, Convolutional Encoders
Encoder State Diagrams
Viterbi Decoding Algorithm
Definition: The total memory M of a convolutional encoder is the
total number of memory elements in the encoder, i.e.,
M=
k
X
h=1
max mh` .
1≤`≤n
Note that max1≤`≤n mh` is the number of memory cells or the
memory order of the shift register for the input with index h.
Definition: The maximal memory order m of a convolutional
encoder is the length of the longest input shift register, i.e.,
m = max max mh` .
1≤h≤k 1≤`≤n
Equivalently, m is equal to the highest degree of any of the
generator polynomials in G(D).
Peter Mathys
ECEN 5682 Theory and Practice of Error Control Codes
Convolutional Codes
Basic Definitions, Convolutional Encoders
Encoder State Diagrams
Viterbi Decoding Algorithm
Definition: The constraint length K of a convolutional encoder is
the maximum number of symbols in a single output stream that
can be affected by any input symbol, i.e.,
K = 1 + m = 1 + max max mh` .
1≤h≤k 1≤`≤n
Note: This definition for constraint length is not in universal use.
Some authors define constraint length to be the maximum number
of symbols in all output streams that can be affected by any input
symbol, which is nK in the notation used here.
Peter Mathys
ECEN 5682 Theory and Practice of Error Control Codes
Convolutional Codes
Basic Definitions, Convolutional Encoders
Encoder State Diagrams
Viterbi Decoding Algorithm
Example: Encoder #1. Binary rate R = 1/2 encoder with
constraint length K = 3 and transfer function matrix
G(D) = g (1) (D) g (2) (D) = 1 + D 2 1 + D + D 2 .
A block diagram for this encoder is shown in the figure below.
+
. . . , u2 , u1 , u0
(1)
(1)
(1)
(2)
(2)
(2)
. . . , c2 , c1 , c0
•
+
+
. . . , c2 , c1 , c0
Fig.5 Binary Rate 1/2 Convolutional Encoder with K = 3
Peter Mathys
ECEN 5682 Theory and Practice of Error Control Codes
Convolutional Codes
Basic Definitions, Convolutional Encoders
Encoder State Diagrams
Viterbi Decoding Algorithm
At time t = 0 the contents of the two memory cells are assumed to
be zero. Using this encoder, the data sequence
u = (u0 , u1 , u2 , . . .) = (1, 1, 0, 1, 0, 0, 1, 1, 1, 0, 1, 0 . . .) ,
for example, is encoded as follows
u = 110100111010...
u = 110100111010...
uD = 110100111010...
uD 2 = 110100111010...
----------------c (2) = 100011101001.....
uD 2 =
c (1)
110100111010...
----------------= 111001110100.....
After multiplexing this becomes
(1)
(2)
(1)
(2)
(1)
(2)
c = (c0 c1 , c2 c3 , c4 c5 , . . .) = (c0 c0 , c1 c1 , c2 c2 , . . .)
= (11, 10, 10, 00, 01, 11, 11, 10, 01, 10, 00, 01, . . .) .
The pairs of code symbols that each data symbol generates are
called code frames.
Peter Mathys
ECEN 5682 Theory and Practice of Error Control Codes
Convolutional Codes
Basic Definitions, Convolutional Encoders
Encoder State Diagrams
Viterbi Decoding Algorithm
Definition: Consider a rate R = k/n convolutional encoder, let
u = (u0 u1 . . . uk−1 , uk uk+1 . . . u2k−1 , u2k u2k+1 . . . u3k−1 , . . .)
(1)
(2)
= (u0 u0
(k)
(1)
(2)
. . . u0 , u1 u1
(k)
(1)
(2)
. . . u1 , u2 u2
(k)
. . . u2 , . . .) ,
and let
c = (c0 c1 . . . cn−1 , cn cn+1 . . . c2n−1 , c2n c2n+1 . . . c3n−1 , . . .)
(1)
(2)
= (c0 c0
(n)
(1)
(2)
. . . c0 , c1 c1
(n)
(1)
(2)
. . . c1 , c2 c2
(n)
. . . c2 , . . .) .
Then the set of data symbols (uik uik+1 . . . u(i+1)k−1 ) is called the
i-th data frame and the corresponding set of code symbols
(cin cin+1 . . . c(i+1)n−1 ) is called the i-th code frame for
i = 0, 1, 2, . . . .
Peter Mathys
ECEN 5682 Theory and Practice of Error Control Codes
Convolutional Codes
Basic Definitions, Convolutional Encoders
Encoder State Diagrams
Viterbi Decoding Algorithm
Example: Encoder #2. Binary rate R = 2/3 encoder with constraint length
K = 2 and transfer function matrix
#
" (1)
# "
(2)
(3)
g1 (D) g1 (D) g1 (D)
1+D D 1+D
G(D) =
=
(1)
(2)
(3)
D
1
1
g2 (D) g2 (D) g2 (D)
A block diagram for this encoder is shown in the figure below.
+
(1)
(1)
(1)
. . . , u2 , u1 , u0
•
•
+
(2)
(2)
(1)
(1)
(1)
(2)
(2)
(2)
(3)
(3)
(3)
. . . , c2 , c1 , c0
+
. . . , c2 , c1 , c0
+
+
. . . , c2 , c1 , c0
(2)
. . . , u2 , u1 , u0
•
•
Fig.6 Binary Rate 2/3 Convolutional Encoder with K = 2
Peter Mathys
ECEN 5682 Theory and Practice of Error Control Codes
Convolutional Codes
Basic Definitions, Convolutional Encoders
Encoder State Diagrams
Viterbi Decoding Algorithm
In this case the data sequence
u = (u0 u1 , u2 u3 , u4 u5 , . . .) = (11, 01, 00, 11, 10, 10, . . .) ,
is first demultiplexed into u(1) = (1, 0, 0, 1, 1, 1, . . .) and
u(2) = (1, 1, 0, 1, 0, 0, . . .), and then encoded as follows
u (1) = 100111...
u D = 100111...
u (2) D = 110100...
---------c (1) = 101110....
(1)
u
u (1) = 100111...
u D = 100111...
u (2) = 110100...
---------c (3) = 000000....
(1)
(1)
D = 100111...
u (2) = 110100...
---------c (2) = 100111....
Multiplexing the code sequences c(1) , c(2) , and c(3) yields the
single code sequence
(1)
(2)
(3)
(1)
(2)
(3)
c = (c0 c1 c2 , c3 c4 c5 , . . .) = (c0 c0 c0 , c1 c1 c1 , . . .)
= (110, 000, 100, 110, 110, 010, . . .) .
Because this is a rate 2/3 encoder, data frames of length 2 are
encoded into code frames of length 3.
Peter Mathys
ECEN 5682 Theory and Practice of Error Control Codes
Convolutional Codes
Basic Definitions, Convolutional Encoders
Encoder State Diagrams
Viterbi Decoding Algorithm
Definition: Let u = (u0 , u1 , u2 , . . .) be a data sequence (before
demultiplexing) and let c = (c0 , c1 , c2 , . . .) be the corresponding
code sequence (after multiplexing). Then, in analogy to block
codes, the generator matrix G of a convolutional encoder is defined
such that
c = uG .
Note that G for a convolutional encoder has infinitely many rows
and columns.
(`)
Let G(D) = [gh ] be the transfer function matrix of a
convolutional encoder with generator polynomials
P
(`)
(`) i
gh (D) = m
i=0 gih D , h = 1, 2, . . . , k, ` = 1, 2, . . . , n, where m
is the maximal memory order of the encoder. Define the matrices
2
(1)
gi1
6 (1)
g
6 i2
Gi = 6
6 ..
4 .
(1)
gik
(2)
gi1
(2)
gi2
..
.
(2)
gik
...
...
...
3
(n)
gi1
(n) 7
gi2 7
.. 7
7,
. 5
(n)
gik
Peter Mathys
i = 0, 1, 2, . . . , m .
ECEN 5682 Theory and Practice of Error Control Codes
Convolutional Codes
Basic Definitions, Convolutional Encoders
Encoder State Diagrams
Viterbi Decoding Algorithm
In terms of these matrices, the generator matrix G can be
conveniently expressed as (all entries below the diagonal are zero)


G0 G1 G2 . . . Gm
0
0 ...

G0 G1 . . . Gm−1 Gm
0 . . .



G0 . . . Gm−2 Gm−1 Gm . . .




..
..
..
..


.
.
.
.
.
G=

G0
G1
G2 . . .



G0
G1 . . .




G
.
.
.
0


..
.
Note that the first row of this matrix is the unit impulse response
(after multiplexing the outputs) from input stream 1, the second
row is the unit impulse response (after multiplexing the outputs)
from input stream 2, etc.
Peter Mathys
ECEN 5682 Theory and Practice of Error Control Codes
Convolutional Codes
Basic Definitions, Convolutional Encoders
Encoder State Diagrams
Viterbi Decoding Algorithm
Example: Encoder #1 has m = 2,
G0 = 1 1 ,
G1 = 0 1 ,
and thus generator matrix

11 01
00 11


G = 00 00
00 00

..
..
.
.
11
01
11
00
..
.
00
11
01
11
..
.
00
00
11
01
..
.
G2 = 1 1 ,
00
00
00
11
..
.

...
. . .

. . .
.
. . .

Using this, it is easy to compute, for example, the list of (non-zero)
datawords and corresponding codewords shown on the next page.
Peter Mathys
ECEN 5682 Theory and Practice of Error Control Codes
Convolutional Codes
Basic Definitions, Convolutional Encoders
Encoder State Diagrams
Viterbi Decoding Algorithm
u = (u0 , u1 , . . .)
c = (c0 c1 , c2 c3 , . . .)
1,0,0,0,0,...
1,1,0,0,0,...
1,0,1,0,0,...
1,1,1,0,0,...
1,0,0,1,0,...
1,1,0,1,0,...
1,0,1,1,0,...
1,1,1,1,0,...
11,01,11,00,00,00,00,...
11,10,10,11,00,00,00,...
11,01,00,01,11,00,00,...
11,10,01,10,11,00,00,...
11,01,11,11,01,11,00,...
11,10,10,00,01,11,00,...
11,01,00,10,10,11,00,...
11,10,01,01,10,11,00,...
One thing that can be deduced from this list is that most likely the
minimum weight of any non-zero codeword is 5, and thus, because
convolutional codes are linear, the minimum distance, called
minimum free distance for convolutional codes for historical
reasons, is dfree = 5.
Peter Mathys
ECEN 5682 Theory and Practice of Error Control Codes
Convolutional Codes
Basic Definitions, Convolutional Encoders
Encoder State Diagrams
Viterbi Decoding Algorithm
Example: Encoder #2 has m = 1,
1 0 1
1 1 1
G0 =
,
G1 =
,
0 1 1
1 0 0
and therefore generator

101
011

000

000


G = 000
000

000

000

..
.
matrix
111
100
101
011
000
000
000
000
..
.
000
000
111
100
101
011
000
000
..
.
Peter Mathys
000
000
000
000
111
100
101
011
..
.
000
000
000
000
000
000
111
100
..
.

...
. . .

. . .

. . .

. . .
.
. . .

. . .

. . .

ECEN 5682 Theory and Practice of Error Control Codes
Convolutional Codes
Basic Definitions, Convolutional Encoders
Encoder State Diagrams
Viterbi Decoding Algorithm
The first few non-zero codewords that this encoder produces are
u = (u0 u1 , . . .)
c = (c0 c1 c2 , . . .)
10,00,00,...
01,00,00,...
11,00,00,...
10,10,00,...
01,10,00,...
11,10,00,...
10,01,00,...
01,01,00,...
11,01,00,...
10,11,00,...
01,11,00,...
11,11,00,...
101,111,000,000,...
011,100,000,000,...
110,011,000,000,...
101,010,111,000,...
011,001,111,000,...
110,110,111,000,...
101,100,100,000,...
011,111,100,000,...
110,000,100,000,...
101,001,011,000,...
011,010,011,000,...
110,101,011,000,...
Peter Mathys
ECEN 5682 Theory and Practice of Error Control Codes
Convolutional Codes
Basic Definitions, Convolutional Encoders
Encoder State Diagrams
Viterbi Decoding Algorithm
Definition: The code generated by a q-ary convolutional encoder
with transfer function matrix G(D) is the set of all vectors of
semi-infinite sequences of encoded symbols c(D) = u(D) G(D),
where u(D) is any vector of q-ary data sequences.
Definition: Two convolutional encoders with transfer function
matrices G1 (D) and G2 (D) are said to be equivalent if they
generate the same codes.
Definition: A systematic convolutional encoder is a convolutional
encoder whose codewords have the property that each data frame
appears unaltered in the first k positions of the first code frame
that it affects.
Note: When dealing with convolutional codes and encoders it is
important to carefully distinguish between the properties of the
code (e.g., the minimum distance of a code) and the properties of
the encoder (e.g., whether an encoder is systematic or not).
Peter Mathys
ECEN 5682 Theory and Practice of Error Control Codes
Convolutional Codes
Basic Definitions, Convolutional Encoders
Encoder State Diagrams
Viterbi Decoding Algorithm
Example: Neither encoder #1 nor encoder #2 are systematic. But
the following binary rate 1/3 encoder, which will be called encoder
#3, with constraint length K = 4 and transfer function matrix
G(D) = 1 1 + D + D 3 1 + D + D 2 + D 3 ,
is a systematic convolutional encoder. Its

111 011 001 011 000
000 111 011 001 011


G = 000 000 111 011 001
000 000 000 111 011

..
..
..
..
..
.
.
.
.
.
generator matrix is

000 000 . . .
000 000 . . .

011 000 . . .
.
001 011 . . .

..
..
.
.
Note that the first column of each triplet of columns has only a
single 1 in it, so that the first symbol in each code frame is the
corresponding data symbol from the data sequence u.
Peter Mathys
ECEN 5682 Theory and Practice of Error Control Codes
Convolutional Codes
Basic Definitions, Convolutional Encoders
Encoder State Diagrams
Viterbi Decoding Algorithm
Much more interesting systematic encoders can be obtained if one
allows not only FIR (finite impulse response), but also IIR (infinite
impulse response) filters in the encoder. In terms of the transfer
function matrix G(D), this means that the use of rational
polynomial expressions instead of generator polynomials as matrix
elements is allowed. The following example illustrates this.
Example: Encoder #4. Binary rate R = 1/3 systematic encoder
with constraint length K = 4 and rational transfer function matrix
1 + D + D3 1 + D + D2 + D3
G(D) = 1
.
1 + D2 + D3
1 + D2 + D3
A block diagram of this encoder is shown in the next figure.
Peter Mathys
ECEN 5682 Theory and Practice of Error Control Codes
Basic Definitions, Convolutional Encoders
Encoder State Diagrams
Viterbi Decoding Algorithm
Convolutional Codes
»
1 + D + D3
1 + D2 + D3
G(D) = 1
1 + D + D2 + D3
1 + D2 + D3
–
.
c(1) (D)
+
+
•
+
u(D)
•
+
c(2) (D)
c(3) (D)
•
+
Fig.7 Binary Rate 1/3 Systematic Convolutional Encoder with K = 4
Peter Mathys
ECEN 5682 Theory and Practice of Error Control Codes
Convolutional Codes
Basic Definitions, Convolutional Encoders
Encoder State Diagrams
Viterbi Decoding Algorithm
Encoder State Diagrams
Convolutional encoders have total memory M. Thus, a
time-invariant q-ary encoder can be regarded as a finite state
machine (FSM) with q M states and it can be completely described
by a state transition diagram called encoder state diagram. Such a
state diagram can be used to encode a data sequence of arbitrary
length. In addition, the encoder state diagram can also be used to
obtain important information about the performance of a
convolutional code and its associated encoder.
Peter Mathys
ECEN 5682 Theory and Practice of Error Control Codes
Convolutional Codes
Basic Definitions, Convolutional Encoders
Encoder State Diagrams
Viterbi Decoding Algorithm
Example: Encoder state diagram for encoder #1. This is a binary encoder with
G(D) = [1 + D 2 1 + D + D 2 ] that uses 2 memory cells and 22 = 4 states. With
reference to the block diagram in Figure 5, label the encoder states as follows:
S0 = 00 ,
S1 = 10 ,
S2 = 01 ,
S3 = 11 ,
where the first binary digit corresponds to the content of the first (leftmost)
delay cell of the encoder, and the second digit corresponds to the content of
the second delay cell.
At any given time t (measured in frames), the encoder is in a particular state
S (t) . The next state, S (t+1) , at time t + 1 depends on the value of the data
frame at time t, which in the case of a rate R = 1/2 encoder is just simply ut .
(1)
(2)
The code frame ct ct
S
(t)
that the encoder outputs at time t depends only on
and ut (and the transfer function matrix G(D), of course). Thus, the
(1)
(2)
possible transitions between the states are labeled with ut /ct ct . The
resulting encoder state diagram for encoder #1 is shown in the following figure.
Peter Mathys
ECEN 5682 Theory and Practice of Error Control Codes
Basic Definitions, Convolutional Encoders
Encoder State Diagrams
Viterbi Decoding Algorithm
Convolutional Codes
S1
1/10
1/11
0/00
S0
0/01
1/00
0/11
S3
1/01
0/10
S2
Fig.8 Encoder State Diagram for Binary Rate 1/2 Encoder with K = 3
To encode the data sequence u = (0, 1, 0, 1, 1, 1, 0, 0, 1, . . .), for instance, start
in S0 at t = 0, return to S0 at t = 1 because u0 = 0, then move on to S1 at
t = 2, S2 at t = 3, S1 at t = 4, S3 at t = 5, S3 at t = 6 (self loop around S3 ),
S2 at t = 7, S0 at t = 8, and finally S1 at t = 9. The resulting code sequence
(after multiplexing) is
c = (00, 11, 01, 00, 10, 01, 10, 11, 11, . . .) .
Peter Mathys
ECEN 5682 Theory and Practice of Error Control Codes
Convolutional Codes
Basic Definitions, Convolutional Encoders
Encoder State Diagrams
Viterbi Decoding Algorithm
Example: Encoder state diagram for encoder #2 with
"
#
1+D D 1+D
G(D) =
D
1
1
and block diagram as shown in Figure 6. This encoder also has
M = 2, but each of the two memory cells receives its input at time
t from a different data stream. The following convention is used to
label the 4 possible states (the upper bit corresponds to the upper
memory cell in Figure 6)
0
S0 = ,
0
1
S1 = ,
0
0
S2 = ,
1
1
S3 = .
1
Because the encoder has rate R = 2/3, the transitions in the
encoder state diagram from time t to time t + 1 are now labeled
(1) (2) (1) (2) (3)
with ut ut /ct ct ct . The result is shown in the next figure.
Peter Mathys
ECEN 5682 Theory and Practice of Error Control Codes
Basic Definitions, Convolutional Encoders
Encoder State Diagrams
Viterbi Decoding Algorithm
Convolutional Codes
10/010
S1
00/111
10/110
11/001
10/101
10/001
11/110
00/000
S0
S3
11/101
00/011
01/100
00/100
01/011
01/000
11/010
S2
01/111
Fig.9 Encoder State Diagram for Binary Rate 2/3 Encoder with K = 2
Peter Mathys
ECEN 5682 Theory and Practice of Error Control Codes
Convolutional Codes
Basic Definitions, Convolutional Encoders
Encoder State Diagrams
Viterbi Decoding Algorithm
Example: The figure on the next slide shows the encoder state
diagram for encoder #4 whose block diagram was given in Figure
7. This encoder has rational transfer function matrix
1 + D + D3 1 + D + D2 + D3
G(D) = 1
,
1 + D2 + D3
1 + D2 + D3
and M = 3. The encoder states are labeled using the following
convention (the leftmost bit corresponds to the leftmost memory
cell in Figure 7)
S0 = 000 , S1 = 100 , S2 = 010 , S3 = 110 ,
S4 = 001 , S5 = 101 , S6 = 011 , S7 = 111 .
Peter Mathys
ECEN 5682 Theory and Practice of Error Control Codes
Basic Definitions, Convolutional Encoders
Encoder State Diagrams
Viterbi Decoding Algorithm
Convolutional Codes
1/100
S1
0/011
1/111
S3
0/011
0/001
0/010
S0
0/000
1/110
S2
S5
0/000
1/110
S7
1/100
1/111
1/101
S4
1/101
0/010
0/001
S6
Fig.10 Encoder State Diagram for R = 1/3, K = 4 Systematic Encoder
Peter Mathys
ECEN 5682 Theory and Practice of Error Control Codes
Convolutional Codes
Basic Definitions, Convolutional Encoders
Encoder State Diagrams
Viterbi Decoding Algorithm
Trellis Diagrams
Because the convolutional encoders considered here are
time-invariant, the encoder state diagram describes their behavior
for all times t. But sometimes, e.g., for decoding convolutional
codes, it is convenient to show all possible states of an encoder
separately for each time t (measured in frames), together with all
possible transitions from states at time t to states at time t + 1.
The resulting diagram is called a trellis diagram.
Example: For encoder #1 with G(D) = 1 + D 2 1 + D + D 2
and M = 2 (and thus 4 states) the trellis diagram is shown in the
figure on the next slide.
Peter Mathys
ECEN 5682 Theory and Practice of Error Control Codes
Basic Definitions, Convolutional Encoders
Encoder State Diagrams
Viterbi Decoding Algorithm
Convolutional Codes
01
•
S3
01
•
10
10
S2
01
•
10
10
•
01
10
•
11
S0
•
t=0
•
11
00
•
t=1
11
•
t=2
•
01
•
11
•
t=3
•
01
···
00
•
11
00
10
00
11
00
•
10
00
S1
01
•
11
•
11
00
•
t=4
00
•
t=5
Fig.11 Trellis Diagram for Binary Rate 1/2 Encoder with K = 3
Peter Mathys
ECEN 5682 Theory and Practice of Error Control Codes
Convolutional Codes
Basic Definitions, Convolutional Encoders
Encoder State Diagrams
Viterbi Decoding Algorithm
Note that the trellis always starts with the all-zero state S0 at time
t = 0 as the root node. This corresponds to the convention that
convolutional encoders must be initialized to the all-zero state
before they are first used. The labels on the branches are the code
frames that the encoder outputs when that particular transition
from a state at time t to a state at time t + 1 is made in response
to a data symbol ut . The highlighted path in Figure 11, for
example, coresponds to the data sequence u = (1, 1, 0, 1, 0, . . .)
and the resulting code sequence
c = (11, 10, 10, 00, 01, . . .) .
Peter Mathys
ECEN 5682 Theory and Practice of Error Control Codes
Convolutional Codes
Basic Definitions, Convolutional Encoders
Encoder State Diagrams
Viterbi Decoding Algorithm
Viterbi Decoding Algorithm
In its simplest and most common form, the Viterbi algorithm is a
maximum likelihood (ML) decoding algorithm for convolutional
codes. Recall that a ML decoder outputs the estimate ĉ = ci iff i
is the index (or one of them selected at random if there are
several) which maximizes the expression p
(v|ci ) , over all
Y |X
codewords c0 , c1 , c2 , . . . . The conditional pmf p
defines the
Y |X
channel model with input X and output Y which is used, and v is
the received (and possibly corrupted) codeword at the output of
the channel. For the important special case of memoryless
channels used without feedback, the computation of p
can be
Y |X
considerably simplified and brought into a form where metrics
along the branches of a trellis can be added up and then a ML
decision can be obtained by comparing these sums. In a nutshell,
this is what the Viterbi algorithm does.
Peter Mathys
ECEN 5682 Theory and Practice of Error Control Codes
Convolutional Codes
Basic Definitions, Convolutional Encoders
Encoder State Diagrams
Viterbi Decoding Algorithm
Definition: A channel with input X and output Y is said to be
memoryless if
p(yj |xj , xj−1 , . . . , x0 , yj−1 , . . . , y0 ) = p
Y |X
(yj |xj ) .
Definition: A channel with input X and output Y is used without
feedback if
p(xj |xj−1 , . . . , x0 , yj−1 , . . . , y0 ) = p(xj |xj−1 , . . . , x0 ) .
Theorem: For a memoryless channel used without feedback
p
(y|x) =
Y |X
N−1
Y
p
Y |X
(yj |xj ) ,
j=0
where N is the length of the channel input and output vectors X
and Y.
Proof: Left as an exercise.
Peter Mathys
ECEN 5682 Theory and Practice of Error Control Codes
Convolutional Codes
Basic Definitions, Convolutional Encoders
Encoder State Diagrams
Viterbi Decoding Algorithm
Definition: The ML decoding rule at the output Y of a discrete
memoryless channel (DMC) with input X , used without feedback
is: Output code sequence estimate ĉ = ci iff i maximizes the
likelihood function
p
(v|ci ) =
Y |X
N−1
Y
p
Y |X
(vj |cij ) ,
j=0
over all code sequences ci = (ci0 , ci1 , ci2 , . . .) for i = 0, 1, 2, . . . .
The pmf p
is given by specifying the transition probabilities of
Y |X
the DMC and vj are the received symbols at the output of the
channel. For block codes N is the blocklength of the code. For
convolutional codes we set N = n (L + m), where L is the number
of data frames that are encoded and m is the maximal memory
order of the encoder.
Peter Mathys
ECEN 5682 Theory and Practice of Error Control Codes
Convolutional Codes
Basic Definitions, Convolutional Encoders
Encoder State Diagrams
Viterbi Decoding Algorithm
Definition: The log likelihood function of a received sequence v at
the channel output with respect to code sequence ci is the
expression
X
N−1
(vj |cij ) ,
log p
log p
(v|ci ) =
Y |X
Y |X
j=0
where the logarithm can be taken to any basis.
Peter Mathys
ECEN 5682 Theory and Practice of Error Control Codes
Basic Definitions, Convolutional Encoders
Encoder State Diagrams
Viterbi Decoding Algorithm
Convolutional Codes
Definition: The path metric µ(v|ci ) for a received sequence v given
a code sequence ci is computed as
µ(v|ci ) =
N−1
X
µ(vj |cij ) ,
j=0
where the symbol metrics µ(vj |cij ) are defined as
µ(vj |cij ) = α log[p
Y |X
(vj |cij )] + f (vj ) .
Here α is any positive number and f (vj ) is a completely arbitrary
real-valued function defined over the channel output alphabet B.
Usually, one selects for every y ∈ B
f (y ) = − log min p
(y |x) ,
x∈A
Y |X
where A is the channel input alphabet. In this way the smallest
symbol metric will always be 0. The quantity α is then adjusted so
that all nonzero metrics are (approximated by) small positive
integers.
Peter Mathys
ECEN 5682 Theory and Practice of Error Control Codes
Basic Definitions, Convolutional Encoders
Encoder State Diagrams
Viterbi Decoding Algorithm
Convolutional Codes
Example: A memoryless BSC with transition probability < 0.5 is
characterized by
p
Y |X
(v |c)
c =0
c =1
minc p
Y |X
v =0
v =1
1−
1−
(v |c)
Thus, setting f (v ) = − log[min p
c
Y |X
(v |c)], yields
f (0) = f (1) = − log .
Peter Mathys
ECEN 5682 Theory and Practice of Error Control Codes
Basic Definitions, Convolutional Encoders
Encoder State Diagrams
Viterbi Decoding Algorithm
Convolutional Codes
With this, the bit metrics become
µ(v |c)
v =0
v =1
c =0
c =1
α(log(1−)− log )
0
0
α(log(1−)− log )
Now choose α as
α=
1
,
log(1 − ) − log so that the following simple bit metrics for the BSC with < 0.5
are obtained
µ(v |c)
v =0
v =1
c =0
c =1
1
0
0
1
Peter Mathys
ECEN 5682 Theory and Practice of Error Control Codes
Convolutional Codes
Basic Definitions, Convolutional Encoders
Encoder State Diagrams
Viterbi Decoding Algorithm
Definition: The partial path metric µ(t) (v|ci ) at time t,
t = 1, 2, . . ., for a path, a received sequence v, and given a code
sequence ci , is computed as
µ(t) (v|ci ) =
t−1
X
(`)
µ(v(`) |ci ) =
tn−1
X
µ(vj |cij ) ,
j=0
`=0
(`)
where the branch metrics µ(v(`) |ci ) of the `-th branch,
` = 0, 1, 2, . . ., for v and a given ci are defined as
(`+1)n−1
(`)
µ(v(`) |ci ) =
X
µ(vj |cij ) .
j=`n
Peter Mathys
ECEN 5682 Theory and Practice of Error Control Codes
Convolutional Codes
Basic Definitions, Convolutional Encoders
Encoder State Diagrams
Viterbi Decoding Algorithm
The Viterbi algorithm makes use of the trellis diagram to
compute the partial path metrics µ(t) (v|ci ) at times t = 1, 2, . . . , N
for a received v, given all code sequences ci that are candidates for
a ML decision, in the following well defined and organized manner.
(1) Every node in the trellis is assigned a number that is equal to
the partial path metric of the path that leads to this node.
By definition, the trellis starts in state 0 at t = 0 and
µ(0) (v|ci ) = 0.
(2) For every transition from time t to time t + 1, all q (M+k)
(there are q M states and q k different input frames at every
(t)
time t) t-th branch metrics µ(v(t) |ci ) for v given all t-th
codeframes are computed.
Peter Mathys
ECEN 5682 Theory and Practice of Error Control Codes
Convolutional Codes
Basic Definitions, Convolutional Encoders
Encoder State Diagrams
Viterbi Decoding Algorithm
(3) The partial path metric µ(t+1) (v|ci ) is updated by adding the
t-th branch metrics to the previous partial path metrics
µ(t) (v|ci ) and keeping only the maximum value of the partial
path metric for each node in the trellis at time t + 1. The
partial path that yields the maximum value at each node is
called the survivor, and all other partial paths leading into the
same node are eliminated from further consideration as a ML
decision candidate. Ties are broken by flipping a coin.
(4) If t + 1 = N (= n(L + m) where L is the number of data
frames that are encoded and m is the maximal memory order
of the encoder), then there is only one survivor with
maximum path metric µ(v|ci ) = µ(N) (v|ci ) and thus ĉ = ci
is announced and the decoding algorithm stops. Otherwise,
set t ← t + 1 and return to step 2.
Peter Mathys
ECEN 5682 Theory and Practice of Error Control Codes
Convolutional Codes
Basic Definitions, Convolutional Encoders
Encoder State Diagrams
Viterbi Decoding Algorithm
Theorem: The path with maximum path metric µ(v|ci ) selected by
the Viterbi decoder is the maximum likelihood path.
Proof: Suppose ci is the ML path, but the decoder outputs
ĉ = cj . This implies that at some time t the partial path metrics
satisfy µ(t) (v|cj ) ≥ µ(t) (v|ci ) and ci is not a survivor. Appending
the remaining path that corresponds to ci to the survivor cj at
time t thus results in a larger path metric than the one for the ML
path cj . But this is a contradiction for the assumption that cj is
the ML path.
QED
Example: Encoder #1 (binary R = 1/2, K = 3 encoder with
G(D) = [1 + D 2 1 + D + D 2 ]) was used to generate and transmit
a codeword over a BSC with transition probability < 0.5. The
following seqence was received:
v = (10, 10, 00, 10, 10, 11, 01, 00, . . .) .
To find the most likely codeword ĉ that corresponds to this v, use
the Viterbi algorithm with the trellis diagram shown in Figure 12.
Peter Mathys
ECEN 5682 Theory and Practice of Error Control Codes
Basic Definitions, Convolutional Encoders
Encoder State Diagrams
Viterbi Decoding Algorithm
Convolutional Codes
3
•
S3
10
S2
1
01 •
4
01
•
10
X
10
01
10
4
01 •
10
•
1
11
2
S0 •
00
v:
10
0
•
11 •
3
11 X
11
00
1
•
X
•
00
2
10
6
01 •
X
X 00
11 •
00
•
5.
10
7
X •
10
7
01 •
00
X •
10
10
00
11 •
7
00
5.
10
8.
01
•
X
01
10
X
8.
01 •
10
10
•
X
11
01 •
01
10
11.
X •
X
10
11.
01 •
X
11 X
11
4
00
01
X
X
S1
5
X •
X 00
X 00
11 •
9
11 •
10
11
X •
11
00
7
10
X •
00
9
11
X
•
00
11 X
00
10
01
···
11 •
13
X
•
12
00
Fig.12 Viterbi Decoder: R = 1/2, K = 3 Encoder, Transmission over BSC
At time zero start in state S0 with a partial path metric µ(0) (v|ci ) = 0. Using
the bit metrics for the BSC with < 0.5 given earlier, the branch metrics for
each of the first two brances are 1. Thus, the partial path metrics at time
t = 1 are µ(1) (10|00) = 1 and µ(1) (10|11) = 1.
Peter Mathys
ECEN 5682 Theory and Practice of Error Control Codes
Convolutional Codes
Basic Definitions, Convolutional Encoders
Encoder State Diagrams
Viterbi Decoding Algorithm
(1)
Continuing to add the branch metrics µ(v(1) |ci ), the partial path metrics
µ(2) ((10, 10)|(00, 00)) = 2, µ(2) ((10, 10)|(00, 11)) = 2,
µ(2) ((10, 10)|(11, 01)) = 1, and µ(2) ((10, 10)|(11, 10)) = 3 are obtained at time
t = 2. At time t = 3 things become more interesting. Now two branches enter
into each state and only the one that results in the larger partial path metric is
kept and the other one is eliminated (indicated with an “X”). Thus, for
instance, since 2 + 2 = 4 > 1 + 0 = 1, µ(3) ((10, 10, 00)|(00, 00, 00)) = 4
whereas the alternative path entering S0 at t = 3 would only result in
µ(3) ((10, 10, 00)|(11, 01, 11)) = 1. Similarly, for the two paths entering S1 at
t = 3 one finds either µ(3) ((10, 10, 00)|(00, 00, 11)) = 2 or
µ(3) ((10, 10, 00)|(11, 01, 00)) = 3 and therefore the latter path and
corresponding partial path metric survive. If there is a tie, e.g., as in the case
of the two paths entering S0 at time t = 4, then one of the two paths is
selected as survivor at random. In Figure 12 ties are marked with a dot
following the value of the partial path metric. Using the partial path metrics at
time t = 8, the ML decision at this time is to choose the codeword
corresponding to the path with metric 13 (highlighted in Figure 12), i.e.,
ĉ = (11, 10, 01, 10, 11, 11, 01, 00, . . .)
Peter Mathys
=⇒
û = (1, 1, 1, 0, 0, 1, 0, 1, . . .).
ECEN 5682 Theory and Practice of Error Control Codes
Convolutional Codes
Basic Definitions, Convolutional Encoders
Encoder State Diagrams
Viterbi Decoding Algorithm
Definition: A channel whose output alphabet is the same as the
input alphabet is said to make hard decisions, whereas a channel
that uses a larger alphabet at the output than at the input is said
to make soft decisions.
Note: In general, a channel which gives more differentiated output
information is preferred (and has more capacity) than one which
has the same number of output symbols as there are input
symbols, as for example the BSC.
Definition: A decoder that operates on hard decision channel
outputs is called a hard decision decoder, and a decoder that
operates on soft decision channel outputs is called a soft decision
decoder.
Peter Mathys
ECEN 5682 Theory and Practice of Error Control Codes
Convolutional Codes
Basic Definitions, Convolutional Encoders
Encoder State Diagrams
Viterbi Decoding Algorithm
Example: Use again encoder #1, but this time with a soft
decision channel model with 2 inputs and 5 outputs as shown in
the following figure.
1
•
Input
X
0
•
•
1
•
!
•
∆
•
@
•
0
Output
Y
Fig.13 Discrete Memoryless Channel (DMC) with 2 Inputs and 5 Outputs
Peter Mathys
ECEN 5682 Theory and Practice of Error Control Codes
Basic Definitions, Convolutional Encoders
Encoder State Diagrams
Viterbi Decoding Algorithm
Convolutional Codes
The symbols “@” and “!” at the channel output represent “bad”
0’s and “bad” 1’s, respectively, whereas “∆” is called an erasure
(i.e., it is uncertain whether ∆ is closer to a 0 or a 1, whereas a
bad 0, for example, is closer to 0 than to 1). The transition
probabilities for this channel are
p
Y |X
(v |c)
v =0
v =@
v =∆
v =!
v =1
0.5
0.06
0.2
0.1
0.14
0.14
0.1
0.2
0.06
0.5
c =0
c =1
After taking (base 2) logarithms
log2 [p
Y |X
(v |c)]
c=0
c=1
− log2 [minc p
Y |X
(v |c)]
v =0
v =@
v =∆
v =!
v =1
−1.00
−4.06
−2.32
−3.32
−2.84
−2.84
−3.32
−2.32
−4.06
−1.00
4.06
3.32
2.84
3.32
4.06
Peter Mathys
ECEN 5682 Theory and Practice of Error Control Codes
Convolutional Codes
Basic Definitions, Convolutional Encoders
Encoder State Diagrams
Viterbi Decoding Algorithm
Using
µ(v |c) = α log2 [p
Y |X
(v |c)] − log2 [min p
Y |X
c
(v |c)]
with α = 1 and rounding to the nearest integer yields the bit
metrics
µ(v |c)
v =0
v =@
v =∆
v =!
v =1
c =0
c =1
3
0
1
0
0
0
0
1
0
3
The received sequence
v = (11, !@, @0, ∆0, 1!, 00, ∆0, 10, . . .) ,
can now be decoded using the Viterbi algorithm as shown in Figure
14 on the next page.
Peter Mathys
ECEN 5682 Theory and Practice of Error Control Codes
Basic Definitions, Convolutional Encoders
Encoder State Diagrams
Viterbi Decoding Algorithm
Convolutional Codes
8
•
S3
10
S2
6
01 •
9
01
•
10
X
10
11
01 •
X
•
S1
•
6
11
1
S0 •
00
v:
11
0
•
00
0
•
1
!@
10
13
X •
12
01 •
00
11 •
10
11 •
14
00
11 X
X •
6
@0
01
10
10
X
00
11 X
11
01
10
00
17
X •
16
01 •
X
X 00
11 •
15
X •
00
11
∆0
X •
01
X
10
20
01 •
10
25
X •
23
01 •
00
00
11 •
22
11 •
23
00
X
•
11 X
00
22
00
01
10
10
X
X
16
1!
•
10
11 X
11
20
01
10
X
•
31
01 •
X
···
X 00
11 •
28
11
00
25
∆0
29
X •
X
•
28
10
Fig.14 Viterbi Decoder: R = 1/2, K = 3 Encoder, 2-Input, 5-Output DMC
Clearly, the Viterbi algorithm can be used either for hard or soft decision
decoding by using appropriate bit metrics. In this example the ML decision (up
to t = 8) is
ĉ = (11, 01, 00, 10, 10, 00, 10, 10, . . .) ,
corresponding to û = (1, 0, 1, 1, 0, 1, 1, 0, . . .).
Peter Mathys
ECEN 5682 Theory and Practice of Error Control Codes