Linear-time
Encodable/Decodable ErrorCorrecting Codes
Some results from two papers of M. Sipser
and D. Spielman1
1Much
of the technical wording within is either taken verbatim or paraphrased from [SS96;
S96]; see references.
Coding for Low-Power
Communication Across Noisy
Channels
• Static on the line / Game of “Operator”
(Spielman’s lecture).
• A coding solution: “A” as in “Apple”, “B”
as in “Baseball”, …”Z” as in Zebra.
Messages (live in low-d space)
minimum distance bound
(Encode)
Codewords (live in high-d space) =
Message Bits + Check Bits
+
Diagram for codeword space inspired by digital signature lecture at MIT earlier this year
(Decode)
Motivation
• Recall one question answered in professor
Spielman’s lecture:
• Q1: Do there exist families of codes which as
block length (n  ) grows realize:
– constant information rate  r
/
– bounded relative minimum distance 
n
• A: Yes! (e.g. Justesen’s explicit construction)
• Q2: Can we construct such a family of codes with
linear time algorithms for encoding AND
decoding a constant fraction of error?
• A: Yes again!
Goals
1. Describe randomized and explicit
constructions of families of asymptotically
good error-correcting codes
2. Describe linear-time encoding and
decoding algorithms for these codes
– Sketch proofs of correctness for the decoding
algorithms
– Motivate why these algorithms are in fact
linear-time
Sights and sounds along the way…
1. Linear-time decodable error-correcting
codes (built from expander graphs and existing code)
• Randomized construction
• Explicit construction
2. Linear-time encodable/error-reducible
Error-reduction codes (also from expander graphs
and existing code)
• Explicit construction
3. Linear-time encodable/decodable errorcorrecting codes (from error-reduction codes)
…and mathematical flavors
•
Graph theory
–
–
Clever correspondences between graphs and codes,
graphs and graphs.
Expander and regularity properties of underlying
graphs used to establish
•
•
–
•
minimum distance bounds
correctness of decoding algorithms
Regularity also used to show rate, linear-time
Probability
–
–
The “probabilistic method,” or, a way to prove
existence theorems non-constructively
How to identify properties of randomly constructed
objects
Models of Computation for
Linear-Time1
• Cost of Memory access / Bit-wise operations
(as a function of argument size n):
– Uniform cost model => 1 time unit
– Logarithmic cost model => log n time units
• Machines for sequential decoding algorithms
– Graph inputted as a set of pointers; each vertex indexes
a list of pointers to its neighbors2
• Pointer Machine (constant degree directed graph)
• RAM
• Boolean circuits for parallel decoding algorithms
– Size (= # wires)
– Depth (= longest path length in circuit digraph)
Important things to recall
• Linear Codes: codewords form a vector space
• Gilbert-Varshamov bound: “Good” linear codes
exist for sufficiently long block lengths
• Graphs G = (V, E)
• Expander graphs (Mohamed’s lecture)
– All minority subsets of vertices have many (external)
neighbors (large “boundary”):
| (U ) | c | U |
Expansion factor =
c
• Polynomial-time constructible infinite families of
k-regular Ramanujan graphs (Mohamed’s lecture)1
– Realize maximum possible “spectral gap” (~expansion)
– Explicit construction of Lubotsky-PhilipsSarnak/Margulis using Cayley graphs2
1 [LL00 p1]
• “Unbalanced”, (c,d)-regular bipartite graphs
• Regularity important for showing linear-time; also for counting
arguments used to prove correctness of decoding algorithms.
Example:
Six 2-regular
vertices
A (2,3)-regular graph:
Four 3-regular
vertices
• Degree ratio c/d determines bipartite ratio (edge count)
Defining expansion for (c,d)regular graphs
• Only consider subsets of the c-regular
vertices (LHS)
Definition1: A (c,d)-regular graph is a
(c, d ,  ,  ) -expander if every subset of at most
an  -fraction of the c-regular vertices
expands by a factor of at least  .
1
[SS96, p1711]
Building an Expander Code:
C(B,S)
•
Necessary Ingredients
– B a (c,d)-regular expander graph (c<d)
– S a good block-length d error-correcting code
(existence by the Gilbert-Varshamov bound)
–
A correspondence between the graph
and
B
the codewords of
.
C ( B, S )
v1
n variables
(c-regular)
c
d
C1
cn
cn
vn
constraints
(d-regular)
d
d
Corresponding
n (c-regular) vertices
variables
B  C ( B, S )
cn
d
(d-regular) vertices
constraints


{v1 , v2 ,, vn }
{C1 , C2 ,, Ccn d }
The expander code1 C ( B, S ) :
• A constraint is satisfied if its d neighboring
variables form a codeword of the input code S.
• The expander code C(B,S) is the code of block
length n whose codewords are the settings on the
variables such that all constraints are satisfied
simultaneously.
• The (linear) code C(B,S) is the vector subspace
which solves the set of all linear constraints
• The constraints collectively determine a check
matrix H (from which generator matrix G can be derived)
1
[SS96, p1712]
C(B,S) is asymptotically
1
good
• For:
– Graph B with good enough expansion
– Code S with bounded rate and minimum distance
• Conclusion:
– C(B,S) is a code of (related) bounded rate and
minimum distance, for all block lengths n.
1[SS96, p1712]
• Proof1: Rate Bound (Relies on regularity & rate of S)
– Count the total number of linear restrictions
imposed on the subspace of codewords
– Assume all restrictions are mutually
independent
– Dimension of solution space = # degrees of
freedom left out of block length n
– Corresponds to the # message bits in code
=> value for rate
1
[SS96, p1712]
• Proof (continued): Min Distance Bound
– Sufficient to show that no codeword w exists below a
certain weight.
– Contradiction by regularity and expansion:
– Regularity fixes the number of edges leaving the subset
of all 1 (non-zero) variables in w
– Expansion guarantees these edges enter a lot of
constraints
– => Average #edges/constraint low
– => there must be some constraint with few 1 variables
as neighbors
–
By minimum distance bound of underlying
code S, entire word can’t be a codeword of C(B,S).

Encoding and Decoding C(B,S)
• Encoding
– quadratic time algorithm (multiply message by
generator matrix G)
• Decoding
– Brute force
– Linear time algorithms
• Challenge: Correctness proof requires constructing
a graph B with expansion greater than c/2; no
explicit constructions are known for such graphs.
Constructing a (c,d)-regular graph with
very good expansion (>c/2)
• Explicit constructions: Best known (1996) can
give expansion c/2, but no greater:
– (Lubotsky-Phillips-Sarnak, Margulis 1988)1:
Ramanujan graphs: (p+1)-regular, second-largest
eigenvalue  2 p
– (Kahale 1992)2: Lubotsky et al.’s Ramanujan graphs
have quality expansion, but spectral gap can not certify
expansion greater than c/2
• Random (c,d)-regular graphs will almost always
have greater expansion
– Food for thought: How could this random construction
be done?
Random (c,d)-regular graphs:
How good is their expansion?
• For B a randomly chosen (c,d)-regular bipartite
graph between n variables and (c/d)n constraints,
for all 0    1, with exponentially high
probability any set of n
variables will have at least

n dc (1  (1   )d )  2cH ( ) / log 2 e
neighbors1.

Pf: Bound
probability that the #
neighbors of any subset is far from its expected #
neighbors
1[SS96,
p1721]
Decoding a Random Expander Code C(B,S)
•
Flip variable(s) in more unsatisfied than satisfied
constraints until none remain
– Sequential Algorithm
– Parallel Algorithm
•
Proof of correctness (sequential algorithm):
– Relies on expansion and regularity properties of graph
B, as well as nature of code S. Sufficient to show:
– As long as there are some corrupt variables (but #
below threshold), then some variable will be flipped.
– =>Number of unsatisfied constraints strictly decreases
•
Linear time relies only on regularity of graph B.
– Total # (unsatisfied) constraints linear in block length
– Local graph structure unchanged as block length
grows; fixed code S => constant local cost
Remarks on Construction
• Expansion factor required: 3c 4 for efficient
decoding of one example code
• Bound on rate assumed all constraints
imposed on variables were independent
• Potential applications for codes with some
redundancy in constraints (see Spielman’s thesis1,
Ch.5 PCP’s and “checkable” codes)
1http://www-math.mit.edu/~spielman/Research/thesis.html
Explicit Construction of
Expander Code C(B,S)
• To construct the code C(B,S), we will take
– B the edge-vertex incidence graph of a dregular Ramanujan graph G
(Lubotsky et al: A dense family of good expander graphs)1
– S a good block length d code
1[S96,
p1726]
Edge-vertex incidence graphs
• Definition: Let G be a graph with edge set E and
vertex set V. The edge-vertex incidence graph
B(G) of G is the bipartite graph with vertex set E V
and edge set:
(e, v)  E V : v is an endpoint
of e
• Thus the edge-vertex incidence graph B(G) of our
d-regular Ramanujan graph G will be a (2,d)regular bipartite graph, to which we correspond
dn/2 variables and n constraints.
Edges E of G
v1
Vertices V of G
2
d
dn
C1
n constraints
(d-regular)
variables
(2-regular)
2
vdn2
Cn
Corresponding
B(G )  C ( B, S )
Edges E of G
dn
2 (2-regular) vertices
variables
Vertices V of G
n (d-regular) vertices
constraints


{v1 , v2 ,, vdn2 }
{C1 , C2 ,, Cn }
Parallel Decoding for C(B,S)1
(where S has block length d, min rel dist  )
1. For each constraint, if the variables in that
constraint differ from a codeword of S in at
most de 4 places, then send a “flip” message to
each variable that differs.
2. In parallel, flip the value of every variable
which receives at least one flip message.
1[SS96,
p1716]
Main
1
Theorem
• There exist polynomial-time constructible families
of asymptotically good expander codes in which a
constant fraction of error can be corrected in a
circuit of size O (log n) and depth O(n log n) .
• The action of this circuit can be simulated in linear
time on a Pointer Machine or a RAM under the
uniform cost model. (No natural sequential
decoding algorithm is known).
1
[SS96, p1716]
Proof Sketch
• Outline similar to the proof above for the random
expander case
• One important difference:
– Above, it was the known expansion factor of graph B
which was used to prove
• Minimum distance bound
• Correctness of decoding algorithm
– Here, instead apply a result of Alon-Chung to upper
bound the number of edges contained in a subgraph of a
graph G with a given spectral gap.
– Convenient, since our edge-vertex incidence graph
B(G) is built from a Ramanujan graph G (Lubotsky)
that is qualified in terms of spectral gap, not expansion
(Alon-Chung)1: Let G be a d-regular graph on n
vertices with second largest eigenvalue  . Let
X be a subset of the vertices of G of size n .
Then, the number of edges in the subgraph
induced by X in G is at most
dn
2
1
[SS96, p1712]

2
 d  (1  


• Could use Alon-Chung bound to quantify
the expansion factor of the edge-vertex
incidence graph B(G)
• But a direct application of Alon-Chung
itself works well in proof
• => each decoding round removes a constant
fraction* of error from input words within
threshold of nearest codeword. *(fraction <1 for
sufficiently large choices of block length d of S)
• => after O(log n) rounds, no error remains.
Circuit for Parallel Decoding1
•
Each parallel decoding round can be
implemented by a circuit of linear size and
constant depth
–
–
•
1
constraint satisfaction: constant # of layers of XOR’s
variable “flips”: majority of constant # of inputs
=> Total of O(log n) rounds needs a circuit
of size O(n log n), depth O(log n).
[SS96, p1715]
Linear Time of Sequential Simulation (Uniform Cost Model)
•
During each round, compile a list of all constraints
which might be unsatisfied at the beginning of next
round by including:
– All constraints of previous round’s list which actually
were unsatisfied
– All constraints neighboring at least one variable
which received a “flip” message in current round
1. Total amount of work performed by algorithm is
linear in the combined length of these lists:
–
–
Each round: Check which constraints from input list are
actually unsatisfied; do decodes on S; send “flips”
Regularity, fixed code S => constant local cost
2. Total length of these lists is linear in block length n of
the code C(B,S).
–
–
–
# variable errors decreases by constant ratio each round
Only constraints containing these variables appear in lists
=> convergent geometric series, first term O(n)
Linear Time of Sequential
Simulation (Logarithmic Cost Model)
Challenge:
• Need O(n) bit additions and O(n) memory accesses
• Cost per access is O(log n)
• Total cost O(n log n)
Solution:
• Retrieve an O(log n) byte per access, only need to
make O(n/log n) accesses; total cost O(n)
• Encode each byte with a good linear error-correcting
code
• Algorithm will correct a constant fraction of error in
overall code unless there is a constant fraction of
error in the encoding of a constant fraction of bytes.
Expander Codes
1
Wrap-Up
• Error distance from a correct codeword was
corresponded to a potential1 (of unsatisfied
constraints) in a graph whose expander and
regularity properties guarantee the success of a
linear-time decoding algorithm which steadily
decreases this potential to zero.
• Quadratic time encoding / linear time decoding
• Superior speed of sequential decoding on random
expanders
• Can tune parallel decoding algorithm for better
performance
1[SS96, p1718-20]
Expander Codes Wrap-Up
• Average-case error-correction results in
experiments much better than bounds
predict (worst-case).
• Introducing random errors in variable flips
can further improve performance.
• For random expanders, degree 5 at variables
seemed to give best results.
Linear-time Encodable/Decodable ErrorCorrecting Codes
• Expanders leveraged to produce linear time
encodable/decodable error-reduction codes
• Construction involving a recursive assembly of
smaller error-reduction & error-correcting codes
gives linear time error-correcting codes.
• Decoding algorithm, proof of its correctness are
the same flavor as those already encountered.
• Randomized / Explicit constructions
– Explicit case shown here is a generalization of the
randomized one
Historical
1
Context
• Prior constructions did not always use such welldefined models of computation or deal with
asymptotic complexity
• Juntesen and Sarwate used efficient
implementation of polynomial GCD algorithm to
get O(n log n log log n) encoding, O(n log2 n log
log n) decoding for RS and Goppa codes.
• Gelfand, Dobrushin, and Pinsker gave randomized
constructions for asymptotically good codes with
O(n) encoding; but no polynomial-time algorithm
for decoding is known.
1
[S96, p1723]
Main
1
Theorem
• There exists a polynomial-time constructible
family of error-correcting codes of rate ¼ that
have linear-time encoding algorithms and lineartime decoding algorithms for a constant fraction of
error.
• The encoding can be performed by linear-size
circuits of logarithmic depth and decoding can be
performed by circuits of size O(n log n) and O(log
n) depth, and can be simulated sequentially in
linear time.
[S96, p1729]
Error-Reduction Codes
• Definition: A code C is an error-reduction code if
there is an algorithm ER which, given an input
word w with total # errors below threshold, can
reduce the # of message bit errors in w to a
constant fraction of the # of check bit errors
originally present in w.
– Note: for zero check bit errors, implies error-correction
of all errors in message bits.
• Can construct such codes from good expanders
Building the Error-Reduction
Code: R(B,S)
•
Necessary Ingredients
–
–
–
B a (c,d)-regular expander graph (c<d)
S a good block length d error-correcting code
with k message bits (rate = k/d)
A correspondence between the graph B and
the codewords of R(B,S).
v1
c
d
n variables
(c-regular)
vn
C1
cn
Ccn d
d
check bit
clusters
(d-regular)
B  R ( B, S )
Corresponding
n (c-regular) vertices
message bits
cn
d
(d-regular) vertices
check bit clusters


{v1 , v2 ,, vn }
{C1 , C2 ,, Ccn d }
The error-reduction code R(B,S) 1
• The error-reduction code R(B,S) will have n
message bits and (c/d)nk check bits.
• The n message bits are identified with the n cregular vertices.
• A cluster Ci of k check bits is identified with each
of the (c/d)n d-regular vertices.
• The check bits of cluster Ci are defined to be the
check bits of that codeword in S which has the d
neighbors of cluster Ci as its message bits
• R(B,S) can be encoded in linear time, (constant
time for generating each check bit cluster; linear #
of clusters)
1
[S96, p1726]
Parallel Error-Reduction round of
R(B,S)1
(where  is the minimum distance of S)
• In parallel, for each cluster, if the check bits
in that cluster and the associated message
bits are within relative distance  6 of a
codeword, then send a “flip” signal to every
message bit that differs from the
corresponding bit in the codeword.
• In parallel, every message bit that receives
at least one “flip” message flips its value.
[S96,p1728]
Quality of an Error-Reduction
Round on R(B,S)
• Take B as the edge-vertex incidence graph of a dregular Ramanujan graph G
(Lubotsky et al: A dense family of good expander graphs)1
• Can show every error reduction round removes a
constant fraction* of error from the message bits
of input words which are within threshold of
nearest codeword. *(fraction <1 for sufficiently large choices of
block length d of S)
1[S96,
p1726]
Proof of Quality of ErrorReduction
• Analogous to that seen above for errorcorrection of explicitly constructed
expander codes
• Relies on the same graph properties:
– Regularity (and fixed code S)
– theorem of Alon-Chung (~expansion)
Linear Time of Error-Reduction
on R(B,S)
• Argument analogous to that for linear-time
simulation of parallel error-correction for the
explicitly constructed expander codes above
• For good enough eigenvalue separation
(~expansion) of graph B & good enough min
distance of code S, after O(log n) rounds, error in
message bits will be reduced to within threshold
required by recursive decoding of overall errorcorrecting code Ck.
Recursive Definition of the ErrorCorrecting Codes Ck
{Err-Red Rk}+{Err-Cor Ck-1} => Err-Cor Ck




Rk  2
  
 
Rk 1


Ck 1
Mk
n0 2 k 2 bits
Ak
n0 2 k 3 bits
Bk
3n0 2 k 3 bits
Ck
n0 2 k 2 bits
Recursive Definition of the ErrorCorrecting Codes Ck
To encode message M k :
•
Use Rk-2 to encode M k

n0 2 k 2 bits
•
Use Ck-1 to encode Ak

Ak
n0 2 k 3 bits
Bk
3n0 2 k 3 bits
•
Use Rk-1 to encode Ak  Bk

Ck
n0 2 k 2 bits
•
Codeword of Ck:
(rate  1 4 )
( M k )  ( Ak  Bk  Ck )
k 2
n0 2 k  2 message bits  3n0 2 check bits
 total block length n  n0 2 k bits
Linear-Time Decoding of codeword:
( M k )  ( Ak  Bk  Ck )
•
 ER linear-time error-reduction for Rk
=>  linear-time error-correction for Ck
–
–
Proof by induction
Hypothesis: Assume  linear-time decoding (errorcorrecting) algorithm EC which works for all sub-codes
Ci, i<k.
1. Run ERx on ( Ak  Bk )  Ck
•
Reduces error in ( Ak  Bk )
2. Run EC on ( Ak )  Bk
•
Corrects ALL errors in Ak ! (by quality of ERx, I.H.)
3. Run ER0 on ( M k )  Ak
•
Corrects ALL errors in M k !!! (by assumption on ER0)
1
Remarks
• By modifying this construction one can
obtain asymptotically good linear-time
encodable and decodable error-correcting
codes of any constant rate r.
• Linear time codes of rate ¼ will correct
about as many errors on average as
expander codes of rate ½
1[S96 p1730-1]
Conclusions
1. Graphs - interesting results obtained by choosing
clever correspondences for edges and vertices:
E.g.
–
–
–
Variables-Message bits / Constraints
Edges / Vertices of another graph
Alon’s generalization: correspond variables with all
length-k paths (here k = 2) in another graph
2. Probabilistic Method1
–
A nice way of proving existence theorems. E.g.
Suppose I can’t show or describe to you a particular
object from a set S with property X, but perhaps I can
prove that for an object randomly selected from S,
Pr[X] > 0, ...
1http://www.math.uiuc.edu/~jozef/math475/description.pdf
Some questions to
1
ponder…
• Can rate vs. error correction tradeoffs be improved
by a more natural (linear-time) decoding algorithm
for these codes? (Present algorithm is an unnatural simulation of
a parallel algorithm)
• Does there exist a (linear-time encodable?) code
for which a number of errors matching the GilbertVarshamov bound can be corrected in linear-time?
How good can linear-time codes be?
1[S96 p1730-1]
References (1)
• Berlekamp, Elwyn R. Algebraic Coding Theory.
Laguna Hills, CA: Aegean Park Press. 1984.
• Blahut, Richard E. Theory and Practice of ErrorControl Codes. Reading, MA: Addison-Wesley
Pub. Co. 1983.
• [LR00] Lafferty, John and Dan Rockmore. Codes
and Iterative Decoding on Algebraic Expander
Graphs. International Symposium on Information
Theory and Its Applications. November 5-8,
2000.
References (2)
•[SS96] Sipser, Michael and Daniel A. Spielman.
Expander Codes. IEEE Transactions on Information
Theory, Vol. 42, No. 6, November 1996 p1710-1722
•[S96] Spielman, Daniel A. Linear-Time Encodable
and Decodable Error-Correcting Codes. Ibid. p17231731
•Sudan, Madhu Lecture Notes from “Algorithmic
Introduction to Coding Theory”. Fall 2001(2002)
MIT.