Exercise Problems 9–12: Information Theory
Exercise 9
The information in continuous signals which are strictly bandlimited (lowpass or bandpass)
is quantised, in that such continuous signals can be completely represented by a finite set of
discrete samples. Describe two theorems about how discrete samples suffice for exact reconstruction of continuous bandlimited signals, even at all the points between the sampled values.
Exercise 10
(a) What does it mean for a function to be “self-Fourier?” Name three functions which are
of importance in information theory and that have the self-Fourier property, and in each case
mention a topic or a theorem exploiting it.
(b) Show that the set of all Gabor wavelets is closed under convolution, i.e. that the convolution
of any two Gabor wavelets is just another Gabor wavelet. [HINT: This property relates to the
fact that these wavelets are also closed under multiplication, and that they are also self-Fourier.
You may address this question for just 1D wavelets if you wish.]
(c) Show that the family of sinc functions used in the Nyquist Sampling Theorem,
sinc(x) =
sin(λx)
λx
is closed under convolution. Show further that when two different sinc functions are convolved,
the result is simply whichever one of them had the lower frequency, i.e. the smaller λ.
Exercise 11
(a) An important class of complex-valued functions for encoding information with maximal resolution simultaneously in the frequency domain
and the signal domain are Gabor wavelets. Using an expression for
their functional form, explain:
1. their spiral helical trajectory as phasors, shown here with
projections of their real and imaginary parts;
2. the Uncertainty Principle under which they are optimal;
3. the spaces they occupy in the Information Diagram;
4. some of their uses in pattern encoding and recognition.
(b) Explain why the real-part of a 2D Gabor wavelet has a 2D Fourier transform with two
peaks, not just one, as shown in the right panel of the Figure below.
2D Fourier Transform
0
0
Z
0.5
Z
0.5
1
1
2D Gabor Wavelet: Real Part
0.5
Po
siti
on 0
in
De
gre
ion
es -0.5 0.5 osit
P
-
0 egre
D
in
0.5
es
10
Sp
atia
lF
req 0
ue
n
cy
(CP -10 0 tial
-1 pa
D)
S
0 cy (C
en
qu
e
r
F
10
)
PD
Exercise 12
(a) Compare and contrast the compression strategies deployed in the JPEG and JPEG-2000
protocols. Include these topics: the underlying transforms used; their computational efficiency
and ease of implementation; artefacts introduced in lossy mode; typical compression factors;
and their relative performance when used to achieve severe compression rates.
(b) Define the Kolmogorov algorithmic complexity K of a string of data, and say whether or
not it is computable. What relationship is to be expected between the Kolmogorov complexity
K and the Shannon entropy H for a given set of data? Give a reasonable estimate of K for a
fractal, and explain why it is reasonable. Discuss the following concepts in Kolmogorov’s theory
of pattern complexity: how writing a program that generates a pattern is a way of compressing
it, and executing such a program decompresses it; Kolmogorov incompressibility, and patterns
that are their own shortest possible description.