From Classical to Quantum Information Theory
Exercise set 1
Note: Starred* problems are optional.
(Shannon entropy):
n cards in order 1, 2, . . . , n
Exercise 1.1
A deck of
is provided. First, one randomly chosen card is removed
from the deck; then it is inserted at a randomly chosen position into the deck. What is the
entropy of the resulting deck? If Bob can examine the nal deck, what is his (average) remai-
4
ning uncertainty about the card that Alice had rst removed?
(Properties of the relative entropy): P
D(pkq) := x px log pqxx
same space X .
Exercise 1.2
Recall the denition of relative entropy,
q
on the
(a)
(Klein's inequality)
u
be the
D(pkq) ≥ 0, and that equality holds if and
two dierent ways using Jensen's inequality.)
Show that
(Hint: This can be shown in
(b) Let
for probability distributions
only if
p,
p = q.
uniform distribution on X , i.e. ux = 1/|X | for all x ∈ X . Relate D(pku)
H(p). Find the distributions p on X which maximize H(p).
to
the Shannon entropy
(c)
(Gibbs' inequality)
h : X 7→ R
Let
be any function (Hamiltonian) and
verse temperature). Which probability distributions
functional
(d*)
F (p) := β
(joint convexity)
arguments (p, q).
P
x
px h(x) − H(p)?
p
on
X
β > 0
(in-
minimize the free energy
What is the minimum?
Use Jensen's inequality to show that
D(pkq)
is jointly convex in its
4
Exercise 1.3
Let
X
(The value of a yes-no question mutual information):
X . For a subset S ⊆ X , the random
question whether or not X lies in S , i.e.
1 if X ∈ S
Y :=
0 if X ∈
/ S.
be a random variable on a set
answer to the
Compute the decrease in uncertainty about
quantity
H(X) − H(X|Y ),
X
that comes from knowing the answer
in terms of the probability
(Chain rules):
X1 , X2 , . . . , Xn , Y be random
α := Prob[X ∈ S].
Exercise 1.4
Let
(a)
(chain rule for entropy)
variables.
Show:
H(X1 , X2 , . . . , Xn |Y ) =
n
X
H(Xi |Xi−1 , . . . , X1 , Y ),
i=1
where the term for
i=1
is dened to be
variable
H(X1 |Y ).
Y
Y,
is the
i.e. the
4
(b*)
(chain rule for mutual information)
Show:
I(X1 , X2 , . . . , Xn : Y ) =
n
X
I(Xi : Y |Xi−1 , . . . , X1 ),
i=1
where the term for
i=1
is dened to be
I(X1 : Y ).
4
Exercise 1.5
Let
X, Y, Z
(a)
(Strong subadditivity and Markov chains):
be random variables.
(strong subadditivity)
Show:
I(X : Z|Y ) = H(XY ) + H(ZY ) − H(XY Z) − H(Y ) ≥ 0.
(Hint: Expand
I(X : Z|Y )
When the outcome set of
Y
into a convex combination with weights
p(y).)
is a singleton, this inequality specializes to the subadditivity
property of Shannon entropy, thereby explaining its name.
(b*) Suppose
I(X : Z|Y ) = 0.
Show that
X −Y −Z
(in that order) is a
short Markov chain,
as dened in the lecture.
4