IERG 6154: Network Information Theory
Homework 1
Due: Sep 19, 2016
1. Inequalities: Show that
(a) H(X|Z) ≤ H(X|Y ) + H(Y |Z)
(b) I(X1 , X2 ; Y1 , Y2 ) ≤ I(X1 ; Y1 ) + I(X2 ; Y2 ) if p(x1 , x2 , y1 , y2 ) = p(x1 , x2 )p(y1 |x1 )p(y2 |x2 )
(c) I(X1 , X2 ; Y1 , Y2 ) ≥ I(X1 ; Y1 ) + I(X2 ; Y2 ) if p(x1 , x2 , y1 , y2 ) = p(x1 )p(x2 )p(y1 , y2 |x1 , x2 )
(d) h(X +aY ) ≥ h(X +Y ) when Y ∼ N(0, 1), a ≥ 1. Assume that X and Y are independent.
2. Prove or give a counterexample to the following inequalities:
(a) H(X1 , X2 , X3 ) + H(X1 , X2 , X4 ) + H(X2 , X3 , X4 ) + H(X1 , X3 , X4 ) ≤
H(X1 , X4 ) + H(X2 , X3 ) + H(X3 , X4 )].
3
2 [H(X1 , X2 )
+
(b) H(X1 , X2 , X3 ) + H(X1 , X2 , X4 ) + H(X2 , X3 , X4 ) + H(X1 , X3 , X4 ) ≤ 3[H(X1 , X2 ) +
H(X3 , X4 )].
3. Show the following equality for any pair of sequences Y n , Z n
n
X
n
I(Y i−1 ; Zi |Zi+1
)=
i=1
m
n
X
n
I(Zi+1
; Yi |Y i−1 )
i=1
Ynm
Note: Y and
denote Y1 , Y2 , ...Ym and Yn , Yn+1 , ...Ym respectively. There is an abuse of
n
notation at the edges. Take Y 0 and Zn+1
as trivial random variables.
Remark: This equality is called Csiszar-sum Lemma and is very useful for many converses
and outer bounds. This is also probably the single biggest obstacle for making progress; i.e.
we are stuck by this trick for the converses and we need to go beyond or bypass completely
this line of thinking. More on it later.
4. Prove that when f : R → [0, 21 ] such that H(f (u)) is convex and f is twice differentiable, then
H(f (u) ∗ p) is convex in u for p ∈ [0, 1]. (Fan Cheng’s extension of Mrs. Gerber’s lemma).
In the above, for 0 ≤ x, y ≤ 1 define x ∗ y = x(1 − y) + y(1 − x). As a corollary note that
H(H −1 (u) ∗ p) is convex in u for p ∈ [0, 1] where H −1 (u) as a mapping from [0, 1] to [0, 21 ].
5. Show that log(2x + 2y ) is convex in (x, y). This is used to prove conditional EPI.
6. Let X and Z be independent, zero-mean, continuous random variables with E(X 2 ) = P, E(Z 2 ) =
Q.
1
(a) Let E(X|X + Z) be the conditional expectation of X given X + Z. (If you are not
familiar with conditional expectation, please take E(X|X + Z)
to be a random variable
Ŷ that is the function of X + Z that minimizes E (X − Ŷ )2 among all such functions.
Show that
2
E (X − E(X|X + Z)) ≤
PQ
.
P +Q
(Hint: Use the best linear estimator of X given X + Z and upper bound the variance
using this estimator. In other words let Y1 = a(X + Z) and optimize over the choice of
a)
Q
(b) Show that h(X|X + Z) ≤ 12 log 2πe PP+Q
. Further show that equality holds iff X and
Z are Gaussians. (For this part, EPI may be useful).
(c) Let X̃, Z̃ ∼ N(0, P ), N(0, Q) repectively and assume that they are independent of each
other and of X, Z. Show that I(X̃; X̃ + Z) ≥ I(X̃; X̃ + Z̃) (Hint: Use EPI)
7. Show that
√
√
J( λx + 1 − λy) ≤ λJ(x) + (1 − λ)J(y)
(1)
is equivalent to
1
1
1
≥
+
J(x + y)
J(x) J(y)
(Stam’s inequality)
(2)
8. Costa’s concavity of entropy power: Let X be an arbitrary continuous random variable
with density g(x) such that | log g(x)| ≤ C(1 + x2 ). Let Z be an independent Gaussian with
zero mean and
√ unit variance. Let the bivariate function f (x, t) be the pdf of the random
variable X + tZ and h be the differential entropy of f . (It is easy to show that for every
positive t, the density function f (x, t) and all of its derivatives viewed as a function of x decays
to 0 as |x| → ∞; however, for this homework, you may assume this is so without needing to
establish it).
d2
Subscripts denote partial derivatives ( dx
2 f = fxx ). We use natural logarithms and the bounds
for the Integrals ±∞.
(a) Show that f satisfies the partial differential (heat) equation:
ft =
1
fxx
2
(b) Show that
ht =
Hint:
R∞
−∞
U V 0 dx = −
R∞
−∞
1
2
Z
f
fx
f
2
dx
(3)
U 0 V dx when U V → 0 as |x| → ∞.
(c) Show that (Hint: use the previous hint)
Z 2
Z 4
fx
fxx
2
fx
f
dx =
f
dx
f
f
3
f
2
(4)
(d) Show that
1
htt = −
2
Thus h(X +
below.
√
Z
f
fxx
−
f
fx
f
2 !2
dx.
tZ) is concave in t. However, something stronger is true as we shall see
√
R
f2
(e) Show that e2h(X+ tZ) is concave in t. (Hint: Use f ( ffxx − fx2 + β)2 dx ≥ 0 and choose
an appropriate β to infer the result.) This result was originally established by Costa and
the proof outlined in this homework is due to C. Villani.
3