10.4 AEP for Continuous Random
Variables
Theorem 10.35 (AEP I for Continuous Random Variables)
1
log f (X) ⇥ h(X)
n
in probability as n ⇥ ⇤, i.e., for any
⇥
1
Pr
log f (X)
n
Proof WWLN.
> 0, for n sufficiently large,
⇤
h(X) <
>1
.
Theorem 10.35 (AEP I for Continuous Random Variables)
1
log ______
f (X) ⇥ h(X)
n
in probability as n ⇥ ⇤, i.e., for any
⇥
1
Pr
log ______
f (X)
n
Proof WWLN.
> 0, for n sufficiently large,
⇤
h(X) <
>1
.
Theorem 10.35 (AEP I for Continuous Random Variables)
1
log f (X) ⇥ ______
h(X)
n
in probability as n ⇥ ⇤, i.e., for any
⇥
1
Pr
log f (X)
n
Proof WWLN.
> 0, for n sufficiently large,
⇤
h(X) <
>1
.
______
Theorem 10.35 (AEP I for Continuous Random Variables)
1
log f (X) ⇥ h(X)
n
in probability as n ⇥ ⇤, i.e., for any
⇥
1
Pr
log f (X)
n
Proof WWLN.
> 0, for n sufficiently large,
⇤
h(X) <
>1
.
n
Definition 10.36 The typical set W[X]
with respect to f (x) is the set of
sequences x = (x1 , x2 , · · · , xn ) ⇥ X n such that
1
log f (x)
n
h(X) <
or equivalently,
1
log f (x) < h(X) +
h(X)
<
n
n
where is an arbitrarily small positive real number. The sequences in W[X]
are
called -typical sequences.
Empirical Differential Entropy:
1
log f (x) =
n
n
1⇥
log f (xk )
n
k=1
The empirical differential entropy of a typical sequence is close to the true
differential entropy h(X).
n
Definition 10.36 The typical set W[X]
with respect to f (x) is the set of
sequences x = (x1 , x2 , · · · , xn ) ⇥ X n such that
1
log _____
f (x)
n
h(X) <
or equivalently,
1
log _____
f (x) < h(X) +
h(X)
<
n
n
where is an arbitrarily small positive real number. The sequences in W[X]
are
called -typical sequences.
Empirical Differential Entropy:
1
log f (x) =
n
n
1⇥
log f (xk )
n
k=1
The empirical differential entropy of a typical sequence is close to the true
differential entropy h(X).
n
Definition 10.36 The typical set W[X]
with respect to f (x) is the set of
sequences x = (x1 , x2 , · · · , xn ) ⇥ X n such that
1
log f (x)
n
h(X)
______
<
or equivalently,
1
log f (x) < ______
h(X) +
h(X)
<
______
n
n
where is an arbitrarily small positive real number. The sequences in W[X]
are
called -typical sequences.
Empirical Differential Entropy:
1
log f (x) =
n
n
1⇥
log f (xk )
n
k=1
The empirical differential entropy of a typical sequence is close to the true
differential entropy h(X).
n
Definition 10.36 The typical set W[X]
with respect to f (x) is the set of
sequences x = (x1 , x2 , · · · , xn ) ⇥ X n such that
1
log f (x)
n
h(X) <
or equivalently,
1
log f (x) < h(X) +
h(X)
<
n
n
where is an arbitrarily small positive real number. The sequences in W[X]
are
called -typical sequences.
Empirical Differential Entropy:
1
log f (x) =
n
n
1⇥
log f (xk )
n
k=1
The empirical differential entropy of a typical sequence is close to the true
differential entropy h(X).
n
Definition 10.36 The typical set W[X]
with respect to f (x) is the set of
sequences x = (x1 , x2 , · · · , xn ) ⇥ X n such that
1
log f (x)
n
h(X) <
or equivalently,
1
log f (x) < h(X) +
h(X)
<
n
n
where is an arbitrarily small positive real number. The sequences in W[X]
are
called -typical sequences.
Empirical Differential Entropy:
1
log __
f (x) =
n
n
1⇥
log __
f (xk )
n
k=1
The empirical differential entropy of a typical sequence is close to the true
differential entropy h(X).
Definition 10.37 The volume of a set A in <n is defined as
Z
Vol(A) =
dx
A
Theorem 10.38 (AEP II for Continuous Random Variables) The following
hold for any ✏ > 0:
n
1) If x 2 W[X]✏
, then
2
n(h(X)+✏)
< f (x) < 2
n(h(X) ✏)
2) For n sufficiently large,
n
Pr{X 2 W[X]✏
}>1
✏
3) For n sufficiently large,
(1
✏)2n(h(X)
✏)
n
< Vol(W[X]✏
) < 2n(h(X)+✏)
Definition 10.37 The volume of a set A in <n is defined as
Z
Vol(A) =
dx
A
Theorem 10.38 (AEP II for Continuous Random Variables) The following
hold for any ✏ > 0:
n
1) If x 2 W[X]✏
, then
2
n(h(X)+✏)
< f (x) < 2
n(h(X) ✏)
2) For n sufficiently large,
n
Pr{X 2 W[X]✏
}>1
✏
3) For n sufficiently large,
(1
✏)2n(h(X)
✏)
n
< Vol(W[X]✏
) < 2n(h(X)+✏)
Definition 10.37 The volume of a set A in <n is defined as
Z
Vol(A) =
dx
A
Theorem 10.38 (AEP II for Continuous Random Variables) The following
hold for any ✏ > 0:
n
1) If x 2 W[X]✏
, then
2
n(h(X)+✏)
< f (x) < 2
n(h(X) ✏)
2) For n sufficiently large,
n
Pr{X 2 W[X]✏
}>1
✏
3) For n sufficiently large,
(1
✏)2n(h(X)
✏)
n
< Vol(W[X]✏
) < 2n(h(X)+✏)
Definition 10.37 The volume of a set A in <n is defined as
Z
Vol(A) =
dx
A
Theorem 10.38 (AEP II for Continuous Random Variables) The following
hold for any ✏ > 0:
n
1) If x 2 W[X]✏
, then
2
n(h(X)+✏)
< f (x) < 2
n(h(X) ✏)
2) For n sufficiently large,
n
Pr{X 2 W[X]✏
}>1
✏
3) For n sufficiently large,
(1
✏)2n(h(X)
✏)
n
< Vol(W[X]✏
) < 2n(h(X)+✏)
Definition 10.37 The volume of a set A in <n is defined as
Z
Vol(A) =
dx
A
Theorem 10.38 (AEP II for Continuous Random Variables) The following
hold for any ✏ > 0:
n
1) If x 2 W[X]✏
, then
2
n(h(X)+✏)
< f (x) < 2
n(h(X) ✏)
2) For n sufficiently large,
n
Pr{X 2 W[X]✏
}>1
✏
3) For n sufficiently large,
(1
✏)2n(h(X)
✏)
n
< Vol(W[X]✏
) < 2n(h(X)+✏)
Remarks
1. The volume of the typical set is approximately equal to 2nh(X) when n is
large.
2. The fact that h(X) can be negative does not incur any di⇥culty because
2nh(X) is always positive.
3. If the differential entropy is large, then the volume of the typical set is
large.
Remarks
1. The volume of the typical set is approximately equal to 2nh(X) when n is
large.
2. The fact that h(X) can be negative does not incur any di⇥culty because
2nh(X) is always positive.
3. If the differential entropy is large, then the volume of the typical set is
large.
Remarks
1. The volume of the typical set is approximately equal to 2nh(X) when n is
large.
2. The fact that h(X) can be negative does not incur any di⇥culty because
2nh(X) is always positive.
3. If the differential entropy is large, then the volume of the typical set is
large.
Remarks
1. The volume of the typical set is approximately equal to 2nh(X) when n is
large.
2. The fact that h(X) can be negative does not incur any di⇥culty because
2nh(X) is always positive.
3. If the differential entropy is large, then the volume of the typical set is
large.