1
Information Theory for Wireless Communications
Lecture 8: Channel Capacity of Waveform Channels
Taught by Prof. Erik G. Larsson
Scribed by Reza Moosavi
Spring 2012
In this lecture, we are interested in finding the capacity of the system depicted in Figure 1. We assume
that all involved signals are real-valued and that H(f ) and G(f ) are linear time-invariant (LTI) filters with
frequency responses given by H(f ) and G(f ) respectively. In the given system, the input signal x(t) is
assumed to be approximately time-limited to the interval [− T2 , T2 ] and approximately band-limited to the
frequency band [−B, B] with average power P . We further assume that the noise signal w(t) is stationary
white Gaussian random process, with mean zero and autocorrelation function rw (τ ) =
x(t)
H(f )
u(t)
+
N0
δ(τ ).
2
y(t)
e(t)
G(f )
w(t)
Fig. 1.
Illustration of the system model considered in this lecture.
Since x(t) is time-limited, we can use the Fourier series expansion to write x(t) as
x(t) =
∞
X
n=0
where
x̄n φ̄n (t) +
∞
X
x̃n φ̃n (t),
(1)
n=1

 √1
, − T2 ≤ t ≤
1
t
T
φ̄0 (t) = √ rect
=
 0,
T
T
otherwise,
T
2
2
and for n ≥ 1,
 q
2

cos
T
φ̄n (t) =
 0,
Note that
2π
nt
T
− T2
,
 q
2

sin
T
and φ̃n (t) =
 0,
2π
nt
T
, − T2 ≤ t ≤
T
2
otherwise.
form an orthonormal basis, since it can be easily shown that
n=1
∞
≤t≤
otherwise,
n
o∞
φ̄0 (t), φ̄n (t), φ̃n (t)
Z
T
2
φ̄n (t)φ̄k (t)dt = δn,k ,
−∞
Z
∞
φ̃n (t)φ̃k (t)dt = δn,k ,
−∞
Z
∞
φ̄n (t)φ̃n (t)dt = 0.
−∞
The signal power can be also written as
"∞
#
#
" Z T
∞
∞
∞
X
X
X
2
1 X
1
1
2
2
2
x̄n +
x̃n =
p̄n +
p̃n
x (t)dt = E
P =E
T − T2
T
T
n=0
n=1
n=0
n=1
(2)
where p̄n , E [x̄2n ] and p̃n , E [x̃2n ].
Now consider the Fourier transform for φ̄0 (t):
Φ̄0 (f ) =
Z
∞
−j2πf t
φ̄0 (t)e
dt =
−∞
=
√
r
1
T
Z
T /2
e−j2πf t dt
−T /2
sin (πf T ) √
= T · sinc(πf T ).
T·
πf T
(3)
Using this result and noting that
φ̄n (t) =
√
2π
nt φ̄0 (t),
2 cos
T
φ̃n (t) =
√
2 sin
2π
nt φ̄0 (t),
T
we can write (by using the properties of Fourier transform) that
r r h n
n i
1
T
Φ̄n (f ) =
Φ̄0 f −
+ Φ̄0 f +
=
sinc π(f T − n) + sinc π(f T + n) ,
2
T
T
2
r 1 h n i 1
T
n
Φ̃n (f ) = √ Φ̄0 f −
− Φ̄0 f +
= ·
sinc π(f T − n) − sinc π(f T + n) .
T
T
j
2
j 2
(4)
(5)
The amplitude frequency responses Φ̄n (f ) and Φ̃n (f ) are plotted in Figure 2. As we can see, φ̄n (t) and
φ̃n (t) have most of their energy in the frequency interval [ n−1
, n+1
]. Therefore, we conclude that only
T
T
φ̄n (t) and φ̃n (t) with n = (0), 1, 2, . . . , BT have any significant energy in [−B, B] and hence x(t) can
be well approximated by:
x(t) ≈
BT
X
n=0
x̄n φ̄n (t) +
BT
X
x̃n φ̃n (t).
(6)
n=1
In other words, x(t) can be represented by roughly 2BT + 1 ≈ 2BT real numbers. This number is often
3
called the “degrees of freedom” (DoF) of signal x(t). Thus we have the following important observation:
A signal with bandwidth B and time duration T has roughly 2BT degrees of freedom.
q
|Φ̄n (f )| = |Φ̃n (f )|
T
2
0
Frequency [Hz]
Fig. 2.
n−1 n n+1
T
T
T
Amplitude Fourier transform for φ̄n (t) and φ̃n (t).
Now assume that x(t) has no DC components, i.e.
Z
T /2
x(t)dt = 0,
−T /2
and thus x̄0 = 0, and assume that T ≫ time constants of the filters H(f ) and G(f ). In this case in
order to find signals u(t) and e(t), we can use the fundamental result for LTI filters which states that the
output of the LTI system corresponding to a sinusoidal input with frequency f0 is a sinusoidal signal with
the same frequency f0 and with amplitude scaled by |H(f0 )| and a phase shift of ∠H(f0 ), where H(f )
represents the frequency response of the LTI system.
u(t) =
=
BT
X
"
n=1
BT h
X
n=1
x̄n |H (n/T )|
r
2
cos
T
i
ūn φ̄n (t) + ũn φ̃n (t) ,
r
#
2π
2
2π
nt + ∠H(n/T ) + x̃n |H (n/T )|
sin
nt + ∠H(n/T )
T
T
T
(7)
4
where ūn , x̄n h̄n + x̃n h̃n and ũn , −x̄n h̃n + x̃n h̄n , with
h̄n , ℜ {H(n/T )} = |H(n/T )| cos ∠H(n/T ) ,
h̃n , ℑ {H(n/T )} = |H(n/T )| sin ∠H(n/T ) .
Similarly, we can use the Fourier expansion for the noise signal w(t). We have
w(t) =
∞
X
w̄n φ̄n (t) +
n=0
∞
X
w̃n φ̃n (t).
(8)
n=1
Since the noise is assumed to be white Gaussian random process, w̄n and w̃n are i.i.d.∼ N (0, N0 /2).
1
Therefore by ignoring {w̄n , w̃n }∞
n=BT +1 that has little energy in the frequency interval [−B, B] , we can
write
BT h
i
X
e(t) =
ēn φ̄n (t) + ẽn φ̃n (t) ,
(9)
n=1
with ēn , w̄n ḡn + w̃n g̃n and ẽn , −w̄n g̃n + w̃n ḡn and
ḡn , ℜ {G(n/T )} = |G(n/T )| cos ∠G(n/T ) ,
g̃n , ℑ {G(n/T )} = |G(n/T )| sin ∠G(n/T ) .
We can also rewrite the above equations as
 
  
h̄
h̃n
ū
x̄
  n ,
 n =  n
−h̃n h̄n
ũn
x̃n
|
{z
} | {z }
,Hn
xn
Now we can write y(t) as
y(t) = u(t) + e(t) =
,Gn
BT
X


(ūn + ēn ) φ̄n (t) + (ũn + ẽn ) φ̃n (t) .
| {z }
| {z }
,ȳn
 
 
z̄n
ȳn
z n =   , Hn−1   .
z̃n
ỹn
(10)
wn


n=1
We next define
 
  
ḡn g̃n
ēn
w̄
  n .
and   = 
−g̃n ḡn
ẽn
w̃n
|
{z
} | {z }
(11)
,ỹn
(12)
Note that this transformation is information lossless since we can always recover ȳn and ỹn back from
1
Note that we can remove those noise components by filtering them out at the receiver.
5
z n . We have that
z n = Hn−1 (Hn xn + Gn w n ) = xn +
1
HnT Gn wn .
2
2
h̄ + h̃n
{z
}
|n
(13)
,qn
We see that q n = [q̄n q̃n ]T is Gaussian distributed with mean zero and covariance matrix
T
E qnqn =
1
N0 ḡn2 + g̃n2
HnT Gn E wn w Tn GTn Hn =
·I
·
2 h̄2n + h̃2n
(h̄2n + h̃2n )2
| {z }
(14)
,λn
Therefore, q̄n and q̃n are i.i.d.∼ N (0, λn N0 /2).
Now if we look at the above derivation in more details, we will realize that by using the given transformation, we get 2BT parallel discrete “tone” additive Gaussian sub-channels (AWG) with independent
noises across different sub-channels of the form

 z̄n = x̄n + q̄n ,
q̄n and q̃n ∼ N (0, λn N0 /2),
 z̃ = x̃ + q̃ ,
n
n
for n = 1, . . . , 2BT.
n
The capacities of the above sub-channels are
1
2
log2 1 +
p̄n
λn N0 /2
and
1
2
log2 1 +
p̃n
λn N0 /2
respectively,
and are achieved if x̄n and x̃n are Gaussian distributed with mean zero and variance p̄n and p̃n , respectively.
Thus we can finally find the capacity of the channel depicted in Figure 1 by solving the below optimization
problem
C = max
BT X
1
n=1
2
log2
p̄n
1+
λn N0 /2
1
p̃n
+ log2 1 +
,
2
λn N0 /2
(15)
subject to
p̄n ≥ 0,
(16)
p̃n ≥ 0,
(17)
BT
X
n=1
p̄n + p̃n ≤ P T.
(18)
Since the above problem is a convex optimization problem (meaning that the optimal solution is unique)
and since it is symmetric with respect to p̄n and p̃n , we conclude that at the optimal solution p̄n = p̃n
6
and thus the capacity can be found by solving
C = max
BT X
n=1
log2 1 +
pn
λn N0 /2
,
(19)
subject to
pn ≥ 0,
BT
X
n=1
(20)
pn ≤ P T /2
(21)
instead. Before we solve this problem in a general form, let us give the solution for the case where all
λn :s are equal.
A. Special Case of λn = 1 for all n = 1, . . . , BT :
P
In this case, because of the symmetry, the optimum solution is equal power allocation, i.e. pn = 2B
for
P
all n = 1, . . . , BT . In this case, the capacity becomes C = BT log2 1 + BN
bits. Since these many
0
P
bits are transmitted during T seconds, the transmission rate in this case is R = CT = B log2 1 + BN
0
bits/s. The spectral efficiency, which is defined as the number of bits transmitted during 1 second in 1 Hz
P
C
of bandwidth is also η = BT = log2 1 + BN0 bits/s/Hz for this case. Figure 3 illustrates the spectral
efficiency as a function of signal-to-noise-ratio (SNR) γ ,
P
N0 B
for a fixed bandwidth of B Hz.
7
Spectral Efficiency [bits/s/Hz]
6
Bandwidth limited
5
4
3
2
Power limited
1
0
0
10
20
30
40
50
SNR
Fig. 3.
Spectral efficiency as a function of SNR for a fixed bandwidth.
60
70
80
90
100
7
Note that for x ≪ 1, we have log(1 + x) ≈ x. Thus if SNR is very low, we can write η ≈ γ log2 e.
In other words, at low SNR region, the spectral efficiency scales linearly with SNR. This region is called
the power limited region. On the other hand, if SNR is high, then η = log2 (1 + γ) ≈ log2 (γ). That is
every doubling of SNR results in 1 bit/s/Hz increase in spectral efficiency. This region is therefore called
bandwidth limited region.
Now assume that the power P is fixed and we increase the bandwidth. We would like to find out the
maximum achievable transmission rate R∞ . We have
R∞ = lim R = lim B log2
B→∞
B→∞
P
1+
N0 B
=
P
log2 e.
N0
(22)
Figure 4 illustrates the transmission rate as a function of the bandwidth for the case with P/N0 = 1.
1.5
R∞
Achievable rate [bits/s]
Power limited
1
0.5
0
0
Bandwidth limited
5
10
15
20
25
30
35
40
Bandwidth [Hz]
Fig. 4.
Achievable rate as a function of bandwidth for a fixed P/N0 .
The energy spent for the transmission of one information bit Eb , is Eb = P/R = P/(ηB) joules, since
with spending P watts power in T seconds we have sent C = RT bits. Now using the fact that
η ≤ log2
P
1+
N0 B
Eb
= log2 1 + η
,
N0
we conclude that
Eb
2η − 1
≥
.
N0
η
(23)
8
Thus, we need at least
2η −1
η
SNR per information bit to communicate reliably. More particularly, by letting
η → 0, we can compute the minimum required SNR per bit for reliable communications to be
Eb
N0
=
min
1
≈ −1.59 dB.
log2 e
That is, one needs at least -1.59 dB energy per information bit to communicate reliably, which is a
fundamental information theory limit.
B. Solution to the general case: waterfilling algorithm
In this section, we give the general solution to the optimization problems of the form (19). More
precisely, we are interested in finding the optimal solution to the following problem:
max
N
X
log2 (1 + γn ρn ) ,
(24)
n=1
subject to:
ρn ≥ 0, ∀n = 1, . . . , N
N
X
n=1
ρn ≤ ρ.
(25)
(26)
We assume that γn :s are given and that γ1 ≥ γ2 ≥ · · · ≥ γN > 0. First note that
! N !
N
N
N
X
Y
Y
Y 1
+ ρn
.
max
log2 (1 + γn ρn ) ≡ max
(1 + γn ρn ) ≡ max
γn
γn
n=1
n=1
n=1
n=1
(27)
Therefore, if any of the ρn :s are zero, these must be for ρn , n = M + 1, · · · , N, for some integer M,
since otherwise one can still maximize the objective. Thus we can further simplify (27) using,
! N !
! M !
!
N
N
N
Y
Y 1
Y
Y 1
Y
1
γn
+ ρn
+ ρn
=
γn
γn
γn
γn
n=1
n=1
n=1
n=1
n=M +1
!
!
M
M Y
Y
1
+ ρn
=
γn
γ
n
n=1
n=1
(28)
Using the inequality of arithmetic and geometric means:
N
1 X
xn ≥
N n=1
N
Y
n=1
xn
! N1
,
xn ≥ 0.
(29)
9
with equality if and only if are xn :s are equal, we can write
M Y
1
+ ρn ≤
γ
n
n=1
with equality if
!M
M 1 X 1
+ ρn
≤
M n=1 γn


!M
M
1
1 X 1
+ ρ
,
M n=1 γn M
(30)
1
γn
+ ρn = const. = µ
P
 N ρ =ρ
n=1 n
where the first and the second constraints are due to the first and the second inequalities in (30),
respectively. Therefore, the solution to the optimization problem is

n
o
 ρ∗ = max 0, µ − 1 ,
n
γn
 µ = 1 ρ + PM 1
n=1 γn
M
(31)
Note that to find M, that is to find the number of non-zero ρn :s, we can solve the above equations
P
iteratively. More precisely, we first fix µ, then we compute ρn :s from (31), and if N
n=1 ρn = ρ, then
PN
we return ρn :s as the final solution, otherwise if n=1 ρn < (>)ρ, we further increase (decrease) µ and
repeat the procedure. This procedure is summarized in Algorithm 1.
Algorithm 1 Waterfilling algorithm.
1. Set µ to an arbitrary
value;
n
o
2. Set ρn = max 0, µ − γ1n , ∀n = 1, . . . , N;
P
3. Compute t = N
n=1 ρn ;
if t < ρ then
Set µ = µ + δµ ;
Goto 2;
else if t > ρ then
Set µ = µ − δµ ;
Goto 2;
else
return ρn ;
end if
This procedure is known as waterfilling algorithm and the µ at the optimum solution is called the waterlevel. The reason for this name can be understood from Figure 5, where we have plotted the optimal ρ∗n
for an illustrative example with N = 5. As can be seen from this example, the waterfilling algorithm can
be thought of as filling water into a tank containing N cylinders with heights equal to
1
.
γn
Then µ will
be the water-level and the height of water above each cylinder n will be the optimal solution ρ∗n .
Let us finally apply the waterfilling algorithm to our problem. We have ρ =
PT
2
and γn =
1
.
λn N0 /2
The
ρ∗3 = 0
10
ρ∗5 = 0
PSfrag
1
Fig. 5.
ρ∗4
µ
ρ∗2
ρ∗1 = 0
1
γn
2
3
4
n
5
Waterfilling algorithm for an illustrative example with N = 5.
interpretation of the waterfilling algorithm in this problem is that we assign more power to the “good”
channels, i.e. the channels for whom the noise variance λn N0 /2 is smaller and “bad” channels may not
even be used.
On a final note, let us consider two extreme cases: (i) P is very small (low SNR regime), and (ii) P is
very large (high SNR regime). In case (i), as it can be seen from Figure 6 the optimum power allocation
is to allocate all the power to the best sub-channel.

N

 PT ,
n
=
argmin
2
k=1
pn =

 0,
otherwise
In this case, the capacity is C = log2 1 +
PT
λmin N0
≈
PT
λmin N0
λk
(32)
log2 e. In high SNR regime, the optimum
solution is roughly equal power allocation, as can be seen from Figure 7.
11
ρ∗4 = P
1
γn
µ
1
Fig. 6.
2
3
4
n
5
Waterfilling algorithm for an illustrative example with N = 5 and for low SNR regime.
1
γn
µ
111
000
000
111
0000
1111
0000
1111
000
111
000
111
000
111
0000
1111
0000
1111
000
111
000
111
000
111
0000
1111
0000
1111
000
111
000
111
000
111
0000
1111
0000
1111
000
111
000
111
000
111
0000
1111
0000
1111
000
111
000
111
000
111
0000
1111
0000
1111
000
111
000
111
000
111
0000
1111
0000
1111
000
111
000
111
000
111
0000
1111
0000
1111
000
111
000
111
000
111
0000
1111
0000
1111
000
111
000
111
000
111
0000
1111
0000
1111
000
111
000
111
000
111
0000
1111
0000
1111
000
111
000
111
000
111
0000
1111
0000
1111
000
111
000
111
000
111
0000
1111
000
111
1111
0000
000
111
000
111
0000
1111
0000
1111
000
111
000
111
000
111
0000
1111
000
111
000111
111
000
0000
1111
000
111
000
111
0000
1111
000
111
000
111
0000
1111
000
111
000
111
0000
1111
000
111
000
111
0000
1111
000
111
000
111
0000
1111
000
111
000
111
0000
1111
000
111
000
111
0000
1111
000
111
000 1111
111
0000
000
111
0000
1111
0000
1111
0000
1111
0000
1111
0000
1111
0000
1111
0000
1111
0000
1111
0000
1111
0000
1111
1
Fig. 7.
2
3
4
ρ∗n =
5
Waterfilling algorithm for an illustrative example with N = 5 and for high SNR regime.
P
2B
n