Information Theory of Wireless Networks
David Tse
Wireless Foundations
U.C. Berkeley
Information Theory Summer School
Penn State
June 3, 2008
The Holy Grail
• Shannon’s information theory provides the basis for
all modern-day communication systems.
• His original theory was point-to-point.
• After 60 years we are still very far away from
generalizing the theory to networks.
• We propose approaches to make progress in the
context of wireless networks.
Modeling the Wireless Medium
• broadcast
• interference
• high dynamic range in channel strengths between
different nodes
• Basic model: additive Gaussian channel:
Gaussian Network Capacity: What We Know
Tx
Rx
point-to-point (Shannon 48)
C = log2 (1 + SNR)
Tx 1
Rx1
Rx
Tx 2
multiple-access
(Alshwede, Liao 70’s)
Tx
Rx 2
broadcast
(Cover, Bergmans 70’s)
What We Don’t Know
Unfortunately we don’t know the capacity of most
other Gaussian networks.
Tx 1
Rx 1
Tx 2
Rx 2
Interference
(Best known achievable region: Han & Kobayashi 81)
Relay
S
relay
D
(Best known achievable region: El Gamal & Cover 79)
Recent Progress
• Approximate.
• Similar evolution has happened in other fields:
– fluid and heavy-traffic approximation in queueing networks
– approximation algorithms in CS theory
• Approximation should be good in engineering-relevant
regimes.
Two Regimes
• Interference-limited regime:
– noise is small compared to signals.
– focuses on interactions between signals rather than the
stochastic noise
• Large-network regime:
– networks with large number of nodes
– focuses on macroscopic scaling laws
Overview
• Lecture 1: interference channels
– 2-user interference channel capacity to within 1 bit/s/Hz
– Deeper insight via a deterministic channel model
– Generalization to some many-user interference channels.
• Lecture 2: relay networks
– relay channel capacity to within 1 bit/s/Hz
– single-source-single-destination relay network capacity to
within constant gap
Overview
• Lecture 3: Large networks
– Networks with many source-destination pairs
– Information theoretic optimal scaling laws and architectures
Lecture 1: Interference Channels
Interference
• Interference management is an central problem in
wireless system design.
• Within same system (eg. adjacent cells in a cellular
system) or across different systems (eg. multiple WiFi
networks)
• Two basic approaches:
– orthogonalize into different bands
– full sharing of spectrum but treating interference as noise
• What does information theory have to say about the
optimal thing to do?
Two-User Gaussian Interference Channel
message m1
want m1
message m2
want m2
• Characterized by 4 parameters:
– Signal-to-noise ratios SNR1, SNR2 at Rx 1 and 2.
– Interference-to-noise ratios INR2->1, INR1->2 at Rx 1 and 2.
Related Results
• If receivers can cooperate, this is a
multiple access channel. Capacity is
known. (Ahlswede 71, Liao 72)
• If transmitters can cooperate , this is a
MIMO broadcast channel. Capacity
recently found.
(Weingarten et al 04)
• When there is no cooperation of all, it’s
the interference channel. Open
problem for 30 years.
State-of-the-Art in 2006
• If INR1->2 > SNR1 and INR2->1 > SNR2, then capacity
region Cint is known (strong interference, HanKobayashi 1981, Sato 81)
• Capacity is unknown for any other parameter ranges.
• Best known achievable region is due to HanKobayashi (1981).
• Hard to compute explicitly.
• Unclear if it is optimal or even how far from capacity.
• Some outer bounds exist but unclear how tight (Sato
78, Costa 85, Kramer 04).
Review: Strong Interference Capacity
• INR1->2 > SNR1, INR2->1> SNR2
• Key idea: in any achievable
scheme, each user must be able
to decode the other user’s
message.
• Information sent from each
transmitter must be common
information, decodable by all.
• The interference channel capacity
region is the intersection of the
two MAC regions, one at each
receiver.
Han-Kobayashi Achievable Scheme
common
W1
private
U1
decode
W1
W2
then
U1
decode
common
private
W2
U2
W2
W1
then
U2
• Problems of computing the HK region:
- optimal auxillary r.v.’s unknown
- time-sharing over many choices of auxillary r.v,’s may be
required.
Interference-Limited Regime
• At low SNR, links are noise-limited and interference
plays little role.
• At high SNR and high INR, links are interferencelimited and interference plays a central role.
• Classical measure of performance in the high SNR
regime is the degree of freedom.
Baselines (Symmetric Channel)
• Point-to-point capacity:
Cawgn = log(1 + SNR)
d:o:f : :=
lim
SNR! 1
C
= 1
log SNR
• Achievable rate by orthogonalizing:
1
R = log(1 + 2SNR)
2
d:o:f : :=
1
2
• Achievable rate by treating interference as noise:
SNR
R = log(1 +
)
1 + INR
d:o:f: :=
?
Degree of Freedom
• Let both SNR and INR to grow, but fixing the ratio:
log INR
= ®:
log SNR
• Treating interference as noise:
R
=
d :=
SNR
log(1 +
)
1 + INR
R
lim
= 1¡ ®
SNR;INR! 1 log SNR
Dof plot
1¡
®
2
Optimal Gaussian HK
®
1¡ ®
®
2
Dof-Optimal Han-Kobayashi
• Only a single split: no time-sharing.
• Private power set so that interference is received
at noise level at the other receiver.
Why set INRp = 0 dB?
• This is a sweet spot where the damage to the other
link is small but can get a high rate in own link since
SNR > INR.
Can we do Better?
• We identified the Gaussian HK scheme that achieves
optimal dof.
• But can one do better by using non-Gaussian inputs
or a scheme other than HK?
• Answer turns out to be no.
• The dof achieved by the simple HK scheme is the dof
of the interference channel.
• To prove this, we need outer bounds.
Upper Bound: Z-Channel
• Equivalently, x1 given to Rx 2 as side information.
How Good is this Bound?
2
3
What’s going on?
Scheme has 2 distinct regimes of operation:
log INR
2
log INR
2
>
<
log SNR
3
log SNR
3
Z-channel bound is tight.
Z-channel bound is not tight.
New Upper Bound
• Genie only allows to give away the common
information of user i to receiver i.
• Results in a new interference channel.
• Capacity of this channel can be explicitly computed!
New Upper Bound + Z-Channel Bound is Tight
2
3
Back from Infinity
In fact, the simple HK scheme can achieve within 1
bit/s/Hz of capacity for all values of channel
parameters:
For any(R1 ; R2 ) in Cint, this scheme can achieve
~1 ; R
~2 )
rates ( R
~1
R
~2
R
(Etkin, T. & Wang 06)
¸
R1 ¡ 1
¸
R2 ¡ 1
From 1-Bit to 0-Bit
The new upper bound can further be sharpened to
get exact results in the low-interference regime ( <
1/3).
(Shang,Kramer,Chen 07,
Annaprueddy & Veeravalli08,
Motahari&Khandani07)
From Low-Noise to No-Noise
• The 1-bit result was obtained by first analyzing the
dof of the Gaussian interference channel in the lownoise regime .
• Turns out there is a deterministic interference
channel which captures exactly the behavior of the
interference-limited Gaussian channel.
• Identifying this underlying deterministic structure
allows us to generalize the approach.
Point-to-Point
Gaussian
Cawgn (SNR) =
1
2
Deterministic
log(1 + SNR)
Transmit a real number
Least significant bits are
truncated at noise level.
If
p
SNR = 2n we have
Cdet (n) = n
n / SNR on the dB scale
Multiple Access
Gaussian
SNR1 ¸ SNR2
SNR1
Deterministic
user 2
SNR2
user 1 sends cloud centers, user 2
sends clouds.
mod 2
addition
user 1
Comparing Multiple Access Capacity Regions
Gaussian
Deterministic
SNR1 ¸ SNR2
SNR1
user 2
SNR2
mod 2
addition
user 1
R2
R2
R1 + R2 =
n2
logSNR2
log(1 + SNR1 + SNR2 )
¼ logSNR1
accurate to within
(3; 2)
1 bit per user
logSNR1
R1
n 1 R1
Broadcast
Gaussian
Deterministic
n1 = 5
user 1
SNR1 ¸ SNR2
SNR1
SNR2
n2 = 2
log(1 +SNR2 )


n2
R2
To within 1 bit

R1
n1

log(1 +SNR1)
user 2
Interference
Gaussian
Deterministic
n
In symmetric case, channel
described by two parameters:
SNR, INR
n $ log2 SNR: m $ log2 INR
Capacity can be computed using
a result by El Gamal and Costa 82.
m
Applying El Gamal and Costa
Han-Kobayashi with V1, V2 as common information is optimal.
Csym
1
=
sup min [H (Y1 ) + H (Y2 jV1 ; V2 ); H (Y1 jV1 ) + H (Y2 jV2 )]
2 P X 1 ;P X 2
Optimal inputs X1*, X2* uniform on
the input alphabet.
X1
Simultaneously maximizes all entropy terms. X 2
Y1
V2
V1
Y2
Symmetric Deterministic Capacity
r=
1
1/2
® =
=
®=
r =
2
3
1
3
®=
r =
2
3
2
3
log INR
log SNR
m
n
Extension:
Many-to-One Interference Channel
Gaussian
Deterministic
Deterministic capacity can be computed exactly .
Gaussian capacity to within constant gap, using structured
codes and interference alignment.
(Bresler, Parekh & T. 07)
Example
Tx0
Rx0
Tx1
Rx1
Tx2
Rx2
• Interference from users 1 and 2 is aligned at the MSB
at user 0’s receiver in the deterministic channel.
• How can we mimic it for the Gaussian channel ?
Gaussian
Lattice
codes
Han-Kobayashi
can achieveNot
constant
Optimal
gap
• Suppose users 1 and 2 use a random
Gaussian codebook:
Random Code
Sum of Two Random Codebooks
Lattice Code for Users 1 and 2
Interference from users 1 and 2 fills the space: no
room for user 0. User 0 Code
Tx0
Rx0
Tx1
Rx1
Tx2
Rx2
Interference Channels: Recap
• In two-user case, we showed that an existing strategy
can achieve within 1 bit to optimality.
• In many-to-one case, we showed that a new strategy
can do much better.
• General K-user interference channel still open.
Lecture 2: Relay Networks
Relay Networks
D
S
relays
• We consider here single-source-single-destination
networks.
• Relays cooperate to help source transmit information to the
destination.
• What is the capacity?
History
• The (single) relay channel was first proposed by Van der
Meulen in 1971.
• Cover and El Gamal (1979) provided a whole array of
achievable strategies:
– decode-and-forward
– Partial-decode-and-forward
– Compress-and-forward
• Recent generalization of these techniques to more than 1 relay
(Gupta & Kumar, Xie & Kumar, Kramer, Gastpar and Gupta).
• Performance of these schemes on general networks not easily
characterizable.
Approach
• General upper bound: cutset bound.
• Are any of these schemes within a constant gap of
optimality for general networks?
• If not, find a scheme which is.
• We approach the problem via the deterministic
model.
(Avestimehr, Diggavi & T. 07)
Example 1: Single Relay
Gaussian
hSR
R
hSD
S
Deterministic
hRD
D
nSR
nRD
Theorem:
Gap from cutset bound is at most 1 bit.
nSD
¹ cut set = min (max(nSD ; nSR ); max(nSD ; nRD ))
C
| hRD |2
| hSD |2
gap
| hSR |2
| hSD |2
On average it is much less than 1-bit
Decode-Forward is near optimal
¡
¢
= nSD + min (nSR ¡ nSD ) + ; (nR D ¡ nSD ) +
Cutset bound is achievable.
Decode-Forward is optimal
Example 2: Two relays
Deterministic
Gaussian
hSR1
R1
hR1D
R1
D
S
hSR2
R2
S
hR2D
 (r1 , r2 )  BC S ( R1, R 2)
R  max (r1  r2 : s.t. 
)
(
r
,
r
)

MAC
( R1, R 2 ) D
 1 2
50%
D
R2
max( n SR1,n SR 2 ),max( n R1D ,n R 2D ),
C  min 

,n SR 2  n R1D
n SR1  n R 2D

40%
30%
20%
10%
0
0.4
0.8
1.2
1.6
2
gap

Partial Decode-Forward is near optimal
Partial Decode-Forward is optimal
Example 3: A two-stage network
Deterministic
Gaussian
A1
B1
A1
S
D
D
S
A2
A2
B1
B2
B2
What is a good scheme?
At relay B1, the two LSB’s are
forwarded.
B1 does not do any decoding or
partial decoding of the message.
Questions
• Is the cutset bound always achieved on arbitrary
networks for the deterministic model?
• What is the structure of the capacity-achieving
strategy?
• Can we mimic that to find an approximately optimal
strategy for general Gaussian networks?
Algebraic representation
A1
b1
b2
b3
b4
b5
B1
D
S
A2
c1
c2
c3
c4
c5
B2
• Received Signal:
0 0
0 0
   
j  b1   0 
1
b2  ciN
j
b3 c 2
yB1 y (t) 
– q=max(nij), S: shift matrix (q×q)
matrixare
ofinsize
– S:
All shift
operations
F2 5

S S
qn
ij
53
52
xxi (t)

S
xA 2
A1
General linear finite-field model
• Channel from i to j is described by an arbitrary channel matrix
q
F
Gij operating on 2
• Received signal:
M
y j (t )   Gij xi (t )
i 1
– Deterministic model:
Gij  S
q  nij
– Wireline network also a special case
(mod 2)
Cut-set upper bound
A1
B1
D
S
A2
B2
c

Crelay  C  max min I(X ; Yc | Xc )
PX1 ,...,X M

For deterministic, linear finite field model
C  min rank (Gc )

Achievability of Cutset Bound
Theorem: Cutset bound is achievable,
Crelay  C  min rank (Gc )

 In wireline networks, rank (G c ) is just summation of
the capacity of links from  to c
 This theorem is a generalization of Ford-Fulkerson maxflow min-cut theorem
 Also holds in the multicast scenario
 Generalization of network coding to achieve the multicast capacity of wireline
networks (Ahlswede-Cai-Li-Yeung)
Special case: Staged networks
A1
m3 m2 m1
B1
D
S
A2
ˆ 1, m
ˆ 2 ,...
m
B2
•
•
Lengths of all paths from S to D are the same
Major simplification
– messages do not mix in the network
•
Use a random network coding strategy (similar to Ahlswede et. al. 2000):
– S: map each message into a random codeword of length T symbol times
– Each relay randomly maps the received signal into a transmit codeword
•
Min-cut is achieved
Error Analysis
• Assume m is transmitted
– D can not distinguish between m and m’ if yD=y’D
y’A1:x’A1
yA1
yA1
:x:x
A1A1


Gc x   Gc x'

y’B1:x’B1
yyB1
:x B1
B1:xB1
y’D
yD
x’S
xS

 Gc ( x   x ' )  0
yA2:xA2
y’A2:x’A2
y
:xB2
yB2
B2:xB2
y’B2:x’B2
P(m  m' )   P(nodes  distinguis h m, m' & nodes c don' t distinguis h)

 2
T rank( G
c
)

– By union bound, any rate
R  min rank (Gc )

is achievable
General networks
•
Consider the time-expanded network with k stages, each T symbol times long
•
It is a multi-stage network!
 Apply the same strategy on (super) messages
•
S
Can achieve 1/k of the min-cut of time-expanded network (C)k


r[1]


S[1]
r[2]
A[1]
n2
n3
B[1]
n5
t[1]
n4


r[3]
n1
A[2]
B[2]
n2
n3
n5
n4


r[4]
A
S[4]
A[3]
B[3]
t[3]
k=4
n2
n3
n5
n4


n4
n1
n1
D[3]
D[2]
t[2]


S[3]
S[2]
n1
D[1]


A[4]


B
D
D
n5
n2
D[4]
t[4]
n3
S
B[4]
General networks (cont.)
• Key Question: Is
Ck
C
k  k
lim
, min-cut of the original network?
– Issue: there are more cuts in the time expanded graph
• Yes!
• Min-cut is achieved
S


r[1]


S[1]
r[2]
A[1]
n2
n3
B[1]
n5
t[1]
n4


r[3]
n1
A[2]
B[2]
n2
n3
n5
n4


r[4]
A
S[4]
A[3]
B[3]
t[3]
k=4
n2
n3
n5
n4


n4
n1
n1
D[3]
D[2]
t[2]


S[3]
S[2]
n1
D[1]


A[4]


B
D
D
n5
n2
D[4]
t[4]
n3
S
B[4]
Proof
•
Cut value is:
k1
 H(y
R i 1
| xR c )
i
i1
•
Lower bound cut value by sum of k cut-values of original network.
•
Theorem:
k1
 H(y

i1
k
R i 1
~
Rm 
| x R c )  H(y R1 | x R c )   H(y R˜ m | x R˜ c )
i
k

m
m1
( Ri1  ...  Rim ) ,m=1…k
{i1 ,..., im }{1,..., k }
={nodes that appear at least m times in Ri’s}
•
Proof: Based
 on Submodularity of entropy function
r[1]
S
R1c
r[3]
r[2]
r[4]
S[1]
S[2]
S[3]
S[4]
A[1]
A[2]
A[3]
A[4]
A
3
S
R1
B[1]
D[1]
t[1]
B[2]
R2
D[2]
t[2]
B[3]
R3
D[3]
t[3]
4
1
D
B[4]
R4
t[4]
5
2
D[4]
B
D
Back to Gaussian Networks
Deterministic



Gaussian
S encodes the message over T
symbol times
Each relay randomly maps the
received signal into a transmit
codeword
D decodes the message
deterministically
•
S encodes the message over T
symbol times
•
Each relay,
– Quantizes the received signal at
noise level
– Randomly maps it into a Gaussian
codeword
•
D decodes the message by finding
the one that is jointly typical with yD
optimal
approx. optimal
yˆ A1 : xA1
yˆ B1 : xB1
yD
xS


yˆ A 2 : xA 2 yˆ B 2 : xB 2
Properties of the scheme
• Simple
 Quantize
 Map to a transmit codeword
• Relays don’t need any channel information
• How does it perform?
m ball m’ ball
A1
m ball m’ ball
yˆ D
B1
m’
m S
D

A2
m ball m’ ball
B2
m ball m’ ball
m ball
m’ ball
Approximate Capacity of Gaussian Networks
• Theorem: for any Gaussian relay network
C   C  C
- C is the cut-set upper bound on the capacity
-


  5 |V | is a constant that depends on size of the network, but not
the channel gains or SNR’s of the links
- Uniform approximation of the capacity for all values of channel gains.
Earlier Work on Deterministic Relay Networks
• Single-relay semi-deterministic channel (El Gamal &
Aref 79)
• Aref’s relay network with general deterministic
broadcast but no interference (Aref 80, Ratnakar &
Kramer 06)
• But connection to Gaussian networks missing.