Information Complexity: an
Overview
Rotem Oshman, Princeton CCI
Based on work by Braverman, Barak, Chen, Rao,
and others
Charles River Science of Information Day 2014
Classical Information Theory
• Shannon ‘48, A Mathematical Theory of
Communication:
Motivation: Communication Complexity
𝑓 𝑋, 𝑌 = ?
𝑋
𝑌
Yao ‘79, “Some complexity questions related to
distributive computing”
Motivation: Communication Complexity
More generally: solve some task 𝑇(𝑋, 𝑌)
𝑋
𝑌
Yao ‘79, “Some complexity questions related to
distributive computing”
Motivation: Communication Complexity
• Applications:
– Circuit complexity
– Streaming algorithms
– Data structures
– Distributed computing
– Property testing
–…
Example: Streaming Lower Bounds
• Streaming algorithm:
How much space
is required to approximate f(data)?
algorithm
data
• Reduction from communication complexity
[AMS’97]
Example: Streaming Lower Bounds
• Streaming algorithm:
State of the
algorithm
algorithm
data
• Reduction from communication complexity
[Alon, Matias, Szegedy ’99]
Advances in Communication Complexity
• Very successful in proving unconditional lower
bounds, e.g.,
– Ω 𝑛 for set disjointness [KS’92, Razborov ‘92]
– Ω 𝑛 for gap hamming distance [Chakrabarti, Regev ‘10]
• But stuck on some hard questions
– Multi-party communication complexity
– Karchmer-Wigderson games
• [Chakrabarty, Shi, Wirth, Yao ’01], [Bar-Yossef, Kumar, Jayram,
Srivakumar ‘04]: use tools from information theory
Extending Information Theory to
Interactive Computation
• One-way communication:
– Task: send 𝑋 across the channel
– Cost: 𝐻 𝑋 bits
• Shannon: in the limit over many instances
• Huffman: 𝐻 𝑋 + 1 bits for one instance
• Interactive computation:
– Task: e.g., compute 𝑓 𝑋, 𝑌
– Cost?
Information Cost
• Reminder: mutual information
𝐼 𝑋; 𝑌 = 𝐻 𝑋 − 𝐻 𝑋 𝑌 = 𝐻 𝑌 − 𝐻 𝑌 𝑋
• Conditional mutual information:
𝐼 𝑋; 𝑌 𝑍 = 𝐻 𝑋 𝑍 − 𝐻 𝑋 𝑌, 𝑍
= 𝐸𝑧 𝐼 𝑋; 𝑌 𝑍 = 𝑧
• Basic properties:
– 𝐼 𝑋; 𝑌 ≥ 0
– 𝐼 𝑋; 𝑌 ≤ 𝐻 𝑋 and 𝐼 𝑋; 𝑌 ≤ 𝐻 𝑌
– Chain rule: 𝐼 𝑋𝑌; 𝑍 = 𝐼 𝑋; 𝑍 + 𝐼 𝑌; 𝑍 𝑋
Information Cost
• Fix a protocol Π
• Notation abuse: let Π also denote the
transcript of the protocol
• Two ways to measure information cost:
– External information cost: 𝐼 Π; 𝑋𝑌
– Internal information cost: 𝐼 Π; 𝑌 𝑋 + 𝐼 Π; 𝑋 𝑌
– Cost of a task: infimum over all protocols
– Which cost is “the right one”?
Information Cost: Basic Properties
External information: 𝐼 Π; 𝑋𝑌
Internal information: 𝐼 Π; 𝑌 𝑋 + 𝐼 Π; 𝑋 𝑌
• Internal ≤ external
• Can be much smaller, e.g.:
– 𝑋 = 𝑌 uniform over 0,1
– Π: Alice sends 𝑋 to Bob
𝑛
• But equal if 𝑋, 𝑌 inependent
Information Cost: Basic Properties
External information: 𝐼 Π; 𝑋𝑌
Internal information: 𝐼 Π; 𝑌 𝑋 + 𝐼 Π; 𝑋 𝑌
• External information ≤ communication:
𝐼 Π; 𝑋𝑌 ≤ 𝐻 Π ≤ Π .
Information Cost: Basic Properties
• Internal information ≤ communication cost:
𝐼 Π; 𝑌 𝑋 + 𝐼(Π; 𝑋|𝑌) ≤ Π .
• By induction: let Π = Π1 … Π𝑡 .
• ∀𝑟 ≤ 𝑡 : 𝐼 Π≤𝑟 ; 𝑌 𝑋 + 𝐼 Π≤𝑟 ; 𝑋 𝑌 ≤ 𝑟.
𝐼 Π≤𝑟 ; 𝑌 𝑋 + 𝐼 Π≤𝑟 ; 𝑋 𝑌
what we know after r rounds
= 𝐼 Π<𝑟 ; 𝑌 𝑋 + 𝐼 Π<𝑟 ; 𝑋 𝑌 what we knew after r-1 rounds
I.H. ≤ 𝑟 − 1
+ 𝐼 Π𝑟 ; Y X, Π<𝑟 + 𝐼 Π𝑟 ; X Y, Π<𝑟
what we learn in round r, given what we already know
Information vs. Communication
• Want: 𝐼 Π𝑟 ; Y X, Π<𝑟 + 𝐼 Π𝑟 ; X Y, Π<𝑟 ≤ 1.
• Suppose Π𝑟 is sent by Alice.
• What does Alice learn?
– Π𝑟 is a function of Π<𝑟 and 𝑋, so
𝐼 Π𝑟 ; Y X, Π<𝑟 = 0.
• What does Bob learn?
– 𝐼 Π𝑟 ; Y X, Π<𝑟 ≤ Π𝑟 = 1.
Information vs. Communication
• We have:
Internal information ≤ communication
External information ≤ communication
Internal information ≤ external information
Information vs. Communication
• “Information cost = communication cost”?
– In the limit: internal information! [Braverman, Rao ‘10]
– For one instance: external information! [Braverman,
Barak, Rao, Chen ‘10]
Big question: can protocols be compressed down to
their internal information cost?
– [Ganor, Kol, Raz ’14]: no!
– There is a task with internal IC=𝑘, CC=2𝑘 .
… but: remains open for functions, small output.
Information vs. Amortized
Communication
• Theorem [Braverman, Rao ‘10]:
•
•
•
•
𝐶𝐶(𝐹 𝑛 , 𝜇𝑛 , 𝜖)
lim
= 𝐼𝐶 𝐹, 𝜇, 𝜖 .
𝑛→∞
𝑛
The “≤” direction: compression
The “≥” direction: direct sum
We know: 𝐶𝐶 𝐹 𝑛 , 𝜇𝑛 , 𝜖 ≥ 𝐼𝐶 𝐹 𝑛 , 𝜇𝑛 , 𝜖
We can show: 𝐼𝐶 𝐹 𝑛 , 𝜇𝑛 , 𝜖 = 𝑛 ⋅ 𝐼𝐶 𝐹, 𝜇, 𝜖
Direct Sum Theorem [BR‘10]
𝐼𝐶 𝐹 𝑛 , 𝜇𝑛 , 𝜖 = 𝑛 ⋅ 𝐼𝐶 𝐹, 𝜇, 𝜖 :
• Let Π be a protocol for 𝐹 𝑛 on 𝑛-copy inputs 𝑋, 𝑌
• Construct Π′ for 𝐹 as follows:
– Alice and Bob get inputs 𝑈, 𝑉
– Choose a random coordinate 𝑖 ∈ 𝑛 , set 𝑋𝑖 = 𝑈, 𝑌𝑖 = 𝑉
– Bad idea: publicly sample 𝑋−𝑖 , 𝑌−𝑖
𝑋
𝑈
𝑌
𝑉
Direct Sum Theorem [BR‘10]
𝐼𝐶 𝐹 𝑛 , 𝜇𝑛 , 𝜖 = 𝑛 ⋅ 𝐼𝐶 𝐹, 𝜇, 𝜖 :
• Let Π be a protocol for 𝐹 𝑛 on 𝑛-copy inputs 𝑋, 𝑌
• Construct Π′ for 𝐹 as follows:
– Alice and Bob get inputs 𝑈, 𝑉
– Choose a random coordinate 𝑖 ∈ 𝑛 , set 𝑋𝑖 = 𝑈, 𝑌𝑖 = 𝑉
– Bad idea: publicly sample 𝑋−𝑖 , 𝑌−𝑖
Suppose in Π, Alice sends 𝑋1 ⊕ ⋯ ⊕ 𝑋𝑛 .
In Π, Bob learns one bit ⇒ in Π ′ he should learn 1/𝑛 bit
But if 𝑋−𝑖 is public Bob learns 1 bit about 𝑈!
Direct Sum Theorem [BR‘10]
𝐼𝐶 𝐹 𝑛 , 𝜇𝑛 , 𝜖 = 𝑛 ⋅ 𝐼𝐶 𝐹, 𝜇, 𝜖 :
• Let Π be a protocol for 𝐹 𝑛 on 𝑛-copy inputs 𝑋, 𝑌
• Construct Π′ for 𝐹 as follows:
– Alice and Bob get inputs 𝑈, 𝑉
– Choose a random coordinate 𝑖 ∈ 𝑛 , set 𝑋𝑖 = 𝑈, 𝑌𝑖 = 𝑉
Publicly sample 𝑋1 , … , 𝑋𝑖−1
𝑋
Privately sample 𝑋(𝑖+1) , … , 𝑋𝑛
𝑈
Privately sample 𝑌1 , … , 𝑌𝑖−1
𝑌
Publicly sample 𝑌 𝑖+1 , … , 𝑌𝑛
𝑉
Compression
• What we know: a protocol with
communication 𝐶, internal info 𝐼 and external
info 𝐼𝑒𝑥𝑡 can be compressed to
– 𝐼𝑒𝑥𝑡 ⋅ polylog 𝐶 [BBCR’10]
– 𝐼 ⋅ 𝐶 ⋅ polylog 𝐶 [BBCR’10]
– 2𝑂
𝐼
[Braverman’10]
• Major open question: can we compress to 𝐼 ⋅
polylog 𝐶 ? [GKR, partial answer: no]
Using Information Complexity to Prove
Communication Lower Bounds
• Internal/external info ≤ communication
• Essentially the most powerful technique known
[Kerenidis,Laplante,Lerays,Roland,Xiao’12]: most
lower bound techniques imply IC lower bounds
• Disadvantage: hard to show incompressibility!
– Must exhibit problem with low IC, high CC
– But proving high CC usually proves high IC…
Extending IC to Multiple Players
• Recent interest in multi-player number-inhand communication complexity
• Motivated by “big data”:
– Streaming and sketching, e.g., [Woodruff, Zhang
‘11,’12,’13]
– Distributed learning, e.g., [Awasthi, Balcan, Long ‘14]
Extending IC to Multiple Players
• Multi-player computation traditionally hard to
analyze
• [Braverman,Ellen,O.,Pitassi,Vaikuntanathan]:
Ω 𝑛𝑘 for Set Disjointness with 𝑛 elements, 𝑘
players, private channels, NIH input
Information Complexity on Private
Channels
• First obstacle: secure multi-party computation
• [Goldreich,Micali,Wigderson’87]: any function can be
computed with perfect information-theoretic security
against < 𝑘/2 players
– Solution: redefine information cost, measure both
• Information a player learns, and
• Information a player leaks to all the others.
Extending IC to Multiple Players
• Set disjointness:
– Input: 𝑋1 , … , 𝑋𝑘
– Output: 𝑋1 ∩ ⋯ ∩ 𝑋𝑘 = ∅?
• Open problem: can we extend to gap set
disjointness?
– First step: “purely info-theoretic” 2-party analysis
Extending IC to Multiple Players
• In [Braverman,Ellen,O.,Pitassi,Vaikuntanathan]
we show direct sum for multi-party
– Solving 𝑛 instances = 𝑛 ⋅ solving one instance
• Does direct sum hold “across players”?
– Solving with 𝑘 players = Ω 𝑘 ⋅ solving with 2
players?
– Not always
• Does compression work for multi-party?
Conclusion
• Information complexity extends classical
information theory to the interactive setting
• Picture is much less well-understood
• Powerful tool for lower bounds
• Fascinating open problems:
– Compression
– Information complexity for multi-player
computation, quantum communication, …