Factoring distributions
• Given random variables X1,…,Xn
• Partition variables V into sets A and
VnA as independent as possible
A
V
X1 X3
X4 X6
X1 X3
X4 X6
Formally: Want
X2
X5 X7
X2 X7
X5
VnA
A* = argminA I(XA; XVnA) s.t. 0<|A|<n
where I(XA,XB) = H(XB) - H(XB j XA)
4
Example: Mutual information
• Given random variables X1,…,Xn
• z(A) = I(XA; XVnA) = H(XVnA) – H(XVnA |XA)=z(V\A)
Lemma: Mutual information z(A) is submodular
z(A [ {s}) – z(A) = H(Xsj XA) – H(Xsj XVn(A[{s}) )
Nonincreasing in A: Nondecreasing in A
AµB ) H(Xs|XA) ¸ H(Xs|XB)
s(A) = z(A[{s})-z(A) monotonically nonincreasing
 z submodular 
5
Queyranne’s algorithm
[Queyranne ’98]
Theorem: There is a fully combinatorial,
strongly polynomial algorithm for solving
A* = argminA z(A) s.t. 0<|A|<n
for symmetric submodular functions z
• Runs in time O(n3) [instead of O(n8)…]
6
Why are pendent pairs useful?
• Key idea: Let (t,u) pendent, A* = argmin z(A)
Then EITHER
– t and u separated by A*, e.g., u2A*, tA*.
But then A*={u}!! OR
A*
– u and t are not separated by A*
A*
V
u
t
t
V
A*
u
u
V
t
Then we can merge u and t…
7