Mathematical theory of collective behaviour
Sheet 5
Maxent inference and revision
The tutor will provide feedback on the first two exercises if the sheet is handed in
before the next tutorial session (Mon after Reading Week).
1. Maxent distribution with constrained probabilities
(a) You are told that a biased die with outcomes x ∈ {1, . . . , 6} has probability 0.4 of
coming up x = 5 or x = 6, i.e. P (x = 5) + P (x = 6) = 0.4. Find the maxent
distribution P (x) given this constraint.
Hint: The constraint can be written in terms of the average of a function f1 (x) which
is defined as f1 (x) = 1 for x = 5 or 6, and f1 (x) = 0 otherwise. Find Z and F as
usual, deduce hf1 i, solve for λ1 .
(b) Compute and sketch the entropy (in natural units) of the maxent distribution as a
function of λ1 . Explain why limλ1 →+∞ S[P ] = ln 2 and limλ1 →−∞ S[P ] = ln 4 in the
light of what happens to P (x) in these extreme cases.
(c) Generalize your calculation of the maxent distribution P (x) in (a) to P (x = 5) +
P (x = 6) = p56 for arbitrary p56 between 0 and 1.
(d) Generalize (c) further to the case where there is an additional constraint P (x =
1) + P (x = 2) + P (x = 3) = p123 .
2. Relation between entropy and multinomial coefficients
The number of ways of arranging N balls in a line, where the balls have I different colours
and there are ni balls of colour i, is the multinomial coefficient N !/(n1 ! · · · nI !). Use the
Stirling approximation to show that to exponential accuracy this is exp[N S(p̂1 , . . . , p̂I )]
where p̂i = ni /N and S is the entropy in natural units of the distribution {p̂i }. (Formally,
this means showing that (1/N ) ln[N !/(n1 ! · · · nI !)] → S(p̂1 , . . . , p̂I ) for N → ∞ at fixed
p̂i = ni /N .)
3. Maxent distribution with overlapping constrained probabilities
You are told that a biased die with outcomes x ∈ {1, . . . , 6} has probability 15/23 of
coming up with a value x ≥ 4, and probability 16/23 of coming up with an even x. Find
the maxent distribution P (x) given these constraints.
Hint: The constraints can again be written in terms of averages of suitably defined f1 (x)
and f2 (x). Find Z and F in terms of λ1 and λ2 , and deduce hf1 i and hf2 i. Then solve
for z1 = exp(λ1 ) and z2 = exp(λ2 ).
4. Maxent distribution for a four-sided die
Let x ∈ {1, 2, 3, 4} be the number of spots on a (biased) four-sided die. Your friend has
rolled the die a large number of times and found that the average number of spots rolled
is hxi = 49/15. Find the maximum entropy distribution P (x) subject to this constraint.
Hint: The constraint that hxi = 49/15 gives you an equation for the Lagrange multiplier,
which you can solve by inspection: try integer values for eλ1 .
Next, sketch the entropy S[P ] of the maxent solution as a function of the multiplier λ1 .
What happens to S[P ] and to P (x) for λ1 = 0 and λ1 = ±∞?
5. Geometric and arithmetic means inequality
Using the Lagrange multiplier method, prove that the geometric mean of a set of three
positive numbers (xyz)1/3 is no greater than the arithmetic mean (x + y + z)/3. [Hint:
find the maximum value of xyz, subject to x + y + z = k]. Argue that the same is true for
n positive numbers x1 , . . . , xn , that is (x1 x2 · · · xn )1/n ≤ (x1 + x2 + · · · + xn )/n.
Next, prove the n-variables case again using Jensen’s inequality.
6. Evaluate the expressions
P
• I1 (N ) = N
k,j=1 k exp(j)
PN
• I2 (N ) = i,j,k=1 exp(i + j + k)
P
QN
1/N ` exp(`)
• I3 (N, M ) = M
k=1
`=1 k