PD Dr. Torsten Enßlin
Robin Dehde
Margret Westerkamp
Philipp Frank
Information Field Theory
Summer term 2017
Exercise sheet 4
Please note, that there will be no Thursday tutorial on May, 25th due to public holiday.
Please distribute yourself among the two Wednesday tutorials (8 am- 10 am & 4pm6pm).
Exercise 4 - 1
Bob flips a (probably) manipulated coin as long as he gets tails. The moment the coin lands with
head up, he stops the tossing. n denotes the number of tails he got. The coins’ probability to land
on tail may be f . Bob’s strategy is denoted by by B.
a) Calculate P (n | f, B) (1 point).
b) Calculate the expected number of toins, i.e. hni(n|f,B) ( 2 points).
c) Bob performs one tossing experiment from a) and gets n tails in a row, which he tells Alice. So
far Alice does not know how the experiment was conducted and likes to infer the unfairness of
the coin.
• Until now Alice believes that Bob performed a coin toss experiment of predetermined
length n + 1. This strategy is called A. Calculate the most probable f using P (f | n, A) (2
points).
• Now Bob tells Alice that he ended the tossing when he got the first head. She therefore
infers the most probable f using P (n | f, B) and Bayes Theorem. Calculate P (f | n, B).
• Compare the results and discuss if the finding is surprising (1 point).
d) As Alice knows that the maximum of a non-symmetric probability distribution is not equal to
its expectation value she uses a computer algebra system of her choice to plot the probability
distribution of f .
Calculate hf i(f |n,B) and compare with your results from b) (2 points).
Exercise 4 - 2
A damaged clock only showing full hours d ∈ {0, 1, ..., 11} advances the hour erratically with a
known probability f ∈ [0, 1]. You set the damaged clock to d = 0 the moment you go to sleep.
When you wake up, the clock shows 6 o’clock.
a) What is the probability that after sleeping t ∈ {0, 1, ..., 11} hours the clock shows time d ∈
{0, 1, ..., 11}. (1 point)
b) Your personal experience tells you that you either wake up after 8 or 9 hours of sleep. When
you wake up the clock shows 6 o’clock. For a given f , what are the odds that you slept 8 against
you slept 9 hours? For which f would you be completely clueless about the number of hours
you slept? (2 points)
Exercise 4 - 3
Consider the following coin toss experiment:
• A large number n of coin tosses are performed and the results are stored in a data vector
n
d(n) = (d1 , . . . , dn ) ∈ {0, 1} , where 0 and 1 represent the possible outcomes head and tail.
• Individual tosses are independent from each other.
1/2
PD Dr. Torsten Enßlin
Robin Dehde
Margret Westerkamp
Philipp Frank
Information Field Theory
Summer term 2017
• All tosses are done with the same coin with an unknown bias f ∈ [0, 1]; i.e.,
P(di |f ) = f di (1 − f )1−di .
Assume that a fraction f¯ out of the n coin tosses yielded head.
a) Derive the Gaussian approximation of the PDF P(f |d(n) ) around its maximum. – You can use
a saddle point approximation; i.e., identify the maximum, and taylor-expand the (negative)
logarithm of P(f |d(n) ) around it up to second order in order to identify the variance of the
Gaussian (3 points).
b) Use this Gaussian approximation to derive an approximation for P(dn+1 |d(n) ).
Hint: You can assume that the Gaussian distribution is narrow enough such that the integration
boundaries [0, 1] can be replaced by (−∞, ∞) (2 points).
c) Now calculate the exact posterior mean for f¯ and the exact expression for P(dn+1 |d(n) ) (2
points).
R1
Note: 0 dx xα (1 − x)β = Γ(α + 1)Γ(β + 1)/Γ(α + β + 2), where Γ is the Gamma function.
Exercise 4 - 4
Assume that for a PDF P (x) only a set of constraints —typically moment constraints— are known,
like
Q (x) = P (x | I) = const,
P (x) > 0,
Z
P (x) dx = 1,
Z
P (x)fi (x) dx = αi .
From the principle of maximum entropy one derives that
!
n
X
P (x) = exp λ0 +
λi fi (x)
i=1
with λi such that the constraints are satisfied.
Assume a PDF P (x|α, µ) with x ∈ R, α > 0, µ ∈ R and h|x − µ|i(x|α,µ) = α. Calculate the PDF of
maximum entropy. (3 points)
This exercise sheet will be discussed during the exercises on Wednesday, May 24th, in the morning
as well as in the evening tutorial. Please note that there will be not Thursday tutorial, due to
public holiday May, 25th. Please distribute yourself among the Wednesday tutorial.
www.mpa-garching.mpg.de/∼ensslin/lectures
2/2