Homework 1
1
40-957 Special Topics in Artificial Intelligence:
Probabilistic Graphical Models
Homework 1
Due data: 1392/12/18
Directions: To submit your assignment, make a zip file containing a pdf
file of your writeup (write it by hand and then scan it) and your code,
then submit it to "[email protected]" by email with the subject
"HW1 Submission-student number" (for example: "HW1 Submission90207018")
Late submission policy: late submission carries a penalty of 10% off
your assignment per day after the deadline.
1 Problem 1 (8 points)
Provide the distribution that factorized over the graphical models of Fig. 1, then, sketch the
equivalent factor graph of that graphical models.
Figure 1 Graphical models of problem 1.
2
2 Problem 2 (8 points)
We have 3 random variables (X, Y, Z) which have the joint distribution p(x, y, z) with the
constraint that p(x, y, z) > 0 ∀x, y, z. prove the following statement:
X ⊥ (Y, Z), if X ⊥ Y |Z, and X ⊥ Z|Y.
(1)
If we permit p(x, y, z) = 0 for some (x, y, z), is the above statement still true? if yes, prove
the statement, otherwise give a counterexample.
3 Problem 3 (12 points)
We have 3 i.i.d binary random variables Xi (i = 1, 2, 3) which are drawn from a bernoulli
distribution with parameter p. Now, we define two random variables Z and Z ′ as
Z = X1 ⊗ X2 , Z ′ = X2 ⊗ X3
where ⊗ denotes the XOR operator.
Figure 2 Directed graphical model of problem 4.
(2)
Homework 1
3
3.1 calculate P (X2 , X3 |Z ′ ).
3.2 Draw the DAG of these 5 random variables. List all the independence relations
that this graph implies.
3.3 Draw the MRF of these 5 random variables. List all the independence relations
that this graph implies.
3.4 What is the condition on p in order to have Z ′ ⊥ X3 and Z ⊥ X1 ? Does the
graphs in 3.2 and 3.3 imply these marginal independence relations?
4 Problem 4 (10 points)
Verify the following conditional independence assertions based on the graph of Fig. 2.
4.1 X2 ⊥ X8 |(X3 , X4 , X5 )
4.2 X8 ⊥ X9 |(X3 , X4 , X5 )
4.3 X7 ⊥ X10 |(X3 , X4 , X5 )
4.4 Find the largest set A that satisfies X10 ⊥ A|(X2 , X9 )
4.5 Specify the Markov Blanket of X2
5 Problem 5 (12 points)
Consider the seven random variables Season(S), Dehydration(D), F lu(F ), Headache(H),
Chills(C), N ausea(N ) and Dizziness(Z) with the relations shown in figure 3 with the
conditional probability tables demonsterated in figure 4.
Figure 3 Directed graphical model of problem 5.
4
Figure 4 Conditional probability tables of problem 5.
5.1 given that it is not winter, what is the probability that you do not have the flu?
5.2 given that it is winter and that you have a headache, what is the probability
that you have the flu?
5.3 given you feel nauseous, what is the probability that you have a headache?
5.4 If you know you have a headache, does it increase or decrease your likelihood
of having a chill? Is it intuitively sensible?
6 Problem 6 (10 points)
Consider the four random variables B1 , B2 , B3 , B4 with the following conditional
independence relations:
B1 ⊥ B2 , ¬(B4 ⊥ B1 ), B1 ⊥ B4 |(B2 , B3 )
(3)
Design a pefect map which fullfils above (and only above) independence relations (construct
a bayes net graph).
Homework 1
5
Figure 5 The original binary image is shown on the left, and a noisy version of the image is shown
on the right.
7 Problem 7 (20 points)
Assume we are given a 2D image in which each pixel Xi is binary (Xi ∈ {+1, −1}). Now,
suppose we are also given a noisy version of that image in which each pixel (denoted by
Yi ) is flipped randomly with probability 0.1 (the original image and its noisy version is
demonstrated in figure 5).
In this problem, our goal is to recover the original image given the observed noisy image
based on the Markov network modeling which goes as follows: By denoting the latent
noise-free pixels by X̄ and the observed noisy pixels by Ȳ . We define the energy function
for this model based on the markov network model of figure 6 as
E(X̄ = x, Ȳ = y) = α
∑
i
xi − β
∑
i,j
xi xj − γ
∑
xi yi
(4)
i
where α > 0, β > 0, γ > 0 are the free parameters of the model that must be choosen
(choosing the parameters is up to you). Based on the images provided in hw1_images.mat,
your task is to estimate the true value of each pixel (+1 or -1) by optimizing the above
energy function. To do so, initialize the Xi ’s to their noisy values Yi , and then iterate through
each Xi and check whether setting it’s value to +1 or -1 yields a lower energy. Repeat this
process, making passes through all of the pixels, until the total energy of the model has
converged.
Provide the accuracy rate (fraction of pixels recovered correctly) that you obtain by
comparing your denoised image to the original image, for three different settings of the
parameters α, β, γ. You also must provide a figure of your best denoised image.
NOTE: Comment your code and make it easy to read if you want to get full credit!
8 Problem 8 (20 points)
Probabilistic graphical models (both directed and undirected models) have been successfully
used in wide range of applications (computer vision, Bioinformatics, speech recognition,
etc). Latent Dirichlet Allocation (LDA) is one of the most well-known graphical models
6
Figure 6 Illustration of the Markov network for image denoising.
which has largely been used in topic modeling applications. Read the paper (Sections 1
through 4) and prepare a report based on that paper that addresses the following questions
(you can download the paper from the link: "jmlr.org/papers/v3/blei03a.html"):
8.1 Write down clearly the problem that is addressed in the paper.
8.2 In this paper, four models (unigram, mixture of unigrams, pLSI, and LDA)
have been explained. Provide a brief summery of each model (e.g. description
of each corresponding PGM, input data, output data, number of parameters,
conditional dependence/independence relationships,...).
8.3 Compare these models and provide advantages and disadvantages of each of
these models.