Building Bayesian Networks
Inference by Variable Elimination
Reasoning with Bayesian Networks
Lecture 2: Building Bayesian Networks, Inference by Variable
Elimination
Jinbo Huang
NICTA and ANU
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Overview
I
Consider several real-world applications
I
Formulate each as formal query to Bayesian network
I
Types of queries
I
Construction of Bayesian network
I
Inference (query answering) by variable elimination
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Types of Queries
Example I: Medical Diagnosis
Example II: Pregnancy Testing
Sensitivity Analysis
Network Granularity
Example Bayesian Network: Asia
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Types of Queries
Example I: Medical Diagnosis
Example II: Pregnancy Testing
Sensitivity Analysis
Network Granularity
Probability of Evidence
I
Pr(X = yes, D = no) ≈ 3.96%: positive X-ray, no dyspnoea
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Types of Queries
Example I: Medical Diagnosis
Example II: Pregnancy Testing
Sensitivity Analysis
Network Granularity
Probability of Evidence
I
Pr(X = yes, D = no) ≈ 3.96%: positive X-ray, no dyspnoea
I
E = {X , D}: evidence variables
I
e = {X = yes, D = no}: evidence
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Types of Queries
Example I: Medical Diagnosis
Example II: Pregnancy Testing
Sensitivity Analysis
Network Granularity
Probability of Evidence
I
Pr(X = yes, D = no) ≈ 3.96%: positive X-ray, no dyspnoea
I
E = {X , D}: evidence variables
I
e = {X = yes, D = no}: evidence
I
Evidence can also be arbitray propositional sentence:
X = yes ∨ D = yes
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Types of Queries
Example I: Medical Diagnosis
Example II: Pregnancy Testing
Sensitivity Analysis
Network Granularity
Probability of Evidence
I
Pr(X = yes, D = no) ≈ 3.96%: positive X-ray, no dyspnoea
I
E = {X , D}: evidence variables
I
e = {X = yes, D = no}: evidence
I
Evidence can also be arbitray propositional sentence:
X = yes ∨ D = yes
I
What if tool only supports evidence as variable instantiation?
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Types of Queries
Example I: Medical Diagnosis
Example II: Pregnancy Testing
Sensitivity Analysis
Network Granularity
Probability of Evidence: Auxiliary Nodes
I
Pr(X = yes ∨ D = yes)?
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Types of Queries
Example I: Medical Diagnosis
Example II: Pregnancy Testing
Sensitivity Analysis
Network Granularity
Probability of Evidence: Auxiliary Nodes
I
Pr(X = yes ∨ D = yes)?
I
Add auxiliary node E as child of X & D
X
yes
yes
no
no
D
yes
no
yes
no
E
yes
yes
yes
yes
Pr(e|x, d)
1
1
1
0
I
Query Pr(E = yes) instead
I
CPT is deterministic, can be represented more compactly
Jinbo Huang
Reasoning with Bayesian Networks
Types of Queries
Example I: Medical Diagnosis
Example II: Pregnancy Testing
Sensitivity Analysis
Network Granularity
Building Bayesian Networks
Inference by Variable Elimination
Prior Marginals
X
Pr(x1 , . . . , xm ) =
Pr(x1 , . . . , xn )
xm+1 ,...,xn
I
Projection of joint distribution on subset of variables
I
Common special case: m = 1
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Types of Queries
Example I: Medical Diagnosis
Example II: Pregnancy Testing
Sensitivity Analysis
Network Granularity
Prior Marginals: Asia
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Types of Queries
Example I: Medical Diagnosis
Example II: Pregnancy Testing
Sensitivity Analysis
Network Granularity
Posterior Marginals
X
Pr(x1 , . . . , xm |e) =
Pr(x1 , . . . , xn |e)
xm+1 ,...,xn
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Types of Queries
Example I: Medical Diagnosis
Example II: Pregnancy Testing
Sensitivity Analysis
Network Granularity
Posterior Marginals: Asia
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Types of Queries
Example I: Medical Diagnosis
Example II: Pregnancy Testing
Sensitivity Analysis
Network Granularity
Posterior Marginals: Soft Evidence
def
Pr(β)
Pr(¬β)
I
O(β) =
I
Declare evidence on β by Bayes factor k =
I
“My neighbor’s call increases odds of Alarm by factor of 4”
I
How to formulate belief update as posterior marginals?
Jinbo Huang
O 0 (β)
O(β)
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Types of Queries
Example I: Medical Diagnosis
Example II: Pregnancy Testing
Sensitivity Analysis
Network Granularity
Posterior Marginals: Soft Evidence
def
Pr(β)
Pr(¬β)
I
O(β) =
I
Declare evidence on β by Bayes factor k =
I
“My neighbor’s call increases odds of Alarm by factor of 4”
I
How to formulate belief update as posterior marginals?
I
Emulate soft evidence on event β with new node S
I
S has two states, and two parameters
I
fp = Pr(S|¬β), fn = Pr(¬S|β)
Jinbo Huang
O 0 (β)
O(β)
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Types of Queries
Example I: Medical Diagnosis
Example II: Pregnancy Testing
Sensitivity Analysis
Network Granularity
Soft Evidence as Noisy Sensor
O 0 (β) =
=
Pr0 (β)
Pr(S|β) Pr(β)
Pr(β|S)
=
=
0
Pr(¬β|S)
Pr(S|¬β) Pr(¬β)
Pr (¬β)
1 − fn Pr(β)
1 − fn
=
O(β)
fp Pr(¬β)
fp
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Types of Queries
Example I: Medical Diagnosis
Example II: Pregnancy Testing
Sensitivity Analysis
Network Granularity
Soft Evidence as Noisy Sensor
O 0 (β) =
=
I
Pr0 (β)
Pr(S|β) Pr(β)
Pr(β|S)
=
=
0
Pr(¬β|S)
Pr(S|¬β) Pr(¬β)
Pr (¬β)
1 − fn Pr(β)
1 − fn
=
O(β)
fp Pr(¬β)
fp
Soft evidence with Bayes factor k + emulated by positive
+
n
reading of noisy sensor S with 1−f
fp = k
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Types of Queries
Example I: Medical Diagnosis
Example II: Pregnancy Testing
Sensitivity Analysis
Network Granularity
Soft Evidence as Noisy Sensor
O 0 (β) =
=
Pr0 (β)
Pr(S|β) Pr(β)
Pr(β|S)
=
=
0
Pr(¬β|S)
Pr(S|¬β) Pr(¬β)
Pr (¬β)
1 − fn Pr(β)
1 − fn
=
O(β)
fp Pr(¬β)
fp
I
Soft evidence with Bayes factor k + emulated by positive
+
n
reading of noisy sensor S with 1−f
fp = k
I
Similarly, negative reading emulates Bayes factor k − =
Jinbo Huang
Reasoning with Bayesian Networks
fn
1−fp
Building Bayesian Networks
Inference by Variable Elimination
Types of Queries
Example I: Medical Diagnosis
Example II: Pregnancy Testing
Sensitivity Analysis
Network Granularity
Soft Evidence as Noisy Sensor
O 0 (β) =
=
Pr0 (β)
Pr(S|β) Pr(β)
Pr(β|S)
=
=
0
Pr(¬β|S)
Pr(S|¬β) Pr(¬β)
Pr (¬β)
1 − fn Pr(β)
1 − fn
=
O(β)
fp Pr(¬β)
fp
I
Soft evidence with Bayes factor k + emulated by positive
+
n
reading of noisy sensor S with 1−f
fp = k
I
Similarly, negative reading emulates Bayes factor k − =
I
fp + fn < 1 ⇒ k + > 1 and k − < 1
Jinbo Huang
Reasoning with Bayesian Networks
fn
1−fp
Building Bayesian Networks
Inference by Variable Elimination
Types of Queries
Example I: Medical Diagnosis
Example II: Pregnancy Testing
Sensitivity Analysis
Network Granularity
Posterior Marginals: Soft Evidence Example
Soft evidence that
doubles odds of
“positive X-ray or
dyspnoea”
First add auxiliary node
E
Then add virtual
evidence node V
Set CPT such that
1−fn
fp = 2
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Types of Queries
Example I: Medical Diagnosis
Example II: Pregnancy Testing
Sensitivity Analysis
Network Granularity
Posterior Marginals: Soft Evidence Example
Soft evidence that
doubles odds of
“positive X-ray or
dyspnoea”
First add auxiliary node
E
Then add virtual
evidence node V
Set CPT such that
1−fn
fp = 2
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Types of Queries
Example I: Medical Diagnosis
Example II: Pregnancy Testing
Sensitivity Analysis
Network Granularity
Most Probable Explanation (MPE)
Maximize
Pr(x1 , . . . , xn |e)
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Types of Queries
Example I: Medical Diagnosis
Example II: Pregnancy Testing
Sensitivity Analysis
Network Granularity
Most Probable Explanation (MPE)
Maximize
Pr(x1 , . . . , xn |e)
6= maximizing each
posterior marginal
Pr(xi |e)
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Types of Queries
Example I: Medical Diagnosis
Example II: Pregnancy Testing
Sensitivity Analysis
Network Granularity
Maximum a Posteriori Hypothesis (MAP)
Maximize Pr(m|e) for
subset of variables M
MPE is special case
where M are all
variables
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Types of Queries
Example I: Medical Diagnosis
Example II: Pregnancy Testing
Sensitivity Analysis
Network Granularity
Maximum a Posteriori Hypothesis (MAP)
Maximize Pr(m|e) for
subset of variables M
MPE is special case
where M are all
variables
Projection of MPE on
M approximates MAP
Exact if M & E
determines rest
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Types of Queries
Example I: Medical Diagnosis
Example II: Pregnancy Testing
Sensitivity Analysis
Network Granularity
Constructing Bayesian Networks
I
Define network variables & their domains
I
I
I
Define network structure (edges)
I
I
Query & evidence variables, usually from problem statement
Intermediary variables, harder to determine
What variables are direct causes of X ?
Define network CPTs
I
I
I
Determined objectively from problem statement
Reflection of subjective beliefs
Estimated from data
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Types of Queries
Example I: Medical Diagnosis
Example II: Pregnancy Testing
Sensitivity Analysis
Network Granularity
Medical Diagnosis
I
Flu characterized by fever and body aches, can be associated
with chilling and sore throat
I
Cold associcated with chilling, can cause sore throat
I
Tonsillitis (inflammation of tonsils) causes sore throat, can be
associated with fever
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Types of Queries
Example I: Medical Diagnosis
Example II: Pregnancy Testing
Sensitivity Analysis
Network Granularity
Medical Diagnosis
I
Flu characterized by fever and body aches, can be associated
with chilling and sore throat
I
Cold associcated with chilling, can cause sore throat
I
Tonsillitis (inflammation of tonsils) causes sore throat, can be
associated with fever
I
Capture this knowledge with Bayesian network, diagnose
condition given symptoms
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Types of Queries
Example I: Medical Diagnosis
Example II: Pregnancy Testing
Sensitivity Analysis
Network Granularity
Medical Diagnosis
I
Flu characterized by fever and body aches, can be associated
with chilling and sore throat
I
Cold associcated with chilling, can cause sore throat
I
Tonsillitis (inflammation of tonsils) causes sore throat, can be
associated with fever
I
Capture this knowledge with Bayesian network, diagnose
condition given symptoms
I
Query, evidence, intermediary variables?
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Types of Queries
Example I: Medical Diagnosis
Example II: Pregnancy Testing
Sensitivity Analysis
Network Granularity
Medical Diagnosis
I
Flu characterized by fever and body aches, can be associated
with chilling and sore throat
I
Cold associcated with chilling, can cause sore throat
I
Tonsillitis (inflammation of tonsils) causes sore throat, can be
associated with fever
I
Capture this knowledge with Bayesian network, diagnose
condition given symptoms
I
Query, evidence, intermediary variables?
I
Flu, cold, tonsillitis;
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Types of Queries
Example I: Medical Diagnosis
Example II: Pregnancy Testing
Sensitivity Analysis
Network Granularity
Medical Diagnosis
I
Flu characterized by fever and body aches, can be associated
with chilling and sore throat
I
Cold associcated with chilling, can cause sore throat
I
Tonsillitis (inflammation of tonsils) causes sore throat, can be
associated with fever
I
Capture this knowledge with Bayesian network, diagnose
condition given symptoms
I
Query, evidence, intermediary variables?
I
Flu, cold, tonsillitis; chilling, body ache, sore throat, fever
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Types of Queries
Example I: Medical Diagnosis
Example II: Pregnancy Testing
Sensitivity Analysis
Network Granularity
Medical Diagnosis: Network Structure
Cold Flu
Chilling Body Ache
Jinbo Huang
Tonsillitis
Sore Throat Fever
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Types of Queries
Example I: Medical Diagnosis
Example II: Pregnancy Testing
Sensitivity Analysis
Network Granularity
Medical Diagnosis: Network Structure
Cold
Flu
Tonsillitis
QQ
Q
Q
Q
Q
Q
Q
Q
Q
Q
Q
Q
Q
s
Q
Chilling Body Ache
Jinbo Huang
Sore Throat Fever
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Types of Queries
Example I: Medical Diagnosis
Example II: Pregnancy Testing
Sensitivity Analysis
Network Granularity
Medical Diagnosis: Network Structure
Cold
Flu
Tonsillitis
Q
QQ
S
Q
Q
S Q
Q
S QQ
Q
Q S
Q
Q
Q
S
Q
Q
Q
Q
S
Q
Q
S
Q
Q
Q S
Q
Q
Q
QS
Q
sS
s
Q
w
Q
Chilling Body Ache
Jinbo Huang
Sore Throat Fever
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Types of Queries
Example I: Medical Diagnosis
Example II: Pregnancy Testing
Sensitivity Analysis
Network Granularity
Medical Diagnosis: Network Structure
Cold
Flu
Tonsillitis
Q
QQ
J
S
Q
J
Q
S Q
Q
J
S QQ
Q
Q J
S
Q
Q
Q
S
J
Q
Q Q
Q
J
S
Q
QQ
J
S
Q
Q S
Q
J
Q S Q
Q
Q J
sS
s J
Q
w
Q
^
Chilling Body Ache
Jinbo Huang
Sore Throat Fever
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Types of Queries
Example I: Medical Diagnosis
Example II: Pregnancy Testing
Sensitivity Analysis
Network Granularity
Medical Diagnosis: Naive Bayes
Condition: normal, cold,
flu, tonsillitis
Condition
Q
S
Q
S Q
S QQ
S
Q
Q
S
Q
Q
S
Q
S
Q
Q
S
Q
S
Q
s
Q
w
S
Chilling Body Ache
Cold
Flu
Tonsillitis
Q
QQ
S
J
Q
Q
S Q
J
Q
Q
S
J
Q
Q
Q S
Q
J
Q
Q
S
J
Q
Q Q
Q
S
J
Q
Q
S
J
Q
Q
Q S
Q
J
Q S Q
Q
Q J
sS
s J
Q
w
Q
^
Chilling Body Ache
Sore Throat Fever
Sore Throat Fever
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Types of Queries
Example I: Medical Diagnosis
Example II: Pregnancy Testing
Sensitivity Analysis
Network Granularity
Medical Diagnosis: Naive Bayes
Condition: normal, cold,
flu, tonsillitis
Condition
Q
S
Q
S Q
S QQ
S
Q
Q
S
Q
Q
S
Q
S
Q
Q
S
Q
S
Q
s
Q
w
S
Chilling Body Ache
Cold
Flu
Model inaccuracy: single
fault assumption
Tonsillitis
Q
QQ
S
J
Q
Q
S Q
J
Q
Q
S
J
Q
Q
Q S
Q
J
Q
Q
S
J
Q
Q Q
Q
S
J
Q
Q
S
J
Q
Q
Q S
Q
J
Q S Q
Q
Q J
sS
s J
Q
w
Q
^
Chilling Body Ache
Sore Throat Fever
Sore Throat Fever
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Types of Queries
Example I: Medical Diagnosis
Example II: Pregnancy Testing
Sensitivity Analysis
Network Granularity
Medical Diagnosis: Naive Bayes
Condition: normal, cold,
flu, tonsillitis
Condition
Q
S
Q
S Q
S QQ
S
Q
Q
S
Q
Q
S
Q
S
Q
Q
S
Q
S
Q
s
Q
w
S
Chilling Body Ache
Cold
Flu
Tonsillitis
Q
QQ
S
J
Q
Q
S Q
J
Q
Q
S
J
Q
Q
Q S
Q
J
Q
Q
S
J
Q
Q Q
Q
S
J
Q
Q
S
J
Q
Q
Q S
Q
J
Q S Q
Q
Q J
sS
s J
Q
w
Q
^
Chilling Body Ache
Sore Throat Fever
Sore Throat Fever
Jinbo Huang
Model inaccuracy: single
fault assumption
I
Cold ⇒ Fever & Sore
Throat independent
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Types of Queries
Example I: Medical Diagnosis
Example II: Pregnancy Testing
Sensitivity Analysis
Network Granularity
Medical Diagnosis: Naive Bayes
Condition: normal, cold,
flu, tonsillitis
Condition
Q
S
Q
S Q
S QQ
S
Q
Q
S
Q
Q
S
Q
S
Q
Q
S
Q
S
Q
s
Q
w
S
Chilling Body Ache
Cold
Flu
Model inaccuracy: single
fault assumption
I
Cold ⇒ Fever & Sore
Throat independent
I
Body Ache ⇒ Flu ↑
⇒ Cold & Tonsillitis ↓
Tonsillitis
Q
QQ
S
J
Q
Q
S Q
J
Q
Q
S
J
Q
Q
Q S
Q
J
Q
Q
S
J
Q
Q Q
Q
S
J
Q
Q
S
J
Q
Q
Q S
Q
J
Q S Q
Q
Q J
sS
s J
Q
w
Q
^
Chilling Body Ache
Sore Throat Fever
Sore Throat Fever
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Types of Queries
Example I: Medical Diagnosis
Example II: Pregnancy Testing
Sensitivity Analysis
Network Granularity
Medical Diagnosis: Naive Bayes
Condition: normal, cold,
flu, tonsillitis
Condition
Q
S
Q
S Q
S QQ
S
Q
Q
S
Q
Q
S
Q
S
Q
Q
S
Q
S
Q
s
Q
w
S
Chilling Body Ache
Cold
Flu
Model inaccuracy: single
fault assumption
I
Cold ⇒ Fever & Sore
Throat independent
I
Body Ache ⇒ Flu ↑
⇒ Cold & Tonsillitis ↓
I
No Fever ⇒ Cold ↑
Tonsillitis
Q
QQ
S
J
Q
Q
S Q
J
Q
Q
S
J
Q
Q
Q S
Q
J
Q
Q
S
J
Q
Q Q
Q
S
J
Q
Q
S
J
Q
Q
Q S
Q
J
Q S Q
Q
Q J
sS
s J
Q
w
Q
^
Chilling Body Ache
Sore Throat Fever
Sore Throat Fever
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Types of Queries
Example I: Medical Diagnosis
Example II: Pregnancy Testing
Sensitivity Analysis
Network Granularity
Medical Diagnosis: CPTs
From medical experts, based on known stats or subjective beliefs
I
CPTs for conditions (roots of graph): Pr(condition) without
knowledge of symptoms
I
CPTs for symptoms: Pr(Chilling |Cold, Flu) etc
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Types of Queries
Example I: Medical Diagnosis
Example II: Pregnancy Testing
Sensitivity Analysis
Network Granularity
Medical Diagnosis: CPTs
From medical experts, based on known stats or subjective beliefs
I
CPTs for conditions (roots of graph): Pr(condition) without
knowledge of symptoms
I
CPTs for symptoms: Pr(Chilling |Cold, Flu) etc
Estimated directly from medical records
Case
1
2
3
...
Cold
T
F
?
...
Maximize
Flu
F
T
?
Tonsillitis
?
F
T
N
Y
Chilling
T
T
F
Body Ache
F
T
?
Sore Throat
F
F
T
Fever
F
T
F
Pr(di )
i=1
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Types of Queries
Example I: Medical Diagnosis
Example II: Pregnancy Testing
Sensitivity Analysis
Network Granularity
Medical Diagnosis: Queries
Given symptoms (chilling, body ache, sore throat, fever), pose
MAP query on Cold, Flu, & Tonsillitis
Reduces to MPE if evidence covers all four symptoms
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Types of Queries
Example I: Medical Diagnosis
Example II: Pregnancy Testing
Sensitivity Analysis
Network Granularity
Pregnancy Testing
Three tests to confirm pregnancy of a cow after insemination
I
Scanning test: false+ 1%, false− 10%
I
Blood test to detect progesterone: false+ 10%, false− 30%
I
Urine test to detect progesterone: false+ 10%, false− 20%
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Types of Queries
Example I: Medical Diagnosis
Example II: Pregnancy Testing
Sensitivity Analysis
Network Granularity
Pregnancy Testing
Three tests to confirm pregnancy of a cow after insemination
I
Scanning test: false+ 1%, false− 10%
I
Blood test to detect progesterone: false+ 10%, false− 30%
I
Urine test to detect progesterone: false+ 10%, false− 20%
I
Pr(progesterone|pregnant) = 90%,
Pr(progesterone|¬pregnant) = 1%
I
Pr(pregrant) = 87% after insemination
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Types of Queries
Example I: Medical Diagnosis
Example II: Pregnancy Testing
Sensitivity Analysis
Network Granularity
Pregnancy Testing
Three tests to confirm pregnancy of a cow after insemination
I
Scanning test: false+ 1%, false− 10%
I
Blood test to detect progesterone: false+ 10%, false− 30%
I
Urine test to detect progesterone: false+ 10%, false− 20%
I
Pr(progesterone|pregnant) = 90%,
Pr(progesterone|¬pregnant) = 1%
I
Pr(pregrant) = 87% after insemination
I
Query, evidence, intermediary variables?
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Types of Queries
Example I: Medical Diagnosis
Example II: Pregnancy Testing
Sensitivity Analysis
Network Granularity
Pregnancy Testing
Three tests to confirm pregnancy of a cow after insemination
I
Scanning test: false+ 1%, false− 10%
I
Blood test to detect progesterone: false+ 10%, false− 30%
I
Urine test to detect progesterone: false+ 10%, false− 20%
I
Pr(progesterone|pregnant) = 90%,
Pr(progesterone|¬pregnant) = 1%
I
Pr(pregrant) = 87% after insemination
I
Query, evidence, intermediary variables?
I
Pregnant;
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Types of Queries
Example I: Medical Diagnosis
Example II: Pregnancy Testing
Sensitivity Analysis
Network Granularity
Pregnancy Testing
Three tests to confirm pregnancy of a cow after insemination
I
Scanning test: false+ 1%, false− 10%
I
Blood test to detect progesterone: false+ 10%, false− 30%
I
Urine test to detect progesterone: false+ 10%, false− 20%
I
Pr(progesterone|pregnant) = 90%,
Pr(progesterone|¬pregnant) = 1%
I
Pr(pregrant) = 87% after insemination
I
Query, evidence, intermediary variables?
I
Pregnant; Scanning Test, Blood Test, Urine Test;
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Types of Queries
Example I: Medical Diagnosis
Example II: Pregnancy Testing
Sensitivity Analysis
Network Granularity
Pregnancy Testing
Three tests to confirm pregnancy of a cow after insemination
I
Scanning test: false+ 1%, false− 10%
I
Blood test to detect progesterone: false+ 10%, false− 30%
I
Urine test to detect progesterone: false+ 10%, false− 20%
I
Pr(progesterone|pregnant) = 90%,
Pr(progesterone|¬pregnant) = 1%
I
Pr(pregrant) = 87% after insemination
I
Query, evidence, intermediary variables?
I
Pregnant; Scanning Test, Blood Test, Urine Test;
Progesterone
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Types of Queries
Example I: Medical Diagnosis
Example II: Pregnancy Testing
Sensitivity Analysis
Network Granularity
Pregnancy Testing: Structure and CPTs
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Types of Queries
Example I: Medical Diagnosis
Example II: Pregnancy Testing
Sensitivity Analysis
Network Granularity
Pregnancy Testing: Structure and CPTs
Independencies
Jinbo Huang
I
B & U ind. | L
I
S ind. of B & U | P
I
B & U not ind. | P
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Types of Queries
Example I: Medical Diagnosis
Example II: Pregnancy Testing
Sensitivity Analysis
Network Granularity
Pregnancy Testing: Structure and CPTs
Independencies
I
B & U ind. | L
I
S ind. of B & U | P
I
B & U not ind. | P
Pr(P|S = B = U = −ve)
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Types of Queries
Example I: Medical Diagnosis
Example II: Pregnancy Testing
Sensitivity Analysis
Network Granularity
Pregnancy Testing: Structure and CPTs
Independencies
I
B & U ind. | L
I
S ind. of B & U | P
I
B & U not ind. | P
Pr(P|S = B = U = −ve)
= 10.21%
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Types of Queries
Example I: Medical Diagnosis
Example II: Pregnancy Testing
Sensitivity Analysis
Network Granularity
Sensitivity Analysis
I
Farmer unhappy about result, would like three negative tests
to drop Pr(P|e) to ≤ 5%
I
Willing to replace test kits, need to know required false+ and
false− rates
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Types of Queries
Example I: Medical Diagnosis
Example II: Pregnancy Testing
Sensitivity Analysis
Network Granularity
Sensitivity Analysis
I
Farmer unhappy about result, would like three negative tests
to drop Pr(P|e) to ≤ 5%
I
Willing to replace test kits, need to know required false+ and
false− rates
I
Sentitivity of query result to parameter change
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Types of Queries
Example I: Medical Diagnosis
Example II: Pregnancy Testing
Sensitivity Analysis
Network Granularity
Sensitivity Analysis
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Types of Queries
Example I: Medical Diagnosis
Example II: Pregnancy Testing
Sensitivity Analysis
Network Granularity
Sensitivity Analysis
Improved B or U
test cannot help
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Types of Queries
Example I: Medical Diagnosis
Example II: Pregnancy Testing
Sensitivity Analysis
Network Granularity
Sensitivity Analysis
Improved B or U
test cannot help
Can if we’re less
ambitious (e.g.,
≤ 8%)
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Types of Queries
Example I: Medical Diagnosis
Example II: Pregnancy Testing
Sensitivity Analysis
Network Granularity
Network Granularity
I
How fine-grained should the network be?
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Types of Queries
Example I: Medical Diagnosis
Example II: Pregnancy Testing
Sensitivity Analysis
Network Granularity
Network Granularity
I
How fine-grained should the network be?
I
Progesterone: neither query nor evidence variable
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Types of Queries
Example I: Medical Diagnosis
Example II: Pregnancy Testing
Sensitivity Analysis
Network Granularity
Network Granularity
I
How fine-grained should the network be?
I
Progesterone: neither query nor evidence variable
I
Why include it?
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Types of Queries
Example I: Medical Diagnosis
Example II: Pregnancy Testing
Sensitivity Analysis
Network Granularity
Network Granularity
I
How fine-grained should the network be?
I
Progesterone: neither query nor evidence variable
I
Why include it?
I
False+ & false− rates provided only based on progesterone
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Types of Queries
Example I: Medical Diagnosis
Example II: Pregnancy Testing
Sensitivity Analysis
Network Granularity
Network Granularity
I
How fine-grained should the network be?
I
Progesterone: neither query nor evidence variable
I
Why include it?
I
False+ & false− rates provided only based on progesterone
I
But we can compute rates based on pregnancy
Pr(B = −ve|P) = 36%, Pr(B = +ve|¬P) = 10.6%
Pr(U = −ve|P) = 27%, Pr(U = +ve|¬P) = 10.7%
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Types of Queries
Example I: Medical Diagnosis
Example II: Pregnancy Testing
Sensitivity Analysis
Network Granularity
Network Granularity
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Types of Queries
Example I: Medical Diagnosis
Example II: Pregnancy Testing
Sensitivity Analysis
Network Granularity
Network Granularity
B & U ind. | P
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Types of Queries
Example I: Medical Diagnosis
Example II: Pregnancy Testing
Sensitivity Analysis
Network Granularity
Network Granularity
B & U ind. | P
− B & U count
more against P
(45.09% v. 52.96%)
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Types of Queries
Example I: Medical Diagnosis
Example II: Pregnancy Testing
Sensitivity Analysis
Network Granularity
Network Granularity
B & U ind. | P
− B & U count
more against P
(45.09% v. 52.96%)
+ B & U count
more toward P
(99.61% v. 99.54%)
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Types of Queries
Example I: Medical Diagnosis
Example II: Pregnancy Testing
Sensitivity Analysis
Network Granularity
Network Granularity
B & U ind. | P
− B & U count
more against P
(45.09% v. 52.96%)
+ B & U count
more toward P
(99.61% v. 99.54%)
Intermediary
variables cannot
always be bypassed
Jinbo Huang
Reasoning with Bayesian Networks
Types of Queries
Example I: Medical Diagnosis
Example II: Pregnancy Testing
Sensitivity Analysis
Network Granularity
Building Bayesian Networks
Inference by Variable Elimination
Network Granularity: Bypassing Intermediary Variable
Bypassing X with single child Y
X
θy0 |uv =
θy |xv θx|u
x
Does not affect model accuracy: Pr(q, e) won’t change
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Types of Queries
Example I: Medical Diagnosis
Example II: Pregnancy Testing
Sensitivity Analysis
Network Granularity
Building Bayesian Networks: Summary
I
Query types: Pr(e), Pr(q|e), Pr(q|soft evidence), MPE, MAP
I
Determining query, evidence, intermediary variables
I
Determining network structure and CPTs
I
Case studies: medical diagnosis, pregnancy testing
I
Additional issues: sensitivity analysis, bypassing variables
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Elimination and Factors
Prior Marginals by Elimination
Elimination Orders
Posterior Marginals and Probability of Evidence
Complexity Related to Network and Query Structure
Inference by Variable Elimination
I
Elimination and factors
I
Prior marginals by elimination
I
Elimination orders
I
Posterior marginals and probability of evidence
I
Complexity related to network and query structure
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Elimination and Factors
Prior Marginals by Elimination
Elimination Orders
Posterior Marginals and Probability of Evidence
Complexity Related to Network and Query Structure
Elimination
Pr(D, E )?
D
true
true
false
false
Jinbo Huang
E
true
false
true
false
Pr(D, E )
.30443
.39507
.05957
.24093
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Elimination and Factors
Prior Marginals by Elimination
Elimination Orders
Posterior Marginals and Probability of Evidence
Complexity Related to Network and Query Structure
Elimination
Pr(D, E )?
D
true
true
false
false
E
true
false
true
false
Pr(D, E )
.30443
.39507
.05957
.24093
Sum out variables A, B, C from
network
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Elimination and Factors
Prior Marginals by Elimination
Elimination Orders
Posterior Marginals and Probability of Evidence
Complexity Related to Network and Query Structure
Elimination
Summing out variables A
A
true
false
B
true
true
C
true
true
D
true
true
E
true
true
Pr(.)
0.06384
0.01995
B
true
C
true
D
true
E
true
Pr(.)
0.08379=0.06384+0.01995
Do it for all instantiations of B, C , D, E
Repeat to eliminate B, C
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Elimination and Factors
Prior Marginals by Elimination
Elimination Orders
Posterior Marginals and Probability of Evidence
Complexity Related to Network and Query Structure
Elimination
Summing out variables A
A
true
false
B
true
true
C
true
true
D
true
true
E
true
true
Pr(.)
0.06384
0.01995
B
true
C
true
D
true
E
true
Pr(.)
0.08379=0.06384+0.01995
Do it for all instantiations of B, C , D, E
Repeat to eliminate B, C
Exponential in number of variables
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Elimination and Factors
Prior Marginals by Elimination
Elimination Orders
Posterior Marginals and Probability of Evidence
Complexity Related to Network and Query Structure
Elimination
Summing out variables A
A
true
false
B
true
true
C
true
true
D
true
true
E
true
true
Pr(.)
0.06384
0.01995
B
true
C
true
D
true
E
true
Pr(.)
0.08379=0.06384+0.01995
Do it for all instantiations of B, C , D, E
Repeat to eliminate B, C
Exponential in number of variables
Solution: Elimination in factored form
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Elimination and Factors
Prior Marginals by Elimination
Elimination Orders
Posterior Marginals and Probability of Evidence
Complexity Related to Network and Query Structure
Factors
I
f (x1 , . . . , xn ): function from instantiation to number
I
Can be joint or conditional probability
I
Trivial factor: n = 0
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Elimination and Factors
Prior Marginals by Elimination
Elimination Orders
Posterior Marginals and Probability of Evidence
Complexity Related to Network and Query Structure
Factors: Summing Out
I
Summing out Z ∈ X from f (X), where Y = X\{Z }
X
X
def
(
f )(y) =
f (z, y)
z
Z
I
Commutative
XX
Z
W
f =
XX
W
f
Z
I
P
Summing out multiple variables Z f : marginalizing
variables Z, projecting f on variables Y (other variables)
I
Complexity O(exp(w )), where w = |X|
Jinbo Huang
Reasoning with Bayesian Networks
Elimination and Factors
Prior Marginals by Elimination
Elimination Orders
Posterior Marginals and Probability of Evidence
Complexity Related to Network and Query Structure
Building Bayesian Networks
Inference by Variable Elimination
Factors: Multiplication
I
Multiplying f1 (X) and f2 (Y)
def
(f1 f2 )(z) = f1 (x)f2 (y)
where Z = X ∪ Y, x ∼ z, y ∼ z
I
Commutative and associative
I
Complexity O(m exp(w )) for m factors, where w = |Z|
Jinbo Huang
Reasoning with Bayesian Networks
Elimination and Factors
Prior Marginals by Elimination
Elimination Orders
Posterior Marginals and Probability of Evidence
Complexity Related to Network and Query Structure
Building Bayesian Networks
Inference by Variable Elimination
Prior Marginals by Elimination
Joint probability by chain rule
Pr(a, b, c, d, e) = θe|c θd|bc θc|a θb|a θa
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Elimination and Factors
Prior Marginals by Elimination
Elimination Orders
Posterior Marginals and Probability of Evidence
Complexity Related to Network and Query Structure
Prior Marginals by Elimination
Joint probability by chain rule
Pr(a, b, c, d, e) = θe|c θd|bc θc|a θb|a θa
Q
Joint probability as
of factors
ΘE |C ΘD|BC ΘD|A ΘB|A ΘA
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Elimination and Factors
Prior Marginals by Elimination
Elimination Orders
Posterior Marginals and Probability of Evidence
Complexity Related to Network and Query Structure
Prior Marginals by Elimination
Joint probability by chain rule
Pr(a, b, c, d, e) = θe|c θd|bc θc|a θb|a θa
Q
Joint probability as
of factors
ΘE |C ΘD|BC ΘD|A ΘB|A ΘA
Marginals
Pr(D, E ) =
X
ΘE |C ΘD|BC ΘD|A ΘB|A ΘA
A,B,C
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Elimination and Factors
Prior Marginals by Elimination
Elimination Orders
Posterior Marginals and Probability of Evidence
Complexity Related to Network and Query Structure
Prior Marginals by Elimination
Joint probability by chain rule
Pr(a, b, c, d, e) = θe|c θd|bc θc|a θb|a θa
Q
Joint probability as
of factors
ΘE |C ΘD|BC ΘD|A ΘB|A ΘA
Marginals
Pr(D, E ) =
X
ΘE |C ΘD|BC ΘD|A ΘB|A ΘA
A,B,C
Complexity still exponential in # of variables
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Elimination and Factors
Prior Marginals by Elimination
Elimination Orders
Posterior Marginals and Probability of Evidence
Complexity Related to Network and Query Structure
Prior Marginals by Elimination: Early Summation
I
I
Don’t multiply all factors before summation
Theorem: If X does not appear in f1 , then
X
X
f1 f2 = f1
f2
X
Jinbo Huang
X
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Elimination and Factors
Prior Marginals by Elimination
Elimination Orders
Posterior Marginals and Probability of Evidence
Complexity Related to Network and Query Structure
Prior Marginals by Elimination: Early Summation
I
I
Don’t multiply all factors before summation
Theorem: If X does not appear in f1 , then
X
X
f1 f2 = f1
f2
X
I
X
For example, if X appears only in fn , then
X
X
f1 . . . fn = f1 . . . fn−1
fn
X
X
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Elimination and Factors
Prior Marginals by Elimination
Elimination Orders
Posterior Marginals and Probability of Evidence
Complexity Related to Network and Query Structure
Prior Marginals by Elimination: Early Summation
I
I
Don’t multiply all factors before summation
Theorem: If X does not appear in f1 , then
X
X
f1 f2 = f1
f2
X
I
X
For example, if X appears only in fn , then
X
X
f1 . . . fn = f1 . . . fn−1
fn
X
I
X
Similarly, if X appears only in fn−1 and fn , then
X
X
f1 . . . fn = f1 . . . fn−2
fn−1 fn
X
X
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Elimination and Factors
Prior Marginals by Elimination
Elimination Orders
Posterior Marginals and Probability of Evidence
Complexity Related to Network and Query Structure
Prior Marginals by Elimination: Early Summation
I
Multiply all factors that include X , sum out X from result
I
Early summation reduces factor size, hence complexity of
Jinbo Huang
Reasoning with Bayesian Networks
Q
Building Bayesian Networks
Inference by Variable Elimination
Elimination and Factors
Prior Marginals by Elimination
Elimination Orders
Posterior Marginals and Probability of Evidence
Complexity Related to Network and Query Structure
Prior Marginals by Elimination: Early Summation
Compute Pr(C ): eliminate
A, then B
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Elimination and Factors
Prior Marginals by Elimination
Elimination Orders
Posterior Marginals and Probability of Evidence
Complexity Related to Network and Query Structure
Prior Marginals by Elimination: Early Summation
Compute Pr(C ): eliminate
A, then B
Two factors mention A:
ΘA , ΘB|A
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Elimination and Factors
Prior Marginals by Elimination
Elimination Orders
Posterior Marginals and Probability of Evidence
Complexity Related to Network and Query Structure
Prior Marginals by Elimination: Early Summation
Multiply ΘA and ΘB|A
A
true
true
false
false
Jinbo Huang
B
true
false
true
false
Reasoning with Bayesian Networks
ΘA ΘB|A
.54
.06
.08
.32
Building Bayesian Networks
Inference by Variable Elimination
Elimination and Factors
Prior Marginals by Elimination
Elimination Orders
Posterior Marginals and Probability of Evidence
Complexity Related to Network and Query Structure
Prior Marginals by Elimination: Early Summation
Multiply ΘA and ΘB|A
A
true
true
false
false
B
true
false
true
false
ΘA ΘB|A
.54
.06
.08
.32
Sum out A
B
true
false
Jinbo Huang
P
A ΘA ΘB|A
.62 = .54 + .08
.38 = .06 + .32
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Elimination and Factors
Prior Marginals by Elimination
Elimination Orders
Posterior Marginals and Probability of Evidence
Complexity Related to Network and Query Structure
Prior Marginals by Elimination: Early Summation
Two
P factors left, ΘC |B &
A ΘA ΘB|A , multiply
B
true
true
false
false
Jinbo Huang
C
true
false
true
false
Reasoning with Bayesian Networks
ΘC |B
.186
.434
.190
.190
P
A
ΘA ΘB|A
Building Bayesian Networks
Inference by Variable Elimination
Elimination and Factors
Prior Marginals by Elimination
Elimination Orders
Posterior Marginals and Probability of Evidence
Complexity Related to Network and Query Structure
Prior Marginals by Elimination: Early Summation
Two
P factors left, ΘC |B &
A ΘA ΘB|A , multiply
B
true
true
false
false
C
true
false
true
false
ΘC |B
.186
.434
.190
.190
P
A
ΘA ΘB|A
Sum out B
C
true
false
Jinbo Huang
P
B
ΘC |B
.376
.624
Reasoning with Bayesian Networks
P
A
ΘA ΘB|A
Building Bayesian Networks
Inference by Variable Elimination
Elimination and Factors
Prior Marginals by Elimination
Elimination Orders
Posterior Marginals and Probability of Evidence
Complexity Related to Network and Query Structure
Prior Marginals by Elimination: Early Summation
Two factors left, ΘC |B &
P
A ΘA ΘB|A , multiply
B
true
true
false
false
Biggest factor produced: 4 rows
ΘC |B
.186
.434
.190
.190
P
A
ΘA ΘB|A
Sum out B
C
true
false
Jinbo Huang
C
true
false
true
false
P
B
ΘC |B
.376
.624
Reasoning with Bayesian Networks
P
A
ΘA ΘB|A
Building Bayesian Networks
Inference by Variable Elimination
Elimination and Factors
Prior Marginals by Elimination
Elimination Orders
Posterior Marginals and Probability of Evidence
Complexity Related to Network and Query Structure
Prior Marginals by Elimination: Algorithm
Input: Bayesian network N , variables Q, order π on other variables
Output: prior marginal Pr(Q)
1:
2:
3:
4:
5:
6:
S ← CPTs of network N
for i = Q
1 to |π| do
f ← Pk fk , where fk ∈ S and mentions variable π(i)
fi ← π(i) f
remove
Q all fk from S, add fi
return f ∈S f
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Elimination and Factors
Prior Marginals by Elimination
Elimination Orders
Posterior Marginals and Probability of Evidence
Complexity Related to Network and Query Structure
Prior Marginals by Elimination: Algorithm
Input: Bayesian network N , variables Q, order π on other variables
Output: prior marginal Pr(Q)
1:
2:
3:
4:
5:
6:
S ← CPTs of network N
for i = Q
1 to |π| do
f ← Pk fk , where fk ∈ S and mentions variable π(i)
fi ← π(i) f
remove
Q all fk from S, add fi
return f ∈S f
Complexity (not counting line 6): O(n exp(w )), where n is # of
variables of largest fi , known as width of order π
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Elimination and Factors
Prior Marginals by Elimination
Elimination Orders
Posterior Marginals and Probability of Evidence
Complexity Related to Network and Query Structure
Prior Marginals by Elimination: Algorithm
Input: Bayesian network N , variables Q, order π on other variables
Output: prior marginal Pr(Q)
1:
2:
3:
4:
5:
6:
S ← CPTs of network N
for i = Q
1 to |π| do
f ← Pk fk , where fk ∈ S and mentions variable π(i)
fi ← π(i) f
remove
Q all fk from S, add fi
return f ∈S f
How do we find all fk on line 3 quickly (linear in # of such fk )?
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Elimination and Factors
Prior Marginals by Elimination
Elimination Orders
Posterior Marginals and Probability of Evidence
Complexity Related to Network and Query Structure
Prior Marginals by Elimination: Bucket Elimination
Bucket
E
B
C
D
A
Factors
ΘE |C
ΘB|A , ΘD|BC
ΘC |A
ΘA
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Elimination and Factors
Prior Marginals by Elimination
Elimination Orders
Posterior Marginals and Probability of Evidence
Complexity Related to Network and Query Structure
Prior Marginals by Elimination: Bucket Elimination
Bucket
E
B
C
D
A
Factors
ΘE |C
ΘB|A , ΘD|BC
ΘC |A
Bucket
E
B
C
D
A
ΘA
Jinbo Huang
Factors
ΘB|A , ΘD|BC
P
ΘC |A , E ΘE |C
ΘA
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Elimination and Factors
Prior Marginals by Elimination
Elimination Orders
Posterior Marginals and Probability of Evidence
Complexity Related to Network and Query Structure
Width of Elimination Order
I
Should prefer order with smaller width
I
How to compute width, without actually running elimination?
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Elimination and Factors
Prior Marginals by Elimination
Elimination Orders
Posterior Marginals and Probability of Evidence
Complexity Related to Network and Query Structure
Width of Elimination Order
I
Should prefer order with smaller width
I
How to compute width, without actually running elimination?
I
Only care about size of factors, run abstract version of
algorithm keeping track of factor sizes only
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Elimination and Factors
Prior Marginals by Elimination
Elimination Orders
Posterior Marginals and Probability of Evidence
Complexity Related to Network and Query Structure
Width of Elimination Order
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Elimination and Factors
Prior Marginals by Elimination
Elimination Orders
Posterior Marginals and Probability of Evidence
Complexity Related to Network and Query Structure
Computing Good Elimination Orders
I
Finding optimal order is NP-hard
I
Min-degree: eliminate variable with fewest neighbors
I
Min-fill: eliminate variable leading to fewest fill-in edges
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Elimination and Factors
Prior Marginals by Elimination
Elimination Orders
Posterior Marginals and Probability of Evidence
Complexity Related to Network and Query Structure
Posterior Marginals by Elimination
Pr(D, E |e)? e : A = true, B = false
D
true
true
false
false
Jinbo Huang
E
true
false
true
false
Pr(D, E |e)
.448
.192
.112
.248
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Elimination and Factors
Prior Marginals by Elimination
Elimination Orders
Posterior Marginals and Probability of Evidence
Complexity Related to Network and Query Structure
Posterior Marginals by Elimination
Pr(D, E |e)? e : A = true, B = false
D
true
true
false
false
E
true
false
true
false
Pr(D, E |e)
.448
.192
.112
.248
Compute joint marginals instead
D
true
true
false
false
Jinbo Huang
E
true
false
true
false
Pr(D, E , e)
.21504
.09216
.05376
.11904
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Elimination and Factors
Prior Marginals by Elimination
Elimination Orders
Posterior Marginals and Probability of Evidence
Complexity Related to Network and Query Structure
Posterior Marginals by Elimination
I
Zero out all rows of all factors inconsistent with e
I
Run elimination, result will be joint marginal Pr(Q, e)
I
Add all entries to obtain Pr(e)
I
Pr(Q|e) =
Pr(Q,e)
Pr(e)
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Elimination and Factors
Prior Marginals by Elimination
Elimination Orders
Posterior Marginals and Probability of Evidence
Complexity Related to Network and Query Structure
Posterior Marginals by Elimination
I
Zero out all rows of all factors inconsistent with e
I
Run elimination, result will be joint marginal Pr(Q, e)
I
Add all entries to obtain Pr(e)
I
Pr(Q|e) =
I
Run with Q = ∅ for Pr(e)
Pr(Q,e)
Pr(e)
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Elimination and Factors
Prior Marginals by Elimination
Elimination Orders
Posterior Marginals and Probability of Evidence
Complexity Related to Network and Query Structure
Network Structure and Complexity: Treewidth
I
Complexity of elimination exp. in width of elimination order
I
Treewidth is width of best elimination order for given network
I
Quantifies how close the network resembles a tree
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Elimination and Factors
Prior Marginals by Elimination
Elimination Orders
Posterior Marginals and Probability of Evidence
Complexity Related to Network and Query Structure
Network Structure and Complexity: Treewidth
1
2
Jinbo Huang
3
Reasoning with Bayesian Networks
3
Building Bayesian Networks
Inference by Variable Elimination
Elimination and Factors
Prior Marginals by Elimination
Elimination Orders
Posterior Marginals and Probability of Evidence
Complexity Related to Network and Query Structure
Network Structure and Complexity: Treewidth
I
Trees have treewidth 1
I
# of nodes has no genuine effect on treewidth
I
# of parents per node has effect
I
I
Treewidth ≥ max # of parents per node
Equality holds for polytrees, or singly-connected networks
I
Loops tend to increase treewidth
I
# of loops has no genuine effect
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Elimination and Factors
Prior Marginals by Elimination
Elimination Orders
Posterior Marginals and Probability of Evidence
Complexity Related to Network and Query Structure
Query Structure and Complexity: Network Pruning
I
Consider computation of Pr(Q, e) (includes prior marginals
and probability of evidence as special cases)
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Elimination and Factors
Prior Marginals by Elimination
Elimination Orders
Posterior Marginals and Probability of Evidence
Complexity Related to Network and Query Structure
Query Structure and Complexity: Network Pruning
I
Consider computation of Pr(Q, e) (includes prior marginals
and probability of evidence as special cases)
I
Pruning nodes: All leaves 6∈ Q ∪ E, interatively
I
I
Worst case: All leaves ∈ Q ∪ E, no pruning
Best case: All Q ∪ E are roots, every node 6∈ Q ∪ E pruned
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Elimination and Factors
Prior Marginals by Elimination
Elimination Orders
Posterior Marginals and Probability of Evidence
Complexity Related to Network and Query Structure
Query Structure and Complexity: Network Pruning
I
Consider computation of Pr(Q, e) (includes prior marginals
and probability of evidence as special cases)
I
Pruning nodes: All leaves 6∈ Q ∪ E, interatively
I
I
I
Worst case: All leaves ∈ Q ∪ E, no pruning
Best case: All Q ∪ E are roots, every node 6∈ Q ∪ E pruned
Pruning edges: For each edge U → X , U ∈ E
I
Remove edge, shrink CPT ΘX |U by removing rows inconsistent
with e and removing column U
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Elimination and Factors
Prior Marginals by Elimination
Elimination Orders
Posterior Marginals and Probability of Evidence
Complexity Related to Network and Query Structure
Query Structure and Complexity: Network Pruning
I
Consider computation of Pr(Q, e) (includes prior marginals
and probability of evidence as special cases)
I
Pruning nodes: All leaves 6∈ Q ∪ E, interatively
I
I
I
Pruning edges: For each edge U → X , U ∈ E
I
I
Worst case: All leaves ∈ Q ∪ E, no pruning
Best case: All Q ∪ E are roots, every node 6∈ Q ∪ E pruned
Remove edge, shrink CPT ΘX |U by removing rows inconsistent
with e and removing column U
Effective treewidth is treewidth of pruned network given query
Jinbo Huang
Reasoning with Bayesian Networks
Building Bayesian Networks
Inference by Variable Elimination
Elimination and Factors
Prior Marginals by Elimination
Elimination Orders
Posterior Marginals and Probability of Evidence
Complexity Related to Network and Query Structure
Inference by Variable Elimination: Summary
I
Factors, summing out, multiplication
I
Prior marginals by elimination, bucket elimination
I
Width of elimination order, min-degree, min-fill
I
Posterior marginals and probability of evidence by zeroing out
I
Treewidth and complexity
I
Network pruning based on query
Jinbo Huang
Reasoning with Bayesian Networks