Case Study 2
A hierarchical Bayesian network for event recognition
of human actions and interactions
Sangho Park, J.K. Aggarwal
Multimedia Systems, vol. 10, pp. 164-179, 2004
Fuzzy Systems
Lifelog management
Outline
• Overview
• Post estimation using a hierarchical Bayesian network
• Recognition by DBN
• Relative constraints
• Experiment
• Summary
1/35
Overview
• Recognition of human interaction
– Applications: video surveillance, video-event annotation, virtual reality,
human-computer interaction, and robotics
– Difficulty: Ambiguity caused by body articulation, loose clothing, and
mutual occlusion between body parts
• Previous work
– A method to segment and track multiple body parts in two-person
interactions
– Multilevel processing at pixel level, blob level, and object level
2/35
Motivation
• A methodology
– To estimate body-part pose
– To recognize different two-person
interactions including pointing,
punching, standing hand-in-hand,
pushing, and hugging
• System component
– Bayesian network: estimate body poses
– Dynamic Bayesian network: classify a
sequence of body poses
3/35
Head Pose Estimation
• Environmental setup: lighting conditions, reflectance of light from the head
• Head pose: head’s 3D rotation angles, visible part
4/35
Head Pose Estimation: Example
• V1: angle of the vector
• V2: ratio of the two ellipses
• a, B: fixed
• P(V1=C|H2=B) = 0.18
5/35
Arm Pose Estimation
6/35
Arm Pose Estimation: Example
• P(V5=B|H3=C, H4) = 0.34
7/35
Leg Pose Estimation
8/35
Leg Pose Estimation: Example
9/35
Overall Body Pose Estimation (1)
10/35
Overall Body Pose Estimation (2)
11/35
Body-part Pose Recognition by DBN
• DBN hidden states
– Q1: set of DBNs for legs {“both legs are together on the ground”, “both
legs are spread on the ground”, “and “one foot is moving in the air while
the other is on the ground”}
– Q2: set of DBNs for the torso {“stationary”, “moving forward”, and “at least
one arm gets withdrawn”}
– Q3: set of DBNs for arms {“both arms stay down”, “at least one arm
stretches out”, and “at least one arm gets withdrawn”}
12/35
Interaction Recognition
• Whole-body pose: q1, q2, q3
– {“stand still with arms down”, “move forward with arm(s) stretched
outward”, “move backward with arm(s) raised up”, “stand stationary while
kicking with leg(s) raised up”, etc.}
• Two-person interaction
– Subject = {torso, arm, leg}
– Verb = {raise, lower, stretch, withdraw, stay, move forward, move
backward}
– Object = {head, upper body, hand, lower body}
13/35
Relative Constraints: Spatial
• Examples
– “standing hand-in-hand”: the torsos of the two persons be side by side
and facing in the same direction
– “pointing at the opposite person”: the torsos of face one another
• Relative position and orientation
– Gross level: proximity between two persons
– Intermediate level: relative orientations of the torso poses between the
two persons
– Detailed level: relative configuration of individual body parts
14/35
Relative Constraints: Temporal
• Interval temporal logic: before, meet, overlap, start, during, and finish
• Example
– A pushing interaction
• Event A: a person moving forward with arms stretched outward toward
the second person
• Event B: m of the second person
15/35
Experimental Results
• Human interaction (9): approaching, departing, pointing, standing hand-inhand, shaking hands, hugging, punching, kicking, and pushing
• Image
– 320*240 pixels
– 15 fps
– 6 pairs of different people with various clothing
– 56(?) sequences (6*9)
16/35
Interaction Examples
17/35
BN’s Belief Changes
18/35
Performance of DBNs
• Leave-one-out-cross-validation
– Training: 5 sequences, test: 1 sequence
• Accuracy: 78%
– Approaching: 100
– Departing: 100
– Pointing: 67
– Standing hand-in-hand: 83
– Shaking hands: 100
– Hugging: 50
– Punching: 67
– Kicking: 83
– Pushing: 50

similar interaction


occlusion
similar interaction

similar interaction
19/35
Semantic Interpretation
20/35
Summary
• Contribution
– A hierarchical framework for the recognition of two-person interactions
– BN for managing ambiguity in human interaction
– A human-friendly vocabulary for high-level event description
– Stochastic graphical model
• Future works
– Extending the method to crowd behavior recognition
– Incorporating various camera-view points
– Recognizing more diverse interaction patterns
21/35
Case Study 3
Evolutionary Learning of Dynamic Probabilistic Models
with Large Time Lags
A. Tucker, X. Liu and A. Ogden-Swift
International Journal of Intelligent Systems,
Vol. 16, no. 5, pp.621-646, 2001.
Fuzzy Systems
Lifelog management
Outline
• Introduction
• Background
• Methodology
• Algorithm
• Evaluation
• Conclusions
23/35
Introduction
• Multivariate time-series (MTS)
– A large number of interdependent variables
– Large time lags between causes and effects (ex. Oil refinery processing)
• Learning dynamic Bayesian networks
– Not focused on learning models automatically
– Focused on models with small time lags
– Challenge task for large datasets with large possible time lags
24/35
Background
Dynamic Bayesian Networks
• Bayesian networks
– A set of n nodes {x1,…, xn}, representing the N variables in the domain
– Each node, xi has a finite set of ri mutually exclusive states, vi1 to viri.
– Each node xi with a set of parents, πi has an associated probability table
P(xi|πi).
• Dynamic Bayesian networks consist of BNs at differing time slices
– Links over different time lags (non-contemporaneous links) and within the
same time lag (contemporaneous links)
25/35
Background
Learning Bayesian Network Structures
• K2/K3 algorithms
– Use a greedy search which begins with an empty structure with no links
– Explores the effect of adding each of the possible links to the current
structure
– K2 (a log likelihood metric) / K3 (a description length metric)
• Branch and Bound technique
– Perform an efficient exhaustive search by stopping any further exploration
along a search path based on a bound
• Evolutionary methods
– Larranaga et al. used a genetic algorithm with a repair operator to remove
cycles
– Wong et al. used evolutionary programming with freeze, defrost and a
knowledge guided mutation (KGM)
– Sahami used the mutual information
• Missing data management: Structural EM algorithm with Dempster’s
expectation maximization algorithm
26/35
Background
Evolutionary Learning Bayesian Network Structures
• Learning BNs involves scoring candidate network structures
– Log likelihood
– Description length metric of a network structure
– Description length metric of encoding the dataset given that model
– n: number of nodes
– ri: possible instantiations of the node
– qi: possible instantiations of the parent nodes
– Fij = ∑Fijk
– Fijk: frequency of occurrences in the dataset that the node xi takes on the
value vik and the parent nodes πi take on the instantiation wij
27/35
Methodology
Representation
• Assume that a dynamic network contains no contemporaneous links
• n = N (# of variables at a single time slice) + Q (# of variables at previous time
slice)
• A list of triples represents a possible networks (a,b,l)
– a: the parent variable
– b: the child variable
– l: the time lag
28/35
Methodology
Useful Heuristics
• No contemporaneous links
– Finding a good network structure  finding a group of simple tree
structures
• LagMutation: Each mutation is based on a uniform distribution with mean
equal to the present lag
• Autoregression links (a,a,1)
29/35
Methodology
Seeded GA for Search
• Seed the entire first population with links found from the single link analysis
– Using an approximate method to find a good list of single links rather than
scoring the entire set
– Exploiting this knowledge in the first population by seeding it entirely with
a random selection of good links
• EP method is particularly efficient at finding a good selection of links with
good correlation
– An individual represents a single triples
– Self-adapting parameters (SAP)
30/35
Algorithm
31/35
Evaluation: Efficiency (1)
• Adapting static BN search algorithms for DBN search
– K2/K3
– The genetic algorithm
– The evolutionary program
• Knowledge guided mutation (KGM)
32/35
Evaluation: Efficiency (2)
33/35
Evaluation: Structural Comparison
34/35
Conclusion
• Problem: Learning dynamic probabilistic models with large time lags
• Proposed method: EP-Seeded GA
• Future works: Discretisation & parameterisation
• Brainstorming
– Mutually not-exclusive states  Fuzzy BN
– Hybrid of the GA and K2/K3
35/35