Game Theoretic Image Segmentation
Elizabeth Cassell
Sumanth Kolar
Alex Yakushev
Introduction – Image Segmentation
Distinguish objects from background
 Analysis of underlying structures
 Different conditions and content
 Applications
 Robot vision
 Pattern Recognition
 Biomedical image processing

Some examples of
Image Segmentation
Strategies for Image Segmentation

Threshold techniques


Edge Based techniques


Colour Information
Shape Information
Region Based

Growing Step by Step
Computational Geometry
 Previous shape knowledge. (Circles/Ellipse)
 Several more.. Clustering, Histogram

Current Problem
2 class problem, foreground and background
 No restriction of connected components
 Seed Image given as input.
 Noisy Images.

Game Theoretic Approach
Image Segmentation - two-person non-zero-sum non cooperative
game
Two players, one minimizing, other maximizing an energy function
 Region based segmentation module
Goal is to find the region based on color information
 Boundary finding module
Goal is to find a closed boundary shape
Image representation
M
N
• Grayscale images are
represented as NxM
matrices. yi,j is the
intensity at of a pixel at
i,j
• A pixel is assigned a
class xi,j, in the two-class
case: xi,j is either 1 or 0.
• An exhaustive search
would require 2NxM
operations
Image Examples
• Additive noise
• Reasons - Input device sensor and low signal level,
such as shadow regions or underexposed images
Player 1: Region Based Module
Start from a seed image, compute the energy function E for neighboring
pixels.
Add the pixels, for which the value of E is below a certain threshold.
Repeat, until no more pixels can be added.
Region Based Module (details)
Minimization of the energy term – data fidelity term and second term
enforces smoothness

2
E   yi , j  xi , j      xi , j  xis , js 
i, j
i , j is , j s


 

data fidelity term

2


enforces smoothness
Where yi,j is a pixel value from the original image, xi,j is
the classification of that pixel, and is,js represents the neighborhood of
the pixel
Region Segmentation - Works
Region Segmentation - problems
• Given a very noisy
image, a lot of pixels
will be missed.
Player 2: Boundary Finding
Source:
http://www.lems.brown.edu/~msj/cs29
2/project/intermediate.html
• General idea is to find a closed boundary
around the object of interest.
• Boundary constraints usually include
smoothness, and closeness to a prior
• Boundary Finding method is trying to
maximize a function of the curve parameters
• E.g. If we were looking for ellipses we would
be looking for the right values of x0, y0, a, and
b (center point, major and minor axes)
Boundary Finding
Class of objects with smooth boundaries that are deformable.
Impose global structure information on the segmentation





arg max
M prior  p   M gradient I g , p 
 M  p, I g   arg max

p
p
Where






M  p, I g   ln P  p | I g  ; M prior  p   ln P  p  ; M gradient I g , p   ln P I g | p 

p is the vector of parameters used to parameteri ze the contour
I g is the gradient image
M prior is the prior shape term
M gradient is the likelihood term which depends on the gradient. It is a measure

of the likelihood of p being the true boundary
Boundary Finding - Implementation
Morphological operation – Closing
Dilation followed by erosion
Closing tends to narrow smooth sections of contours, fusing narrow breaks and long thin
gulfs, eliminating small holes, and filling gaps in contours.
Flow diagram - Integration
Region Segmentation
Object Boundary (p)
Image regions (Ir)
Image
Ir
Boundary Finding
p
Region Based Segmentation - Integration



2
2
2
min E  min  yi , j  xi , j      xi , j  xis , js 
x
x
i , j is , j s

 i , j







Region segmentation term


2
2
    xi , j  u    xi , j  v  
( i , j ) A p

( i , j )Ap

Matching region and boundary

 min  f1 x   f 21 x, p 
x
Ap is the area inside the boundary; u is the mean value of the pixels in Ap ;
Ap is the area outside the boundary v is the mean value of the pixels in Ap
Boundary Finding - Integrated
Posed as maximum a posteriori framework
Last term incorporates region based information. Maximum
when the classification by the region based module is correct.

arg max
M  p, I g , I r  

p



arg max
M prior  p   M gradient I g , p    M region I r , p  

p


max
  f 2  p    f12 I r , p 

p
I g is gradient image
I r is region segmented image

p is parameteri zed contour

Test Images
Results
Results
Nash Equilibrium
Objective of each player : Minimise pay off function
Find Nash Equilibrium
A pair of strategies p1 , p 2 constitute a Nash Equilibriu m
if p1 , p 2
   
F  p ,p   F  p ,p 
F 1 p1,p 2  F 1 p1,p 2
2
1
2
2
1
2
Where F 1 and F 2 are the cost functions for player 1
and player 2, respective ly
Nash Existence
For objective functions of this form, Nash equilibrium always
exists for proper choices of alpha and beta
1

  f  p  f  p , p 
 p , p   f  p   f  p , p 
1
F p ,p
2
F2
2
1
1
1
2
1
2
21
2
2
1
12
F 1 , F 2 are cost functions; f1 is contributi on of region based
module alone;  ,  are scaling constants; p , p are strategies
1
2
Conclusion




Game Theory can be applied to image
segmentation
Produces better results than each of the
individual modules
Combination method is more robust to noise
Future work includes learning the seed and
using different region and boundary finding
algorithms