Integrating User Feedback Log into
Relevance Feedback by Coupled SVM for
Content-Based Image Retrieval
9-April, 2005
Steven C. H. Hoi * , Michael R. Lyu *, Rong Jin #
*Department
of Computer Science & Engineering
The Chinese University of Hong Kong
Shatin, N.T., Hong Kong SAR
#
Department of Computer Science and Engineering
Michigan State University
East Lansing, MI 48824, USA
The 1st IEEE EMMA Workshop
in conjunction with 21st IEEE ICDE, Japan, April, 2005.
1
Outline
• Introduction
• Background

Log-based Relevance Feedback
• Coupled Support Vector Machine




Support Vector Machine
Formulation
Alternating Optimization
A Practical Algorithm
• Experimental Results
• Conclusion
2
Introduction
• Content-based Image Retrieval (CBIR)



An important component in visual information retrieval
QBE: query-by-example based on low-level visual
features
Semantic gap: low-level features, high-level concepts
QBE
3
Introduction
• Relevance Feedback (RF)

A powerful tool to attack the semantic gap problem
 Interactive mechanism to solicit users’ feedbacks
 Boost the retrieval performance of CBIR greatly
 Many existing techniques already…
• Problems

Regular relevance feedback needs too many rounds of
interactions for achieving satisfactory results.
4
Introduction
• Motivation
Relevance Feedback
?
User Feedback Log
Problem
Can user feedback log be used to improve the
regular relevance feedback?
5
Background
• Log-based Relevance Feedback (LRF)



Relevance Matrix: R
RF round / Log session: Nl images are marked
Elements: relevant (1), irrelevant (-1), unknown (0)
Image samples
1
-1
1
-1
-1
0
1
-1
-1
1
1
-1
1
-1
-1
-1
0
-1
-1
1
-1
-1
Log Sessions
6
Background
• Learning Problem for LRF

Low-level image content:
X  {x1 , x2 , , x N }

User feedback log:
R  {r1 , r2 ,, rN }

Multi-Modal Learning Problem
7
Coupled Support Vector Machine
• Motivation


How to attack the learning problem on the two modalities?
• Low-level Image content:
X
• User relevance feedback log:
R
Support Vector Machines: superior classification performance
• A Straightforward Solution:

Learn an SVM classifier on each modality respectively
• For image content X, we learn an optimal weighting vector w;
• For log content R, we learn an optimal weighting vector u;

Combine their results together linearly
8
Coupled Support Vector Machine
• A Straightforward Solution

For the image content modality: wTx

For the user feedback log modality: uTr
9
Coupled Support Vector Machine
• Disadvantages of the straightforward solution


Linear combination
Modality Consistence
• Our better solution: Coupled SVM


Learn the two modalities in a unified formulation
Enforce the prediction on the two types of information
to be consistent.
10
Coupled Support Vector Machine
• Formulation: Coupled SVM
11
Coupled Support Vector Machine
• Optimization of Coupled SVM


Hard to be solved directly
Alternating Optimization (AO)
• AO: two-step optimization


Fix Y’, try to find (u, b_u), and (w, b_w)
Fix (u, b_u) and (w, b_w), try to find Y’
12
Coupled Support Vector Machine
• Alternating Optimization

Fix Y’, the primal optimization is equivalent to solving
the two optimization subproblems:
13
Coupled Support Vector Machine
• Alternating Optimization (AO)

By introducing non-negative Lagrange multipliers, the
above two subproblems can be solved
14
Coupled Support Vector Machine
• Alternating Optimization (AO)

After solving (u, b_u) and (w, b_w), fixing them, the
optimal Y’ can be found to fit the data as follows:
15
Coupled Support Vector Machine
• Summary of AO procedure



1) Beginning with a small value of 
2) Performing the two-step AO procedure
3) Repeating 2) by increasing
until it achieves the setting

threshold
• Comments on the Coupled SVM




Can be a general approach for multi-modal learning problems
Need to investigate the convergence issue of Alternating
Optimization
Need to study better methods for solving the optimization
problem
Require to take some practical considerations when fitting for
specific problems.
16
Coupled Support Vector Machine
• A Practical Algorithm

Practical considerations
• Cannot engage all unlabeled samples due to response
requirement for relevance feedback
• Strategy for choosing unlabeled samples
– Closest to the decision boundary of SVM: most informative
according to active learning
– Closest to the labeled samples: to avoid too much effort in
learning the label information
• Introducing a parameter  to control the error for label
correction to avoid overlarge change in the labeled set
17
Coupled Support Vector Machine
• A Practical Algorithm (cont’d)
18
Coupled Support Vector Machine
• A Practical Algorithm (cont’d)
19
Experimental Results
• Dataset


Images selected from COREL image CDs
Two ground-truth datasets
• 20-Category: each category contains 100 images, totally 2,000
• 50-Category: each category contains 100 images, totally 5,000
20
Experimental Results (cont’d)
• Low-level Image Representation

Color Moment
• 9-dimension

Edge Direction Histogram
• 18-dimension
• Canny detector, 18 bins of 20 degrees each

Wavelet-based texture
• 9-dimension
• Daubechies-4 wavelet, 3-level DWT
• Entropies of 9 subimages are generated for the texture feature
21
Experimental Results (cont’d)
• Collection of User Log Data

Log format
• A log session (LS) corresponds a relevance feedback round
• Each log session contains 20 images labeled by users

Log data
• On 20-Category: 161 log sessions
• On 50-Category: 184 log sessions
22
Experimental Results (cont’d)
• CBIR GUI for collecting feedback data
23
Experimental Results (cont’d)
• Performance Evaluation

Measurement Metric
• Average Precision = # relevant images / # returned images

Experimental Setting
• 100 queries
• 20 initially labeled images
• SVM: RBF kernel, parameters set via training data

Comparison Schemes
• RF-SVM
– traditional relevance feedback by SVM
• LRF-2SVM
– log-based relevance feedback by learning two SVMs respectively
• LRF-CSVM
– log-based relevance feedback by Coupled SVM
24
Experimental Results (cont’d)
• Performance Evaluation: on 20-Category Dataset
25
Experimental Results (cont’d)
• Performance Evaluation: on 50-Category Dataset
26
Experimental Results (cont’d)
27
Experimental Results (cont’d)
28
Conclusion
• A log-based relevance feedback scheme was studied by
integrating user feedback log into the content learning of
low-level visual features in content-based image retrieval.
• A general multimodal learning technique, i.e. Coupled
Support Vector Machine, was proposed for studying the
data with multiple modalities.
• A practical algorithm by Coupled SVM was presented to
attack the log-based relevance feedback problem in CBIR.
• Experimental results show our proposed scheme is effective
for the log-based relevance feedback problem.
29
Q&A
30
References
•
Chu-Hong Hoi and Michael R. Lyu, A Novel Log-based Relevance
Feedback Technique in Content-based Image Retrieval, in Proc.
ACM Multimedia, New York, USA, 10-16 October, pp. 24-31, 2004
•
S. Tong and E. Chang. Support vector machine active learning for
image retrieval. In Proc. ACM Multimedia, pages 107--118, 2001.
31