Using Relevance Feedback in
Multimedia Databases
Chotirat “Ann” Ratanamahatana
Eamonn Keogh
7th International Conference on VISual Information Systems
at 10th International Conference on Distributed Multimedia Systems
September 9, 2004
Roadmap
• Time series in multimedia databases and their similarity
measures
• Euclidean distance and its limitation
• Dynamic time warping (DTW)
• Global constraints and R-K Band
• Relevance Feedback and Query Refinement
• Experimental Evaluation
• Conclusions and future work
What are Time Series
• A collection of observations made sequentially
in time.
• People measure things…
• Their blood pressure
• George Bush's popularity rating
• The annual rainfall in San Francisco
• The value of their Google stock
and things…change over time…
Time Series in Multimedia Databases
Image data may best be thought of as time series…
Image to Time Series
Video to Time Series
1.5
1
Steady pointing
0.5
Hand moving to
shoulder level
0
-0.5
Hand moving down
to grasp gun
-1
Hand moving above holster
-1.5
Hand at rest
0
50
100
150
Time Series in Multimedia Databases
Hand moving to
shoulder level
Aiming gun
Hand moving down
to grasp gun
Video
Hand moving
above holster
0
10
20
Hand at rest
30
40
50
60
70
80
90
George Washington’s
Manuscript
1
0.5
0
0
50
100
150
200
250
300
350
400
450
Classification in Time Series
Pattern Recognition is a
type of supervised
classification where an
input pattern is classified
into one of the classes
based on its similarity to
these predefined classes.
Which class does
belong to?
Class A
Class B
Euclidean Distance Metric
Given 2 time series
Q = q1, …, qn and
C = c1, …, cn
their Euclidean distance is
defined as
1.5
1
0.5
0
Q
-0.5
C
-1
-1.5
0
50
100
150
0
50
100
150
1.5
1
D(Q, C ) 
n
2
(
q

c
)
 i i
i 1
0.5
0
-0.5
-1
-1.5
Limitations of Euclidean Metric
Very sensitive to some
distortion in the data
Training data consists
of 10 instances from
each of the 3 classes
Perform a 1-nearest neighbor
algorithm, with “leaving-one-out”
evaluation, averaged over 100 runs.
Euclidean distance Error rate:
29.77%
DTW Error rate:
3.33 %
Dynamic Time Warping (DTW)
Euclidean Distance
One-to-one alignments
Time Warping Distance
Non-linear alignments are allowed
How Is DTW Calculated? (I)
Q
DTW (Q, C ) min 


C

K
k 1
wk
C
Q
Warping path w
How Is DTW Calculated? (II)
Each warping path w can be found using dynamic programming to evaluate
the following recurrence:
 (i, j )  d (qi , c j )  min{  (i  1, j  1),  (i  1, j ),  (i, j  1)}
where γ(i, j) is the cumulative distance of the distance d(i, j) and its minimum
cumulative distance among the adjacent cells.
(i-1, j)
(i-1, j-1)
(i, j)
(i, j-1)
Global Constraints (I)
Prevent any
unreasonable
warping
C
C
Q
Q
Sakoe-Chiba Band
Itakura Parallelogram
Global Constraints (II)
A Global Constraint for a sequence of size m is defined by R, where
Ri = d
0  d  m, 1  i  m.
Ri defines a freedom of warping above and to the right of the diagonal
at any given point i in the sequence.
Ri
Sakoe-Chiba Band
Itakura Parallelogram
Ratanamahatana-Keogh Band
(R-K Band)
Solution: we create an arbitrary shape and size of the band that is
appropriate for the data we want to classify.
How Do We Create an R-K Band?
First Attempt: We could look at the data and manually create the shape of the bands.
(then we need to adjust the width of each band as well until we get a good result)
1.5
1
1
0.5
0.5
0
0
-0.5
-0.5
-1
-1
-1.5
-1.5
-2
-2
-2.5
0
50
100
150
200
250
-2.5
250
250
200
200
150
150
100
100
50
50
50
100
150
200
250
0
100 % Accuracy!
50
100
150
200
250
50
100
150
200
250
Learning an R-K Band Automatically
Our heuristic search algorithm automatically learns the bands from the data.
(sometimes, we can even get an unintuitive shape that give a good result.)
1.5
1
1
0.5
0.5
0
0
-0.5
-0.5
-1
-1
-1.5
-1.5
-2
-2
-2.5
0
50
100
150
200
250
-2.5
250
250
200
200
150
150
100
100
50
50
50
100
150
200
250
0
50
100
150
200
250
50
100
150
200
250
100 % Accuracy as well!
R-K Band Learning With Heuristic Search
Calculate h(1)
Calculate h(2)
h(2) > h(1) ?
Yes
No
R-K Band Learning in Action!
Classification Examples with R-K Bands
Error rate
Euclidean
32.13%
DTW 10%
4.52%
R-K Bands
0.9%
Face Classification
Relevance Feedback
• A well-known and effective method in
improving the query performance,
especially in text-mining domains.
– Refining the query based on user’s reaction
• Only relatively little research has been
done on relevance feedback in images or
multimedia data.
Query Refinement
Averaging a collection of time series using DTW,
according to their weights and warping (DTW) alignments.
Averaged Sequence
Experiment: Datasets
1.
Gun Problem
2.5
2.5
2
2
1.5
1.5
1
1
0.5
0.5
0
0
-0.5
-0.5
-1
0
50
100
150
-1
2.
Leaf Dataset
3.
Handwritten Word Spotting data
0
50
100
150
Experimental Design
Given an initial query, we measure the
precision and recall for each round of
the relevance feedback retrieval.
• Show the 10 best matches (k-nearest neighbors).
• User ranks each result.
• Accumulatively build the training set.
• Learn an R-K band according to the current training data.
• Generate a new query (query refinement), and repeat.
Results: Gun
1
0.9
0.8
0.7
Iteration
Iteration
Iteration
Iteration
Iteration
Precision
0.6
0.5
1
2
3
4
5
0.4
0.3
0.2
0.1
Gun
0
0.1
0.2
0.3
0.4
0.5
0.6
Recall
0.7
0.8
0.9
1
Results: Leaf
1
0.9
0.8
0.7
Precision
0.6
Iteration
Iteration
Iteration
Iteration
Iteration
0.5
0.4
1
2
3
4
5
0.3
0.2
0.1
Leaf
0
0.1
0.2
0.3
0.4
0.5
0.6
Recall
0.7
0.8
0.9
1
Results: Wordspotting
1
0.9
0.8
Iteration
Iteration
Iteration
Iteration
Iteration
0.7
Precision
0.6
1
2
3
4
5
0.5
0.4
0.3
0.2
0.1
WordSpotting
0
0.1
0.2
0.3
0.4
0.5
0.6
Recall
0.7
0.8
0.9
1
Conclusions
• Different shapes and widths of the band
contributes to the classification accuracy /
precision.
• We have shown that incorporating R-K
Band into relevance feedback can reduce
the error rate in classification, and improve
the precision at all recall levels in video
and image retrieval.
Future Work
• Investigate other choices that may make envelope
learning more accurate.
– Heuristic functions
– Search algorithm (refining the search)
• Is there a way to always guarantee an optimal solution?
• Examine the best way to deal with multi-variate time
series for more complex data.
• Explore other utilities of R-K Band and relevance
feedback, specifically on real-world problems: music,
bioinformatics, biomedical data, etc.
Contact: [email protected]
[email protected]
Homepage: http://www.cs.ucr.edu/~ratana
All datasets are publicly available at:
UCR Time Series Data Mining Archive: http://www.cs.ucr.edu/~eamonn/TSDMA