Context-Aware Saliency Detection
Stas Goferman, Lihi Zelnik-Manor, Ayellet Tal
CVPR,2010
Outline





Introduction
Detection of context-aware saliency
Results
Application
Conclusion
Introduction(1)

Saliency identify fixation points?
Original image
Saliency map
Fixation points
This type of saliency is important for understanding human attention as
well as for specific applications such as auto focusing.
Introduction(2)

Saliency identify the dominant object ?
Original image
Saliency map
Itti’s
By labeler
This type of saliency is useful for several high-level tasks, such as object
recognition or segmentation.
Introduction(3)

Context -aware
For some application ,like
summarization of a photo
collection ,and retargeting…
maintain the essential regions of the
image is important.
Introduction(4)

Here , introducing a new type of saliency algorithm
contain not only the prominent objects but also the parts
of the background that convey the context.

The algorithm follows four psychological principles of
human visual attention.

We demonstrate the applicability of two application,
image retargeting and summarization.
Outline





Introduction
Detection of context-aware saliency
Results
Application
Conclusion
Principles of context-aware saliency

1. Local low-level considerations, including factors such
as contrast and color.
the distinctive color and other features area should obtain high attention.

2. Global considerations, which suppress frequently
occurring features, while maintaining features that
deviate from the norm.
redundant information should be suppressed and popping up the novelty part.
Principles of context-aware saliency

3. Visual organization rules, which state that visual forms
may possess one or several centers of gravity about
which the form is organized.
the salient pixels should be grouped together, and not spread all over the
image.

4. High-level factors, such as human faces.
implemented as post-processing.
Principles of context-aware saliency
Comparing different approaches to saliency
(a)Local[24]
(b)Global[7]
(c)Local-global[13]
(d)Context-aware
Local-global single-scale saliency

1.
We should not, however, look at an isolated
pixel, but rather at its surrounding patch.

2.
It suffices to consider the K most similar
patches
Let dcolor(pi , pj) be the Euclidean
distance between the vectorized
patches pi and pj in CIE L*a*b color
space, dposition(pi , pj) be the
Euclidean distance between the
positions of patches pi and pj
The single-scale saliency value
of pixel i at scale r is defined
as left . {qk} k=1 to K , K = 64 in
our experiments
Multi-scale saliency enhancement

3.
Background pixels (patches) are likely to
have similar patches at multiple scales.

4.
The larger Si is, the more salient pixel i is and
the larger is its dissimilarity to the other
patches.
For a patch pi of scale r, we consider
as candidate neighbors all the
patches in the image whose scales
are Rq = {r ,1/2r ,1/4r} .
Let R denote the set of patch sizes to
be considered for pixel i.The saliency
at pixel i is taken as the mean of its
saliency at different scales
Including the immediate context

1: A pixel is considered attended if its saliency value
exceeds a certain threshold( Si > 0.8 in the examples
shown in this paper).

2: The saliency of a pixel is redefined as
Let dfoci(i) be the Euclidean positional distance between pixel i and the
closest focus of attention pixel, normalized to the range [0,1]
Steps

The steps of our saliency estimation algorithm
High-level factors

Final , face detection algorithm
modified
Si
, if Si > face(i)
face(i)
, otherwise
Si =
face detection algorithm of [23], which generates 1 for face
pixels and 0 otherwise.
[23] P. Viola and M. Jones. Rapid Object Detection Using a Boosted Cascade of Simple Features. In CVPR, 2001.
Outline





Introduction
Detection of context-aware saliency
Results
Application
Conclusion
Salient results (1)
Compare to other methods with
three cases image.
(a) a single object over an uninteresting
Background.
(b) the immediate surroundings of the
salient object is also salient.
(c) complex scenes.
From left to right input , result of [24] , result of [7], our result .
[24] D.Walther and C. Koch. Modeling attention to salient proto objects. Neural Networks, 19(9):1395–1407, 2006.
[7] X. Hou and L. Zhang. Saliency detection: A spectral residual approach. In CVPR, pages 1–8, 2007.
Salient results (2)
Comparing our saliency results with [13]
Top: Input images.
Middle: The bounding boxes obtained by
[13] capture a single main object.
Bottom: Our saliency maps convey the story .
[13] T. Liu, J. Sun, N. Zheng, X. Tang, and H. Shum. Learning to Detect A Salient Object. In CVPR, 2007.
Outline





Introduction
Detection of context-aware saliency
Results
Application
Conclusion
Image retargeting
Image retargeting aims at resizing an image by expanding or
shrinking the non-informative regions.
Distortions, if and when introduced, will exist only in regions
of lower significance.
Input
Saliency of [19]
Our saliency
Results of [19]
Our result
Figure 9. Seam carving of 100 ”vertical“ lines. The salient objects are distorted by [19] in contrast to our results.
[19]M. Rubinstein, A. Shamir, and S. Avidan. Improved seam carving for video retargeting. ACM Trans. on Graphics,27(3), 2008.
Summarization through collage creation

this technique consists of three stages
Compute saliency
maps of the images
Extract regions-of-interest(ROI)
by considering both saliency
and image edge information
Assemble the non-rectangular
ROIs ,allowing slight overlaps.
[4]S. Goferman, A. Tal, and L. Zelnik-Manor. Puzzle-like collage. Computer Graphics Forum (EUROGRAPHICS), 29, 2010.
Summarization through collage creation
Summarization of a trip to LA using 14 images.
(a)The saliency maps of the input images
(b)The collage summarization
Outline





Introduction
Detection of context-aware saliency
Results
Application
Conclusion
Conclusion

This paper proposes a new type of saliency – context
aware saliency based on four principles observed in the
psychological literature which detects the important parts
of the scene.

In the future we intend to learn the benefits of this
saliency in more applications, such as image classification
and thumbnailing.
Appendix

Global methods : Saliency Detection: A Spectral Residual Approach [7]
H(Image) = H(Innovation) + H(Prior Knowledge)
R(f) = L(f) − A(f)
A(f) = hn (f) ∗ L(f)