Statistical Image Quality
Measures
Hiroyuki Takeda, Hae Jong Seo, Peyman Milanfar
EE Department
University of California, Santa Cruz
Jan 11, 2008
Overview
Background
CCA-based Similarity Measure (Full-reference)
SVD-based Quality Measure (No-reference)
Conclusion
Slide 1
UCSC MDSP Lab
Objective Quality Assessment
Develop quantitative measures that automatically
predict the perceived image quality
Full-reference
No-reference
Reduced-reference
Applications
Image acquisition, compression, communication,
displaying, printing, restoration
Slide 2
UCSC MDSP Lab
Overview
Background
CCA-based Similarity Measure (Full-reference)
SVD-based Quality Measure (No-reference)
Conclusion
UCSC MDSP Lab
Full-Reference Image Quality Measure
Structural Similarity Measure [1]
Focus on perceived changes in structural information
variation unlike error based approach ( i.e. MSE or PSNR )
MSE : 210
Mean shifted
Blurred
JPEG compressed
Contrast stretched
Salt-pepper
Original image
[1] Zhou Wang et al, “Image Quality Assessment: From Error Visibility to Structural Similarity ”, IEEE TIP ‘ 04
Slide 3
UCSC MDSP Lab
Structural Similarity Measure
Three components : Luminance , Contrast , Structure
Small constant
Image patches being compared
Slide 4
UCSC MDSP Lab
Drawback of SSIM
SSIM: 0.505
Original
Zoom Out
SSIM: 0.549
Translation
SSIM: 0.551
Rotation
Sensitive to spatial translation, rotation, and scale changes
due to simple correlation coefficient
Solution
A powerful statistical tool
: Canonical Correlation Analysis (Hotelling, 1936)
Slide 5
UCSC MDSP Lab
New Statistical Image Quality Measure
Canonical Correlation Analysis (CCA)
: Find out a pair of direction vectors
which
maximally correlate the two datasets

: canonical correlation
: Useful property  Affine–invariance
Slide 6
UCSC MDSP Lab
New Statistical Image Quality Measure
Canonical Correlation Structural Similarity Measure
: Local Search Window at i
th
position
P : Pixel intensity
Gx,Gy : Gradients
A
original image
B
noisy, sigma= 35
50
50
100
100
P
150
CCA
P
150
200
200
250
250
Gx Gy
300
CCA
Gx Gy
300
350
350
400
400
450
P Gx Gy
500
100
200
300
400
CCA
450
P Gx Gy500
100
500
Slide 7
200
300
400
500
UCSC MDSP Lab
New Statistical Image Quality Measure
Mathematical Solution
1) Calculate Covariance Matrix
2) Solve coupled eigen-value problems
3) Define CCSIM as largest canonical correlation
Slide 8
UCSC MDSP Lab
Examples (1)
Original
Image
original image
Zoom Out
1
2
Slide 9
UCSC MDSP Lab
2
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
SSIM: MSSIM = 0.3419WinSize = 5Distribution of CC1(compressed): Pixel + Gradient Value -->Mean(0.73212) Bl
1
SSIM
Slide 10
200
0.8
0.7
150
0.6
0.5
100
0.4
0.3
0.2
0.1
50
2
0.9
0
CCSIM
200
Zoom Out
0.73
150
0.34
100
1
1
50
original image
Distribution of CC1(Compressed): Pixel -->Mean(0.3098) Block Size:5
Original Image
1
Examples (2)
UCSC MDSP Lab
Examples (2)
Original
Image
original image
1
Translation
3
Slide 11
UCSC MDSP Lab
3
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
SSIM: MSSIM = 0.38452WinSize = 5 Distribution of CC1(compressed): Pixel + Gradient Value -->Mean(0.75026) B
1
SSIM
Slide 12
200
0.8
0.7
150
0.6
0.5
100
0.4
0.3
0.2
0.1
50
3
0.9
0
CCSIM
200
Translation
0.75
150
0.38
100
1
1
50
original image
Distribution of CC1(Compressed): Pixel -->Mean(0.3098) Block Size:5
Original Image
1
Examples (2)
UCSC MDSP Lab
Examples (3)
Original
Image
original image
1
Rotation
4
Slide 13
UCSC MDSP Lab
4
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
SSIM: MSSIM = 0.41067WinSize = 5Distribution of CC1(compressed): Pixel + Gradient Value -->Mean(0.77315) Bl
1
SSIM
Slide 14
200
0.8
0.7
150
0.6
0.5
100
0.4
0.3
0.2
0.1
50
4
0.9
0
CCSIM
200
Rotation
0.77
150
0.41
100
1
1
50
original image
Distribution of CC1(Compressed): Pixel -->Mean(0.3098) Block Size:5
Original Image
1
Examples (3)
UCSC MDSP Lab
JPEG Compression Example
Clean image
(QF=100)
1
JPEG(QF=50)
2
8 bits/pixel
JPEG(QF=10)
3
0.899 bits/pixel
Slide 15
0.352 bits/pixel
UCSC MDSP Lab
JPEG Compression Example
SSIM: MSSIM = 0.90153WinSize = 5
Clean Image
0.90
1
1
SSIM: MSSIM
= 0.786WinSize
= 5 Block Size:5
Distribution of CC1(Compressed):
Pixel
-->Mean(0.3098)
1
0.8
0.8
0.7
0.7
0.7
3 0.6
0.6
0.6
0.5
0.5
0.5
0.4
0.4
0.4
0.3
0.3
0.3
0.2
0.2
0.2
100
150
JPEG (QF =50)
2
0.9
0.9
0.8
50
2
0.79
1 0.9
11
200
SSIM
SSIM
0.1
0.1
0.1
Distribution of CC1(compressed): Pixel + Gradient Value -->Mean(0.8533)
Block Size:5
Distribution
of CC1(compressed): Pixel + Gradient Value -->Mean(0.79459) Block Size:5
01
1
00
0.85
JPEG (QF =10)
3
CCSIM
50
150
0.79
200
0.9
0.8
0.8
0.7
0.7
0.6
0.6
0.5
0.5
0.4
0.4
0.3
0.3
0.2
0.2
0.1
0
Slide 16
100
0.9
CCSIM
0.1
0
UCSC MDSP Lab
Clean Image VS Compressed Images
Quality
1
0.95
Pixel + Gradient : Window Size = 5
SSIM : Window Size = 5
SSIM
0.9
CCSIM
0.85
0.8
0.75
10
20
30
40
50
60
70
80
90
100
JPEG quality factor
Slide 17
UCSC MDSP Lab
Denoising Example
Clean Image
original image
1
WGN(sigma=15) Denoised by SKR[2]
noisy, sigma= 15
2
denoised by SKR, sigma= 15
3
[2] Takeda et al., “ Kernel Regression for image processing and reconstruction ”, IEEE TIP ‘ 07
Slide 18
UCSC MDSP Lab
Denoising Example
SSIM: MSSIM = 0.47514WinSize = 5
Clean Image
original image
1
0.47
1
2
Distribution of CC1(Compressed):
-->Mean(0.3098)
SSIM: MSSIM =Pixel
0.88668WinSize
= 5 Block Size:5
1
1
0.9
50
0.8
0.8
0.8
0.7
0.7
0.7
0.6
0.6
0.6
0.5
0.5
0.5
0.4
0.4
0.4
0.3
0.3
0.3
0.2
0.2
0.2
3
100
150
WGN( sigma =15 )
noisy, sigma= 15
2
200
SSIM
0.89
11
SSIM
0.9
0.9
0.1
0.1
0.1
Distribution of CC1(Noisy): Pixel -->Mean(0.4755) Block Size:5
Distribution of CC1(Denoised): Pixel + Gradient Value -->Mean(0.88536) Block Size:5
1
01
0
0
50
100
150
200
0.47
Denoised by SKR
denoised by SKR, sigma= 15
3
CCSIM
0.9
0.8
0.8
0.7
0.7
0.6
0.6
0.5
0.5
0.4
0.4
0.3
0.3
0.2
0.2
0.1
0
Slide 19
0.89
0.9
CCSIM
0.1
0
UCSC MDSP Lab
Clean VS (Noisy & Denoised images)
Clean VS Noisy
Clean VS Denoised
Quality
1
0.8
Quality
SSIM
0.9
Pixel: Window Size = 5
SSIM : Window Size = 5
0.8
0.6
0.7
CCSIM
0.4
0.2
5
10
15
CCSIM
SSIM
Pixel+Gradient: Window Size = 5
SSIM : Window Size = 5
0.6
20
25
30
0.5
5
WGN: Noise level
10
15
20
25
30
WGN: Noise level
Slide 20
UCSC MDSP Lab
Super-resolution
Motion
Estimation
Steering Kernel
Regression
Resolution
enhancement from
video frames captured
by a commercial
webcam
(3COM Model No.3719)
Slide 21
UCSC MDSP Lab
Super-resolution Example
Low resolution
Sequence
(64x64 32 frames)
Clean Image
(512 x 512)
original image
1
2
Super-resolved
by SKR
superresolved by SKR
3
Slide 22
UCSC MDSP Lab
Super-resolution Example
Clean Image
original image
1
1
3
SSIM: MSSIM = 0.86996WinSize = 5Distribution of CC1(Denoised): Pixel + Gradient Value -->Mean(0.91575) B
1
0.87
0.91
0.9
0.8
Low resolution
Sequence( 32 frames)
0.7
0.6
2
0.5
0.4
0.3
0.2
SSIM
CCSIM
Slide 23
0
0.1
0.2
0.3
0.4
0.6
0.5
200
Mean(0.3098) Block Size:5
1
3
0.7
0
0.8
superresolved by SKR
0.1
0.9
Super-resolved by SKR
UCSC MDSP Lab
Overview
Background
CCA-based Similarity Measure (Full-reference)
SVD-based Quality Measure (No-reference)
Conclusion
UCSC MDSP Lab
No-Reference SVD-Based Measure
Singular value decomposition of local gradient matrix:
SVD
NxN
Local orientation dominance
It becomes close to 1 when there is one dominant orientation
in a local area.
It takes on small values in flat or highly textured (or pure noise)
area.
So, this quantity tells us about the “edginess” of the region being
examined.
UCSC MDSP Lab
Properties of Local Orientation Dominance(1)
Density function for i.i.d. white Gaussian noise
N: the window size
N=11
Note : the PDF is independent from the noise
variance, but depends on the window size.
N=9
N=7
N=5
N=3
[1] A. Edelman. Eigenvalues and condition numbers
of random matrices, SIAM Journal on Matrix Analysis
and Applications 9 (1988), 543-560.
[2] X. Feng and P. Milanfar. Multiscale principal component
Analysis for Image Local Orientation Estimation, Proceeding
of 36th Asilomar Conference on Signals, Systems, and
Computers, Pacific Grove, CA, Nov. 2002
Slide 25
UCSC MDSP Lab
Properties of Local Orientation Dominance(2)
The mean values for a variety of test images with added white
Gaussian noise.
N = 11
The mean values for
pure noise are
always constant.
0.06
Remember
the number
Slide 26
UCSC MDSP Lab
The Performance Analysis
Suppose we have a noisy image and a denoised version
using some filter:
: a given noisy image
: the estimated (denoised) image
: the residual image
If the filter cleans up the given image effectively,
The residual image is essentially just noise.
of the residual image must be close to the value expected
for pure noise.
Slide 27
UCSC MDSP Lab
Example (1)
Image denoising by bilateral filter
Bilateral filter has two parameters:
Spatial smoothing parameter
, and radiometric smoothing parameter
Denoising experiment
The original image
A noisy image, Added white Gaussian noise,
SNR=20dB, PSNR=29.25dB, RMSE = 8.67
C. Tomasi and R. Manduchi, “Bilateral filtering for gray and color images”, Proceedings of the 1998 IEEE
International Conference of Computer Vision, Bombay, India, pp. 836-846, January 1998.
Slide 28
UCSC MDSP Lab
The Performance Analysis of Bilateral Filter
The plot of
as a function of the smoothing parameters:
N = 11
Slide 29
UCSC MDSP Lab
Denoising Result
The noisy image
Bilateral filter
PSNR = 42.87dB,
RMSE = 1.833
Slide 30
Residual
UCSC MDSP Lab
The Performance Analysis of Bilateral Filter
The plot of
as a function of the smoothing parameters:
N = 11
Slide 31
UCSC MDSP Lab
Denoising Result
The filter also
removes image
contents.
The noisy image
Bilateral filter
PSNR = 39.57dB
RMSE = 2.68
Slide 32
Residual
UCSC MDSP Lab
What If We Pick the Parameters by the Best RMSE?
The plot of RMSE as a function of the smoothing parameters:
Slide 33
UCSC MDSP Lab
Denoising Result
The noisy image
Bilateral filter,
PSNR = 42.87dB
RMSE = 1.832
Slide 34
Residual
UCSC MDSP Lab
Example (2)
Iterative Steering Kernel Regression
Iteratively cleaning up noisy images
Using the local orientation dominance, we find the optimal
number of iterations.
The original image
The noisy image,
Added white Gaussian noise,
SNR=5.6dB, PSNR = 20.22dB RMSE = 24.87
Slide 35
UCSC MDSP Lab
Denoising Result (1)
The plot of
as a function of the smoothing parameters:
Slide 36
UCSC MDSP Lab
Denoising Result
The noisy image
ISKR, IT = 15,
PSNR = 31.33 dB
RMSE = 6.92
Slide 37
Residual
UCSC MDSP Lab
If the Ground Truth is Available,
The plot of RMSE as a function of the smoothing parameters:
RMSE
Slide 38
UCSC MDSP Lab
Denoising Result
The noisy image
ISKR, IT = 12,
PSNR = 31.69 dB
RMSE = 6.64
Slide 39
Residual
UCSC MDSP Lab
Overview
Background
CCA-based Similarity Measure (Full-reference)
SVD-based Quality Measure (No-reference)
Conclusion
UCSC MDSP Lab
Conclusion
Two new statistical quality measures
CCSIM(CCA-based) : full-reference
SVD-based measure: no-reference
CCSIM is a general version of SSIM
We showed examples of JPEG compression, denoising , and super
Resolution with comparison to SSIM
SVD-based measure is applicable for any
denoising filter.
We illustrated application to global parameter optimization.
Locally adaptive parameter optimization is also possible.
The proposed methods can be easily extended to
video using 3-d local window.
Slide 40
UCSC MDSP Lab
Authors
[1] Hiroyuki Takeda : [email protected]
www.ucsc.edu/~htakeda
[2] Hae Jong Seo : [email protected]
www.ucsc.edu /~rokaf
[3] Peyman Milanfar : [email protected]
www.ucsc.edu/~milanfar
UCSC MDSP Lab
Thank you !
UCSC MDSP Lab
Super-resolution Example
Clean Image
original image
1
Down-sampled(2)
+WGN(sigma=15)
2
Super-resolved
by SKR
noisy, sigma= 15
3
Extra 1
UCSC MDSP Lab
Super-resolution Example
Clean Image
original image
1
1
3
SSIM: MSSIM = 0.70632WinSize = 5
Distribution of CC1(Noisy): Pixel + Gradient Value -->Mean(0.85761) Block Si
1
0.71
0.85
0.9
Down-sampled(2)
+WGN (sigma=15)
0.8
0.7
0.6
2
0.5
0.4
0.3
0.2
SSIM
CCSIM
0.1
0
200
0.1
0.2
0.3
0.4
0.5
0.6
1
Mean(0.3098) Block Size:5
Extra 2
0.7
0
3
0.8
noisy, sigma= 15
0.9
Super-resolved by SKR
UCSC MDSP Lab