Tallin, November 2007
Introduction to Selected Concepts
of Image Processing
Part II
Jiří Jan
Department of Biomedical Engineering
Brno University of Technology
Czech Republic
II. IMAGE ENHANCEMENT
Purpose: ad hoc enhancement of the subjective impression or better utilisation of the
information content, or partial compensation of imperfections of the imaging
process
easier image analysis and classification
better diagnostic efficiency
(selective enhancement of some features at the expense of some other ones, but
NOT improvement of image quality or information content,
NOT approaching the undistorted original image, based on a distorted
one
intuitive, rather heuristic approaches (often try-and-err approach)
(though some substantiation is desirable and usually useful)
Areas of application:
-
contrast transforms, colour transforms
sharpening, edge enhancement
smoothing, noise reduction, retouching
geometrical distortion compensation
others (e.g. illumination equalisation, …)
1
1. Contrast transforms
realised by
point-wise operations gi,k = N(fi,k)
usually defined by a plot
or
by a look-up-table
(LUT)
usually global (isoplanar)
but adaptability is possible
realisation
by means of ROM/RAM
Piecewise linear contrast transform
increase of contrast in a part of the grey scale
at the cost of decreasing the contrast in other
parts of the scale
- special case: applying full contrast in a chosen limited input range of grey f1 … f2
(i.e. increase of image dynamics from f1 .. f2 to the available range 0..gmax )
g i ,k =
g max
( fi,k − f1 )
f 2 − f1
2
Example: (increase of dynamics by piecewise linear transform)
original image
enhanced image
related histograms
Example of a monotonous nonlinear transform:
(compression of a part of input range, expansion of another range, incomplete use of the output range)
original and enhanced images
related histograms
3
Other (possibly piecewise) linear transforms :
negative
(zebra-transform)
Example of linearly descending (monotonous) point-wise transform
original
(negative)
its negative
corresponding histograms
4
Example
of nonlinear
monotonous
transform:
input image and its histogram
γ - correction
(γ>1, γ<1)
resulting image with
histogram
after correction with γ = 1,75
resulting image with
histogram
after correction with γ = 0.65
Examples of a non-monotonous nonlinear transform:
5
Histogram Equalisation
Histogram indicates the correctness of exposition (i.e. utilisation of the available
dynamic range)
When all the shades of grey are present approximately equally, the image makes a good
impression („brilliant“ image).
this can be achieved
- by optimum exposition under well arranged illumination, or
- by subsequent image modification:
histogram equalisation (HE)
– means to find a suitable point-wise contrast transform g=
N(f)
Application of HE in view of the following use of the image:
– to improve the appearance of the image
– as a normalisation of the grey scale in differently exposed images,
useful both
- for visual evaluation or comparison of images and,
- as a pre-processing step for automatic analysis
Continuous formulation of HE
the brightness (grey shade) is taken as a stochastic variable
f
histogram is considered approximation of the probability distribution of
f
Equalisation procedure – as transformation of probability distribution:
the stochastic variable
f
(original grey shades) with its distribution function Pf
(f)
is transformed
into a new stochastic variable g
distribution function
(f)
(grey shades of the enhanced image) with the required
Pg ( g )
by means of the relation
g ( f ) = Pg−1 (Pf ( f ))
g
Specifically, for HE it is required
the inverse function is
pg ( g ) = 1/gmax ,
Pg−1 ( x ) = g max x ,
and therefore, the sought transform function is
thus
x ∈ 0,1
Pg ( g ) = ∫
0
1
g
dg =
g max
g max
g ( f ) = g max Pf ( f )
(in practice, the approach suffers from imperfections due to discretisation – also other approaches
used, designed for the discrete environment)
6
Example: equalising histogram of an image
original image
histogram of the input image
output image after equalisation
histogram after equalisation
histogram after a slight image smoothing
Local-histogram equalisation
as a way to adaptive equalisation
Example:
7
Pseudo-colouring
expressing the grey scale by a scale of „false“ colours
g iR,k = N R ( f i ,k ),
g iG,k = N G ( f i ,k ),
i.e. g i , k = N RGB ( f i , k )
g iB,k = N B ( f i , k )
Example
grey-scale image
pseudo-colour image
8
Example
grey-scale image
pseudo-colour image
Example
grey-scale image
pseudo-colour image
9
Colour image – decomposition into RGB components and respective histograms
original
(brightness
histogram)
G
R
B
Example – point-wise colour transforms
original
its negative
transformed image after
non-monotonous nonisochromic transform
10
2. Image sharpening and edge enhancement
Purpose:
- subjective impression improvement
- enhancement of details thus improvement of image readability (texts, numbers,
textures)
- edge enhancement → better segmentation and pattern recognition
Interpretation of sharpening:
- in the frequency domain – accentuating high spatial frequencies
- in the original domain – difference operators (as the opposite to integrating nature of
blur or smoothing)
Result: profile of a typical edge before and after sharpening – schematically
Differential and difference operators
Directional (anisotropic) operators
- partial derivatives
∂f ( x, y ) ∂f ( x, y )
,
∂x
∂y
-
2nd partial derivatives
∂2 f
,
∂x 2
∂2 f
∂y 2
- directional derivative in the direction s, given by a direction angle ϑ
∂f
∂f
∂f
=
cos ϑ +
sin ϑ
∂s ∂x
∂y
11
Isotropic operators:
linear combinations (either odd derivatives in an even power or even derivatives in an odd power)
⎛ ∂m f
⎜⎜ m
⎝ ∂x
most commonly used:
- amplitude of gradient
- Laplacian
k
⎛ ∂m f
⎞
⎟⎟ + ⎜⎜ m
⎝ ∂y
⎠
⎞
⎟⎟
⎠
k
⎛ ∂f ⎞ ⎛ ∂f ⎞
∇f (x, y ) = ⎜ ⎟ + ⎜⎜ ⎟⎟
⎝ ∂ x ⎠ ⎝ ∂y ⎠
2
L ( f (x, y )) = ∇ 2 f (x, y ) =
2
∂2 f ∂2 f
+ 2
∂x 2
∂y
Vector operators, e.g.
- gradient of grey shade
∇ f ( x, y ) =
∂f
∂f
j+
i
∂x
∂y
with its amplitude field
⎛ ∂f ⎞ ⎛ ∂f ⎞
∇ f (x, y ) = ⎜ ⎟ + ⎜⎜ ⎟⎟
⎝ ∂ x ⎠ ⎝ ∂y ⎠
with its direction field
ϑ ( x, y ) = arctan ⎜⎜
2
⎛ ∂f
⎝ ∂x
2
∂f ⎞
⎟
∂y ⎟⎠
12
Difference approximations of derivatives in discrete environment
∂f ( x i , y k ) Δ x f i , k
≈
,
∂x
Δx
- approximations of the 1st derivatives
where the 1st differences are
- second differences (approximations of the
2nd derivative)
∂f ( x i , y k ) Δ y f i , k
≈
,
Δy
∂y
Δ x f i , k = f i , k − f i −1, k , Δ y f i , k = f i , k − f i , k −1 .
Δ2x f i , k = Δ x f i +1, k − Δ x f i , k = f i +1, k + f i −1, k − 2 f i , k ,
Δ2y f i , k = Δ y f i , k +1 − Δ y f i , k = f i , k +1 + f i , k −1 − 2 f i , k .
- the corresponding masks of local (linear) operators
⎡0 − 1 0 ⎤
⎡0 − 1 0 ⎤
⎡ 0 0 0⎤
1
Δ x f i , k : ⎢⎢0 1 0 ⎥⎥ or ⎢⎢0 0 0 ⎥⎥ , Δ y f i ,k : ⎢⎢ − 1 1 0 ⎥⎥ etc.
2
⎢⎣0 0 0 ⎥⎦
⎢⎣0 1 0 ⎥⎦
⎢⎣ 0 0 0 ⎥⎦
⎡0 1 0 ⎤
⎡0 0 0 ⎤
Δ2x f i , k : ⎢⎢0 − 2 0 ⎥⎥ , Δ2y f i , k : ⎢⎢1 − 2 1 ⎥⎥
⎣⎢0 1 0 ⎥⎦
⎣⎢0 0 0 ⎦⎥
Example: 1-st difference operators
original image
horizontal difference
vertical difference
approximate derivative along x
approximate derivative along y
⎡0 − 1 0 ⎤
⎡0 − 1 0 ⎤
1⎢
Δ x f i ,k : ⎢⎢0 1 0 ⎥⎥ or
0 0 0 ⎥⎥ ,
2⎢
⎢⎣0 1 0 ⎦⎥
⎣⎢0 0 0 ⎦⎥
⎡ 0 0 0⎤
Δ y f i ,k : ⎢⎢ − 1 1 0 ⎥⎥
⎣⎢ 0 0 0 ⎥⎦
etc.
13
Approximations of isotropic operators
- amplitude of gradient
- approximation of the square-root:
or max ( a , b )
a2 + b2 ≈ a + b ,
- other possible approximations of the 1st. derivatives (under directions ±450)
(Roberts operator : with max choice)
Δ s f i ,k :
⎡0 0 − 1⎤
⎢0 0 0 ⎥
⎥
2 2 ⎢
⎣⎢1 0 0 ⎥⎦
1
a
⎡− 1 0 0⎤
⎢ 0 0 0⎥
⎥
2 2 ⎢
⎣⎢ 0 0 1 ⎥⎦
1
- Laplacian (as the sum of perpendicular 2nd differences)
⎡0 1 0 ⎤
L { f i , k } : ⎢⎢1 − 4 1 ⎥⎥
⎣⎢0 1 0 ⎥⎦
Difference operators, more robust with respect to image noise
(but less locally determined)
- for directional differences
Δ x f i ,k :
⎡ − 1 − 1 − 1⎤
1⎢
0
0
0 ⎥⎥ , or
6⎢
⎢⎣ 1
1
1 ⎥⎦
Δ y f i ,k :
⎡ − 1 0 1⎤
1⎢
− 2 0 2 ⎥⎥ etc.
8⎢
⎢⎣ − 1 0 1 ⎥⎦
- for Laplacian
L { f i ,k }:
⎡1 1 1⎤
⎢1 − 8 1⎥
⎥
⎢
⎢⎣1 1 1⎥⎦
or
2
1⎤
⎡1
⎢ 2 − 12 2 ⎥
⎥
⎢
⎢⎣1
2
1 ⎥⎦
14
Sharpening operators
Sharpening in the original domain
Unsharp negative mask: subtraction of an unsharp version of the image L from the original F –
- relative suppression of low-frequency components, dependent on the weight a
G = aF − L
the masks e.g.:
⎡0 0 0 ⎤
10 ⎢
0 1 0 ⎥⎥
9 ⎢
⎢⎣0 0 0 ⎥⎦
−
⎡1
1⎢
1
9⎢
⎢⎣1
1
1
1
1⎤
1⎥⎥
1⎥⎦
⎡ − 1 − 1 − 1⎤
1⎢
− 1 9 − 1⎥⎥
9⎢
⎢⎣ − 1 − 1 − 1⎥⎦
=
Another interpretation: subtraction of Laplacian from the original,
⎡0 0 0⎤
⎢0 1 0⎥
⎢
⎥
⎢⎣0 0 0 ⎥⎦
−
⎡1 1 1⎤
⎢1 − 8 1⎥
⎢
⎥
⎢⎣1 1 1⎥⎦
⎡0 0 0 ⎤
⎢0 1 0 ⎥
⎢
⎥
⎢⎣0 0 0 ⎥⎦
−
⎡0 1 0 ⎤
⎢1 − 4 1 ⎥
⎢
⎥
⎢⎣0 1 0 ⎥⎦
=
⎡ − 1 − 1 − 1⎤
⎢ − 1 9 − 1⎥
⎢
⎥
⎢⎣ − 1 − 1 − 1⎥⎦
or
=
⎡ 0 −1 0 ⎤
⎢ − 1 5 − 1⎥
⎥
⎢
⎢⎣ 0 − 1 0 ⎥⎦
Sketch of formal sharpening derivation by subtracting weighted Laplacian
Formulation:
- original image f(x,y) ~ initial distribution g(x,y,0) of dissolved dye in a thin water layer
- blurred image ~ distribution g(x,y,t) after a certain time of diffusion action on the dye
according to diffusion equation:
∂g ( x , y , t )
= k∇ 2 g ( x , y , t )
∂t
( every point of the original image, i.e. the point dye concentration, is blurred due to
diffusion into a PSF similar to 2D Gaussian function)
Derivation of the sharpening formula:
Expanding
g(x,y,t)
into the Taylor series around t = τ yields
f ( x , y ) = g ( x , y ,0 ) = g ( x , y , τ ) − τ
∂g
(x, y ,τ ) + ... .
∂t
Substituting for the time derivative from the above eq. and neglecting the higher order terms provides
f ( x, y ) ≈ g ( x, y ,τ ) − kτ ∇ 2 g ( x, y ,τ )
ƒ
15
Example: use of sharpening operators
original
sharpened images
sharpening masks
⎡ − 1 − 1 − 1⎤
⎢ − 1 12 − 1⎥
⎢
⎥
⎢⎣ − 1 − 1 − 1⎥⎦
⎡ − 1 − 1 − 1⎤
⎢ − 1 9 − 1⎥
⎢
⎥
⎢⎣ − 1 − 1 − 1⎥⎦
Example of application of a sharpening operator
original
image
sharpened
image
details
16
Sharpening via frequency domain
examples of spectrum modification (accentuating relative share of high-frequency components):
- linear accentuation (or modified)
H (u , v ) = w =
(u
2
)
+ v2 ≈ u + v
- exponential accentuation (α > 0)
H (u , v ) = eα w
- suppressing dynamics of the amplitude spectrum (0<α <1)
G (u , v ) = F (u , v ) e jϕ (u ,v ) , α ∈ 0,1
α
Example of sharpening by suppressing the dynamics of the amplitude spectrum
- original
- sharpened with α=0.9
α=0.7
α=0.5
17
Adaptive sharpening
principle:
enhanced sharpening only at or in the neighbourhood of edges, while in the „flat“
areas less or no sharpening
(with respect to more visible noise disturbance in the flat areas and suppressed
perception of noise in the vicinity of edges or in high contrast areas)
realisation:
- local degree of edge presence must be evaluated
- depending on the edge presence, the sharpening degree is controlled
(e.g.: by varying the central element in the sharpening mask, in dependence on the local variance
or local absolute gradient value)
example:
original
adaptively sharpened image
(with flat areas out of the edges)
3. Noise suppression
–
utilises available knowledge on the noise character,
but in principle, „blind“ suppression is used,
i.e. without a more detailed noise identification
Problem: suppression of noise tends to cause
loss of image acuity
– i.e. – a compromise must be sought
18
noise suppression
Classification of noise:
- according to dependence on image content
- independent on image content (mostly, e.g. thermal noise of optical sensors, impulse interference)
- dependent on image content (rarely, e.g. the grain of photographic material)
- according to spatial and amplitude distribution:
- „grey“ noise (afflicts usually all pixels, has a continuous amplitude distribution, ofte Gaussian)
- impulse noise „pepper and salt“ (afflicts only isolated pixels or their small groups, but with a large
amplitude, often leading to limit values – black and white)
- according to noise relation to image content
- additive noise (added to the original pixel-shade, e.g. thermal noise)
- multiplicative noise (multiplies the original pixel-shade, e.g. due to irregular illumination)
- more complex relations (e.g. convolutive noise – stochastic blur)
- according to spectral content
- wideband noise (in extreme – white noise, most of the previous types (e.g. thermal, impulse,…)
- narrowband noise (usually manifests itself as moiré, possibly of several spectral components)
Suppression of narrow-band noise
- in the frequency domain – by narrow band-stop filters
Example – before filtering
spectrum
filtering in spectral domain
detail of spectrum
resulting image
19
Suppression of wideband „grey“ noise
- afflicted of the image pixels,
- zero mean value of noise, the variance determines the SNR
Principle of the approach:
(weighted) averaging of identical or simialr useful grey values → the noise components
tend to cancel each other
- improvement of SNR via decreasing the mean noise power:
uniform weighting provides the highest noise suppression, the resulting noise power is
⎧
N
σ υ2 aver = var ⎨∑
⎩ i =1
σ2
1 ⎫ N 1
υ i ⎬ = ∑ 2 var {υ i } = ν
N ⎭ i =1 N
N
i.e. one N-th of the original noise power – in the same relation, the power SNR is improved
wideband noise suppression
A – averaging of N identical (or almost identical) images
i.e. when available:
- either more images of the same scene wit different realisations of noise
- or periodic structures in a single image,
- or time sequences of images – video
→ leads to deterioration of the time resolution, smeared movements
g i,k =
1
N
N
∑
i =1
i
f m ,n ,
(m, n ) ∈ A
B – spatial averaging in a single image (smoothing) – averaging in frame of a small sub-image
g i , k = ∑ cm , n f m , n ,
(m, n ) ∈ A i ,k
A i ,k
character,
i.e. application of convolutional – mask operator with the averaging (i.e. high-stop)
usually isoplanar, e.g.
⎡1 1 1⎤
⎡1 1 1⎤
⎡1 2 1 ⎤
1⎢
⎥ , 1 ⎢1 2 1⎥ , 1 ⎢ 2 4 2 ⎥
1
1
1
⎥ 10 ⎢
⎥ 16 ⎢
⎥
9⎢
⎢⎣1 1 1⎥⎦
⎢⎣1 1 1⎥⎦
⎢⎣1 2 1 ⎥⎦
(or also anisoplanar – adaptive, see later)
20
Example of averaging more realisations of the same image
noisy image (Gaussian noise)
average of two images
average of four images
Example of noise suppression by spatial smoothing
original noisy image
after smoothing with uniform mask 5 x 5
smoothed by adaptive mask
21
another example of smoothing an image by an invariant mask
details
Adaptive suppression of „grey“ noise
(dual to adaptive sharpening)
principle (with respect to human sight properties):
strong smoothing in the continuous (flat) areas, out of edges and details
- weak or no smoothing in local high-contrast areas (edges, details)
-
a - selective smoothing
decisive criterion:
local variance of grey values in a local area Bi,k of a chosen size:
if it is less than a chosen threshold p
σ i2,k =
2
1
∑ ( f m ,n − f i ,k ) < p,
count (B i , k ) − 1 Bi ,k
(m, n )
area average will be used
g i ,k =
1
∑ f m ,n
count (B i , k ) B i ,k
otherwise it is left unchanged
22
Adaptive noise suppression
b - „smart“ smoothing
principle:
adaptively growing averaging area around the pixel i,k – only pixels conforming to
the criterion
f i ,k − f i + j , k +l < s ,
j , l ∈ − L, L
where the threshold s depends on the noise level in image thus on the variance σi,k2
are added to the average.
This way, it is prevented that different pixels, e.g. behind an edge, would be
added → the area adapts its shape to the edge form (and remains in the region of
approximately equal grey values)
stopping criterion (of adding):
-
after reaching a chosen count of pixels N,
-
or after reaching a chosen sum of values S
(smoothing is then more effective at low levels, as needed e.g. in SPECT)
-
or if all suitable pixels according to the criterion are exhausted
(around edges or details, lower smoothing is achieved, as needed)
Example of „smart“ smoothing
with the indicated parameters
noisy image
noise level σ = 0.1 (max=1)
smoothed image
N=50, L=11, s=0.3
smoothed image
N=50, L=11, s=1.0
23
c - other methods of adaptive smoothing of „grey“ noise
- method of gradient controlled diffusion (GCD)
more formalised
- noisy image represented as the initial dye distribution,
-
smoothing by simulated GCD (i.e. by solving the diffusion equation)
during a chosen time period (a parameter),
with locally dependent diffusion velocity, derived from the local detail content,
e.g. characterised by local abs. gradient
-
many other heuristic approaches based on the above mentioned principles
Suppression of impulse noise
Method of detection and substitution of the false pixels values
characterised by:
- isolated noisy (false) pixels
- strong, up to saturated affliction of the false pixels
(„pepper and salt“ noise)
steps:
1.
detection of false pixels
2.
substitute the false pixel values by new grey degrees obtained somehow from the
neighbouring not afflicted pixels (mostly by interpolation)
24
Impulse noise suppression
Detection of false pixels :
- based on obvious difference between the fals pixel (i,k) and neighbouring pixels in a
local area A
some heuristic criteria:
- sum of absolute differences
- count of unsimilar pixels in A
- absolute difference from the average of A
∑∑
j
count
f i ,k − f i + j ,k +l > s ,
( j, l )∈ A
l
{( j,l )∈ A : f
f i ,k −
1
N
∑∑ f
j
i ,k
}
− f i + j ,k +l > s > M
i + j ,k +l
>s;
( j, l )∈ A
l
The behaviour of detectors is influenced – besides their type - also by the (heuristic)
parameter choice.
Other factors may come into the detecting criterion, e.g. periodicity of false pixels.
Impulse noise suppression
Substitution for false pixel values
method:
some estimate based on neighbouring (supposedly) undistorted pixels, e.g.
- nearest neighbour (e.g. LU),
- average of a chosen local area (excluding the false pixel)
- interpolation from neighbouring pixels – linear, bilinear, bicubic, …
- for binary images: the colour (black or white) prevailing in the chosen neighbourhood,
possibly taking into account connectivity etc.
-----------------------------------------------------If there is uncertainty concerning the detection, its likelihood may be quantified by the function p
p (criterionv alue − threshold ) ∈ 0,1
and, instead of binary resolution substitute/unchanged a compromising value can be used:
g i ,k = p f i ,k + (1− p ) f i , k
where f
is the substitution (interpolated) value
i ,k
---------------------------------------------------------------
Important comment:
the method „detection – substitution“ is always a nonlinear image processing
(i.e. does not conform to the superposition principle even though the interpolation might be linear)
25
Suppression of impulse noise by median filtering (or by other order-statistics
filtering)
-
conceptually simpler than the detection-substitution method
-
but demanding computation:
- for each mask position (each input pixel position) generate the ordered sequence of input grey
values
- select the median of the sequence (or any other chosen ordered member of the sequence) as
the output value
- assign the output value into the reference input pixel under the mask (mostly central)
-
nonlinear approach – difficult analysis
-
effective for impulse noise, rather ineffective for grey noise
Example of median filtering
image distorted by pepper-and-salt noise
result of the median filtering
details
26
Example: impulse noise suppression by median filtering
noisy image – pepper and salt noise
result of the median filtering
comment: in case of grey noise afflicting all pixels, the median filter is ineffective
original image with Gaussian noise
details: after adaptive smoothing
the image after median filtering
after median filtering
27
A comment on suppressing non-additive noise
method (homomorphic filtering):
– non-additive noise should be first converted to additive, by a suitable trasnform
– then, linear filtering can be applied, chosen with respect to the signal and noise
properties
– after the filtering, inverse transform to the previous one must be used to remove the
(mostly nonlinear) distortion of the signal
This approach is the sc.
homomorphic filtering
when the noise is multiplicative (e.g. due to uneven illumination)
the suitable transforms are
– logarithmic
– exponential
Suppressing non-additive noise
Example: application of homomorphic filtering
multiplicative noise caused by uneven illumination approximately suppressed
– logarithmic transform, mild high-pass filtering and exponential transform used
input image
log-transformed
resulting image
(exp. transform after low-stop filter)
28
29