Journal of Mathematical Imaging and Vision 8, 109–119 (2000)
c 2000 Kluwer Academic Publishers. Manufactured in The Netherlands.
°
Vector Median Filters, Inf-Sup Operations, and Coupled PDE’s:
Theoretical Connections
VICENT CASELLES
Department of Informatics and Mathematics, University of the Illes Balears, 07071 Palma de Mallorca, Spain
GUILLERMO SAPIRO AND DO HYUN CHUNG
Department of Electrical and Computer Engineering, University of Minnesota, Minneapolis, MN 55455, USA
[email protected]
Abstract. In this paper, we formally connect between vector median filters, inf-sup morphological operations, and
geometric partial differential equations. Considering a lexicographic order, which permits to define an order between
vectors in IR N , we first show that the vector median filter of a vector-valued image is equivalent to a collection of
infimum-supremum morphological operations. We then proceed and study the asymptotic behavior of this filter.
We also provide an interpretation of the infinitesimal iteration of this vectorial median filter in terms of systems of
coupled geometric partial differential equations. The main component of the vector evolves according to curvature
motion, while, intuitively, the others regularly deform their level-sets toward those of this main component. These
results extend to the vector case classical connections between scalar median filters, mathematical morphology, and
mean curvature motion.
Keywords: vector median filtering, inf-sup operations, asymptotic behavior, anisotropic diffusion, curvature
motion, coupled PDE’s
1.
Introduction
Median filtering has been widely used in image processing as an “edge preserving” filter.1 The basic idea
is that the pixel value is replaced by the median of
the pixels contained in a window around it. This idea
has been extended to vector-valued images [6, 15, 16],
based on the fact that the median is also the value that
minimizes the L 1 norm between all the pixels in the
window. More precisely, the median of a finite series
M
is the number u j ∈ U such that
of numbers U = {u i }i=1
M
X
i=1
ku i − u j k1 ≤
M
X
ku i − u z k1 ,
∀u z ∈ U.
been used for directions as well [15, 16] (the median is
selected as the vector that minimizes the sum of angles
between every pair in U).
When considering u(x) : IR2 → IR as an image defined in the continuous plane, there is a close relationship between median filtering, inf-sup morphological
operations, and partial differential equations [2, 4], see
Section 2. The goal of this paper is to extend these theoretical results to the vectorial case, u(x) : IR2 → IR N .
This is presented in Section 3 . Other PDE’s approaches
for vector-valued images can be found for example in
[7, 11, 17].
(1)
i=1
N
Here, the pixels values {u i }i=1
can be either a scalar or
a vector. In the case of vector-valued images, (1) has
2.
Scalar Case
The background material in this section is adapted from
[4]. We refer the interested reader to these notes and
110
Caselles, Sapiro and Chung
references therein for details and proofs. (See also [9]
for connections and references to connections between
stack (median) filters and morphology.)
Definition 1. Let u(x) : IR2 → IR be the image, a map
from the continuous plane to the continuous line. The
median of u(x), with support given by a set B of
bounded measure centered at x, is defined as:
med B (u)(x)
½
:= inf λ ∈ IR : measure(x ∈ B : u(x) ≤ λ)
¾
measure(B)
.
(2)
≥
2
the disk centered at x0 and with radius t. Then,
med D(x0 ,t) (u)
1
= u(x0 ) + κ(u)k∇uk(x0 )t 2 + O(t 2+1/3 ), (4)
6
if k∇u(x0 )k 6= 0, and
¯
¯med D(x
0 ,t)
¯
− u(x0 )¯ ≤ |4u(x0 )|t 2 + O(t 3 )
otherwise.
This result also leads to the following very interesting relation: If t → 0, that is, the region of support
shrinks to a point, iterating the median filter is equivalent to solving the geometric partial differential equation
Definition 2. The median filtering of the whole image
u is the result of applying (2) for all pixels x in the
image.
From now on, we will consider and analyze only
the local operation around a given point x, that is,
Definition 1. It is understood that we refer to the application of this operation to the whole image.
Having these definitions in mind, it is easy to prove
the following result:
Proposition 1.
sup u(x),
med B (u) = inf
0
(3)
where
½
measure(B)
,
B := B 0 : measure(B 0 ) ≥
2
¾
B0 ⊂ B .
B ∈B x∈B 0
This relation shows that the median is an inf-sup
morphological operation. This is the first relation we
will extend to vector-valued images in the next Section.
We now report on the relation between median filtering and mean curvature motion, which constitutes the
second result we will extend to vector-valued images
(a first version of the following theorem, connecting
Gaussian weighted median filtering with mean curvature motion is due to Bence, Merriman, and Osher and
to Barles and Evans; see [4]).
Theorem 1. Let u be three times differentiable, κ(u)
the curvature of the level-set of u,2 and let D(x0 , t) be
(5)
∂u
= κk∇uk.
∂t
(6)
This relation is basically obtained by moving the cur2
vature term to the left side
√ and dividing by t , with
a rescaling of time t → t. See [4] for the formal
derivation.
This is one of the most popular image flows used
for image enhancement via anisotropic diffusion [1],
see also [10]. This flow is diffusing the image in the
direction of the level-sets of u, that is, perpendicular
to ∇u. Therefore, iterated median filtering is basically
anisotropic diffusion. From this point of view, median
filtering, which is classically used in image processing
in its discrete form and with finite support B, and was
of course derived long before curvature motion, can be
considered (a-posteriori) as a non-consistent numerical
implementation of anisotropic diffusion. This explains
for example the observed instabilities of discrete median filters, which do not occur in correct numerical
implementations of (6).
3.
Vectorial Case
As pointed out in the introduction, median filtering was
extended to vector-valued images using the L 1 minimization property. In order to use the definition analogous to the one given by Eq. (2), we need to impose an
order in the vector space. This order is necessary to obtain the vectorial extensions of the results presented in
the previous Section. In Section 4 we discuss the use
of other definitions to obtain median-type operations
for vector-valued images.
Vector Median Filters
In order to be able to compare between two vectors,
we assume a lexicographic order.3 Given two N dimensional vectors u and v, we say that u ≥ v if and only if
u = v, or u 1 > v1 , or u i = vi , for all 1 ≤ i < j ≤ N
and u j > v j . This order means that we compare the
coordinates in order, until we find the first component
that is different between the vectors. It is well known
that the lexicographic order is a total order, and any
two vectors in IR N can be compared. Given a set of
vectors 3 ∈ IR N we define its supremum (infimum) as
the least of the upper bounds (respectively, the greatest
of the lower bounds). That is, u ∗ = sup 3 means that
u ≤ u ∗ for all u ∈ 3 and if u ≤ v for all u ∈ 3 then
also u ∗ ≤ v. Analogous relations hold for inf 3 with
≥ instead of ≤. (The supremum and infimum always
exist in the lexicographic order. See also Lemma 1 below.) With this order in mind, the definition given by
Eq. (2) is consistent for the vectorial case as well, with
minor modifications:
Definition 3. Let u(x) : IR2 → IR N be the image, a
map from the continuous plane to the continuous space
IR N . The vector median filter of u, with support given
by a set B of bounded measure centered at x, is defined
as (over lines stands for set closure):
med B (u)
½
¾
measure(B)
.
:= inf λ ∈ IR N : measure(x ∈ B : u(x) ≤ λ) ≥
2
(7)
Remark. It is necessary to define the vector median
filter over the closure of a set to guarantee that if a series
of vectors u ² is greater or equal than a fix vector v, the
limit of the series is also greater or equal than v. This
property does not hold in the general case if the closure
is omitted (e.g., v = [0, 1000] and u = [², 0], ² → 0).
Note also that as in the scalar case, we consider only
the local behavior of the filter, while the median of the
whole image is obtained shifting B and applying (7) to
all the image pixels.
We proceed below to develop the main results of
this paper concerning the relations between vector median filtering (with a lexicographic order), inf-sup morphological operators, and geometric PDE’s. We should
note that for the developments below, it is enough to
consider N = 2, that is, a two dimensional vector. The
operations we show for N = 2 will hold as well for
N > 2, where we relate the component i + 1 to the
111
component i in the same way we will relate the second component (i = 2) to the first one (i = 1) in the
developments below. In other words, in order to process the component i + 1 we simulate the behavior
of the component i as if i = 1, and then the behavior of the component i + 1 is obtained as if it
were the second component of the vector. The process
is then performed in pairs (two-dimensional vectors),
(i = 1, i = 2), (i = 2, i = 3), (i = 3, i = 4), . . . , (i =
N − 1, i = N ), and the processed vector is given by
the two components of the first pair and all the second
components for the rest.
We should also note that in many cases there is
no natural lexicographic order between all the vector components, but there is just a relation of “importance” between one vector and the rest. For example,
in color representations like Lab and Yuv, the first component is usually more important than the other two,
but there is no natural order between the last two by
themself. In this case, the system is treated as a collection of two dimensional vectors of the form (u 1 , u i ),
1 < i ≤ N . For the Lab space for example, two pairs
of two-dimensional vectors are processed, (L , a) and
(L , b), and the processed Lab vector if given by the two
components of the first pair and the second component
of the second pair.
3.1.
Vector Median Filtering as an Inf-Sup Operator
Lemma 1 contains implicitly the fact that, if u is a
scalar function, we may restrict the sets in B to be level
sets of u. This is an obvious fact, if u : IR2 → IR is
a measurable function and B a subset of IR2 of finite
measure we always have
med B (u) = inf
sup u(x),
0
B ∈S x∈B 0
where
(8)
½
S := [u ≤ b] ∩ B : b ∈ IR,
¾
measure(B)
measure([v ≤ b] ∩ B) ≥
.
2
A formula similar to (8) also holds in the vectorial
case.
Proposition 2. Let u = (u 1 , u 2 ) : IR2 → IR2 be a
bounded measurable function such that measure([u 1 =
α]) = 0 for all α ∈ IR and B a subset of IR2 of finite
112
Caselles, Sapiro and Chung
This justifies (10). In a similar way, one can show that
measure. Then
sup u(x),
med B (u) = inf
0
(9)
B ∈G x∈B 0
where
½
G := [u ≤ λ] ∩ B : λ ∈ IR2 ,
measure([u ≤ λ] ∩ B) ≥
¾
measure(B)
.
2
Proof: For simplicity, for any function v from IR2
to IR N , N = 1, 2 we shall denote by [v ≤ λ] the set
{x ∈ B : v(x) ≤ λ}, λ ∈ IR N . To prove this proposition, we first observe that the infimum in the right hand
side of (9) is indeed attained. We have that
ϕ := inf
sup u(x) :
½
= inf
x∈B 0
B0
∈G
¾
sup u(x) : B ∈ G .
0
x∈B 0
(10)
Since the infimum in the lexicographic order of a
closed set is attained then ϕ = (ϕ1 , ϕ2 ) ∈
{supx ∈ B 0 u(x) : B 0 ∈ G}. Let Bn0 := [u ≤ λ0n ] ∈ G be
such that λn := sup Bn0 u → ϕ. Observe that Bn0 =
[u ≤ λn ]. Let λn = (λn,1 , λn,2 ). Let λ∗n,1 = sup{λn,1 ,
λn+1,1 , . . .}, λ∗n,2 = sup{λn,2 , λn+1,2 , . . .}. Then λ∗n,1 ↓
ϕ1 , λ∗n,2 ↓ ϕ2 . Thus, we have λ∗n = (λ∗n,1 , λ∗n,2 ) ≥
λn , λ∗n is decreasing and λ∗n → ϕ. It follows
that [u 1 ≤ ϕ1 ] = ∩n [u 1 ≤ λ∗n,1 ]. Since measure
([u 1 = α]) = 0 for all α ∈ IR and [u ≤ λ] = [u 1 < λ1 ] ∪
[u 1 = λ1 , u 2 ≤ λ2 ] for all λ = (λ1 , λ2 ) ∈ IR2 , then measure ([u ≤ λ]) = measure([u 1 ≤ λ1 ]) for all λ ∈ IR2 .
As a consequence, we have that
measure([u ≤ ϕ]) = measure([u 1 ≤ ϕ1 ])
= lim measure([u 1 ≤ λ∗n,1 ])
n
= lim measure([u ≤ λ∗n ])
n
≥ lim sup measure([u ≤ λn ])
n
measure(B)
.
≥
2
Now, by the definition of ϕ, we have
ϕ ≤ sup u ≤ ϕ.
[u≤ϕ]
and
med B (u) = inf{λ : λ ∈ F B }.
, we have
Now, since measure([u ≤ ϕ]) ≥ measure(B)
2
med B (u) ≤ ϕ. Since also measure([u ≤ med B (u)]) ≥
measure(B)
then ϕ ≤ sup[u≤med B (u)] u ≤ med B (u). Both
2
inequalities prove the Proposition.
2
For simplicity, we shall assume in what follows that
u = (u 1 , u 2 ) : IR2 → IR2 is a continuous function and
measure([u 1 = α]) = 0 for all α ∈ IR. As above we
shall write [v ≤ λ] to mean [v ≤ λ] ∩ B.
¾
½
med B (u) ∈ F B
¾
½
measure(B)
:= λ : measure(x ∈ B : u(x) ≤ λ) ≥
2
(11)
Proposition 3. Assume that u = (u 1 , u 2 ) : IR2 →
IR2 is a continuous function, measure([u 1 = α]) = 0
for all α ∈ IR and B is a compact subset of IR2 . Then
µ
med B (u) =
Proof:
med B u 1
¶
inf[x∈B:u 1 (x)=med B u 1 ] u 2 (x)
(12)
Let
µ
µ :=
med B u 1
inf[x∈B:u 1 (x)=med B u 1 ] u 2 (x)
¶
(13)
Since we assumed that measure([u 1 = α]) = 0 for
all α ∈ IR we have that measure([u ≤ µ]) ≥
measure(B)
. Then, by definition of med B (u) we have
2
that med B (u) ≤ µ. Now, let λ = (λ1 , λ2 ) ∈ F B . Since
measure([u 1 ≤ λ1 ]) = measure([u ≤ λ]) ≥ measure(B)
2
then med B (u 1 ) ≤ λ1 . If med B (u 1 ) < λ1 , then µ ≤ λ.
Thus we may assume that med B (u 1 ) = λ1 . Since
[u ≤ λ] = [u 1 < λ1 ] ∪ [u 1 = λ1 , u 2 ≤ λ2 ], if
inf[x ∈ B:u 1 (x) = med B u 1 ] u 2 (x) > λ2 , then [u ≤ λ] =
[u 1 < λ1 ]. Since u 1 is continuous then med B (u 1 ) ≤
sup[u≤λ] u 1 < λ1 . This contradiction proves that
inf[x∈B:u 1 (x)=med B u 1 ] u 2 (x) ≤ λ2 . Thus µ ≤ λ. We conclude that µ ≤ med B (u). Therefore µ = med B (u). 2
This Proposition means that for the first component
of the vector, the median is as in the scalar case, while
Vector Median Filters
for the second one, the result is obtained looking for
the infimum over all the pixels of u 2 corresponding to
the positions on the image plane where the median of
the first component is obtained. This result is expected,
since the first component already selects the whole possible vectors, and then the positions in the plane where
the second component can select from are determined.
Note also that new vectors, and then new pixel values,
are not created, as expected from a median filter. These
properties will also hold for the alternative definition
we present now:
Definition 4. Let u = (u 1 , u 2 ) : IR2 → IR2 be a measurable function and B ⊆ IR2 be of finite measure.
Define
sup u(x),
med∗B (u) = inf
0
where
½
H := [u 1 ≤ b] ∩ B : b ∈ IR,
measure(B)
measure([u 1 ≤ b] ∩ B)≥
2
¾
Proposition 4. Let u = (u 1 , u 2 ) : IR2 → IR2 be a continuous function and B be a compact subset of IR2 .
Then
µ
¶
med B u 1
med∗B (u) =
(15)
sup[x∈B:u 1 (x)=med B u 1 ] u 2 (x)
To prove this proposition we need the following
simple Lemma:
Lemma 1. Let πi : IR2 → IR be the projection of IR2
onto the i coordinate. Let 3 ⊆ IR2 . Then
i) π1 (sup 3) = sup π1 (3).
ii) If π1 (ū) = π1 (sup 3) for some ū ∈ 3 then
π2 (sup 3) = sup{π2 (u) : u ∈ 3,
π1 (u) = π1 (sup 3)}.
(16)
In particular, if 3 is compact we always have (16).
Similar statement holds for the infimum.
µ
q=
Let
med B u 1
sup[x∈B:u 1 (x)=med B u 1 ] u 2 (x)
Since measure ([u 1 ≤ med B (u 1 )]) ≥ measure(B)
, we have
2
that med∗B (u) ≤ sup[x∈B:u 1 (x)≤med B u 1 ] u. Now, observe that if u 1 (x) = med B u 1 then u 2 (x) ≤
sup[u 1 =med B u 1 ] u 2 . From this relation it follows that
sup[x∈B:u 1 (x)≤med B u 1 ] u ≤ q. Hence med∗B (u) ≤ q. To
prove the opposite inequality, observe that if b ∈ IR
then b ≥
is such that measure([u 1 ≤ b]) ≥ measure(B)
2
med B (u 1 ) and, in consequence, [u 1 ≤ med B (u 1 )] ⊆
[u 1 ≤ b]. Since by Lemma 1, q = sup[u 1 ≤med B (u 1 )] u,
we have q ≤ sup[u 1 ≤b] u for all b ∈ IR such that
. Hence q ≤ med∗B (u).
measure([u 1 ≤ b]) ≥ measure(B)
2
2
Observe that the infimum in (15) is attained.
3.2.
Asymptotic Behavior and “Coupled
Geometric PDE’s”
(14)
B ∈H x∈B 0
Proof of Proposition 4:
113
¶
.
Since π1 (med B u) = π1 (med∗B u) = med B u 1 , then
Theorem 1 describes the asymptotic behavior of the
first coordinate of the vector median med D(x0 ,t) u for
a three times differentiable function u : IR2 → IR2 ,
D(x0 , t) being the disk of radius t > 0 centered at
x0 ∈ IR2 . Let us describe the asymptotic behavior of the
second coordinate of med D(x0 ,t) u. The formula will be
4
written explicitly only when Du 2 (x0 ) · Du ⊥
1 (x 0 ) 6= 0.
⊥
If Du 2 (x0 ) · Du 1 (x0 ) = 0, the formula can be written using the Taylor expansion of u 2 given in the next
Proposition.
Proposition 5. Let u : IR2 → IR2 be three times
differentiable, u = (u 1 , u 2 ). Assume that Du 1 (x0 ) 6= 0.
Then
u 2 (x) = u 2 (x0 ) + hx − x0 , e1 iDu 2 (x0 ) · e1
1
+ t 2 κ(u 1 )(x0 )Du 2 (x0 ) · e2
6
1
− κ(u 1 )(x0 )hx − x0 , e1 i2 Du 2 (x0 ) · e2
2
1
+ hD 2 u 2 (x0 )e1 , e1 ihx − x0 , e1 i2 + o(t 2 ),
2
(17)
for x ∈ [u 1 = med D(x0 ,t) u 1 ] ∩ D(x0 , t), where e1 =
Du ⊥
Du 1 (x0 )
1 (x 0 )
, e2 = |Du
, such that {e1 , e2 } is positive ori|Du 1 (x0 )|
1 (x 0 )|
ented. In particular, if Du 2 (x0 ) · Du ⊥
1 (x 0 ) 6= 0, then
¢
¡
π2 med D(x0 ,t) u = sup[u 1 =med D(x ,t) u 1 ]∩D(x0 ,t) u 2
0
¯
¯
¯
¯
Du ⊥
1 (x 0 ) ¯
¯
= u 2 (x0 ) − t ¯ Du 2 (x0 ) ·
|Du 1 (x0 )| ¯
+ O(t 2 ).
(18)
114
Caselles, Sapiro and Chung
Similarly,
¢
¡
π2 med∗D(x0 ,t) u = inf[u 1 =med D(x0 ,t) u 1 ]∩D(x0 ,t) u 2
¯
¯
¯
¯
Du ⊥
1 (x 0 ) ¯
¯
= u 2 (x0 ) + t ¯ Du 2 (x0 ) ·
|Du 1 (x0 )| ¯
+ O(t 2 ).
The value of the corresponding infimum or supremum
depend on the sign of the terms containing hx − x0 , e1 i2
and we shall not write them explicitly.
Proof:
ously
Let x ∈ [u 1 = med D(x0 ,t) u 1 ] ∩ D(x0 , t). Obvi-
(19)
Before giving the proof of this proposition let
us observe that the above formula for u 2 coincides
with the asymptotic expansion (4) if u 2 = u 1 . Indeed, if x ∈ [u 1 = med D(x0 ,t) u 1 ] ∩ D(x0 , t), using that
Du 2 (x0 ) · e1 = 0 and Du 2 (x0 ) · e2 = |Du 1 (x0 )|, we
have
1
u 2 (x) = u 2 (x0 ) + t 2 κ(u 1 )(x0 )|Du 1 (x0 )|
6
1
− κ(u 1 )(x0 )hx − x0 , e1 i2 |Du 1 (x0 )|
2
1 2
+ hD u 2 (x0 )e1 , e1 ihx − x0 , e1 i2 + o(t 2 ).
2
Since u 2 = u 1 and |Du 1 (x0 )|κ(u 1 )(x0 ) = hD 2 u 2 (x0 )
e1 , e1 i, the last two terms in the above expression cancel each other and we have
u 2 (x) = u 1 (x) = med D(x0 ,t) u 1
1
= u 2 (x0 ) + t 2 κ(u 1 )(x0 )|Du 1 (x0 )| + o(t 2 ),
6
expression consistent with (4).
More generally, if Du 2 (x0 ) · e1 = 0, we may write
1
u 2 (x) = u 2 (x0 ) + t 2 κ(u 1 )(x0 )|Du 1 (x0 )|
6
1
− κ(u 1 )(x0 )hx − x0 , e1 i2 Du 2 (x0 ) · e2
2
1
+ hD 2 u 2 (x0 )e1 , e1 ihx − x0 , e1 i2 + o(t 2 ).
2
for x ∈ [u 1 = med D(x0 ,t) u 1 ] ∩ D(x0 , t). Now, observe that Du 2 (x0 ) · e2 = ±|Du 2 (x0 |. In particular, if
Du 2 (x0 )
is colinear
Du 2 (x0 )·e2 = |Du 2 (x0 | 6= 0, since |Du
2 (x 0 )|
to e1 , the expression for u 2 (x) can be reduced to
1
u 2 (x) = u 2 (x0 ) + t 2 κ(u 1 )(x0 )|Du 1 (x0 )|
6
1
+ |Du 2 (x0 )|(κ(u 2 )(x0 ) − κ(u 1 )(x0 ))
2
× hx − x0 , e1 i2 + o(t 2 ).
x − x0 = hx − x0 , e1 ie1 + hx − x0 , e2 ie2 .
To compute hx − x0 , e2 i we expand u 1 in Taylor series
up to the second order and write the identity u 1 (x) =
med D(x0 ,t) u 1 as
u 1 (x0 ) + hDu 1 (x0 ), x − x0 i
1
+ hD 2 u 1 (x0 )(x − x0 ), x − x0 i + o(t 2 )
2
= med D(x0 ,t) u 1 .
Using (4), we have
hDu 1 (x0 ), x − x0 i
1
= t 2 κ(u 1 )(x0 )|Du 1 (x0 )|
6
1
− hD 2 u 1 (x0 )(x − x0 ), x − x0 i + o(t 2 ).
2
Thus, we may write
1
x − x0 = hx − x0 , e1 ie1 + t 2 κ(u 1 )(x0 )e2
6
hD 2 u 1 (x0 )(x − x0 ), x − x0 i
e2 + o(t 2 ).
−
2|Du 1 (x0 )|
(20)
Introducing this expression for x − x0 in the right hand
side of (20) we obtain
1
x − x 0 = hx − x0 , e1 ie1 + t 2 κ(u 1 )(x0 )e2
6
hD 2 u 1 (x0 )e1 , e1 i
−
hx − x0 , e1 i2 e2 + o(t 2 ),
2|Du 1 (x0 )|
expression which can be written as
1
x − x 0 =hx − x0 , e1 ie1 + t 2 κ(u 1 )(x0 )e2
6
1
− κ(u 1 )(x0 )hx − x0 , e1 i2 e2 + o(t 2 ),
2
(21)
Vector Median Filters
u 1 (x0 )e1 ,e1 i
since κ(u 1 )(x0 ) = hD |Du
. Introducing this in
1 (x 0 )|
the Taylor expansion of u 2 ,
2
u 2 (x) = u 2 (x0 ) + hDu 2 (x0 ), x − x0 i
1
+ hD 2 u 2 (x0 )(x − x0 ), x − x0 i + o(t 2 )
2
we obtain (17), the first part of the proposition.
Now, from (21) we have for x ∈ [u 1 = med D(x0 ,t) u 1 ]
∩ D(x0 , t),
x − x0 = hx − x0 , e1 ie1 + O(t 2 ).
In particular, the curve [u 1 = med D(x0 ,t) u 1 ] ∩ D(x0 , t)
intersects the axis e2 at some point x̄ such that x̄ − x0 =
O(t 2 ). We also deduce that
sup... hx − x0 , e1 i = sup... |x − x0 | + O(t 2 ) = t + O(t 2 )
where the sup’s are taken in [u 1 = med D(x0 ,t) u 1 ] ∩
D(x0 , t). Taking the supremum of the last expression
for u 2 on [u 1 = med D(x0 ,t) u 1 ]∩ D(x0 , t) we obtain (19).
In a similar way we deduce (18).
2
gives it a very intuitive interpretation. The next terms
in the Taylor expansion of u 2 depend on curvatures of
u 1 and u 2 . These terms play a role when the previous
one is zero, and, in particular, this will happen when the
level sets of both components of the vector are equal.
∗
u can
The precise form of med D(x0 ,t) u and med D(x
0 ,t)
be deduced from the asymptotic expansion for u 2 previous to the proof of the last proposition and we shall
not write them explicitly. Let us only mention that if
u 2 = u 1 on a neighborhood of a point, then u 2 moves
with curvature motion, as expected.
To illustrate the case when Du 1 (x0 ) = 0 and x0 is
non-degenerate, we first assume that x0 = (0, 0) and
u 1 (x1 , x2 ) = Ax12 + Bx22 , x = (x1 , x2 ) ∈ IR2 , A, B > 0.
It is immediate to compute
½
med D(x0 ,t) u 1 = inf α ∈ IR : measure([u 1 ≤ α])
¾
measure(D(x0 , t)
≥
2
√
2
t AB
.
=
2
2
In analogy to the scalar case, this result can also lead
to deduce the following result: Modulo the different
scales of the two coordinates, when the median filter
is iterated, and t → 0, the second component of the
vector, u 2 (x), is moving its level-sets to follow those
of the first component u 1 (x),5 which are by themself
moving with curvature motion. This is expressed with
the equation
¯
¯
¯ Du 2 (x, t) Du ⊥
¯
∂u 2 (x, t)
1 (x, t) ¯
¯
=±¯
·
|Du 2 (x, t)|,
∂t
|Du 2 (x, t)| |Du 1 (x, t)| ¯
(22)
where the sign depends on the exact definition of the
median being used. Deriving this equation from the
asymptotic result presented above is much more complicated than in the scalar case, and this is beyond
the scope of this paper. In spite of the very attractive
notation, this is not a well-defined partial differential
equation since the right hand side is not defined when
Du 1 (x) = 0 (see remark below), and certainly this happens in images (and in those which are solutions of
mean curvature motion). Note that Du 1 (x) = 0 means
that there is no level-set direction at that place, and
then the level-sets of u 2 have “nothing to follow.” This
equation clarifies the meaning of the vector median and
115
√
t AB
The set X ≡
q [u 1 = q2 ] ∩ D(x0 , t) = [((x1 , x2 )
2
∈ D(x0 , t) : BA x12 + BA x22 = t2 ]. Again, it is
straightforward to obtain
sup X u 2 (x1 , x2 )
t
= u 2 (0, 0) + √
Ãr
2
A 2
u +
B 2x
r
B 2
u
A 2y
!1/2
+ o(t).
Consider now the case where u 1 is constant in a neighborhood of x0 . Suppose that u 1 = α in D(x0 , t). Then
either using (2) or (3) we conclude that
µ
med D(x0 ,t) u =
α
sup D(x0 ,t) u 2 (x)
¶
.
In this case,
sup u 2 (x) = u 2 (x0 ) + t|Du 2 (x0 )| + o(t).
D(x0 ,t)
We conclude that there is no common simple expression for all cases. Therefore, in contrast with the scalar
case, the asymptotic behavior of the median filter when
the gradient of the first component, u 1 (x), is zero is not
uniquely defined and decisions need to be taken when
the equation is implemented (see below).
116
Caselles, Sapiro and Chung
3.2.1. Projected Mean Curvature Motion. If we set
aside for a moment the requirement for an inf-sup morphological operator, and start directly from the definitions in (12) and (15) (instead of (9) and (14)), we
obtain an interesting alternative to the median filtering
of vector-valued images (we once again consider only
two dimensional vectors):
Using Lemma 1 it is possible to show that med∗∗ , as
defined in (23), is also a morphological inf-sup operation of the type of (9) and (14). This time, the set over
which the inf-sup operations are taken is given by
½
R := λ = (λ1 , λ2 ) : measure([u 1 ≤ λ1 ])
measure(B)
,
2
[u 1 ≤ λ1 ] 6= ∅, measure([u 1 = λ1 , u 2 ≤ λ2 ])
¾
measure([u 1 = λ1 ] ∩ B)
.
≥
2
≥
Definition 5. Let u = (u 1 , u 2 ) : IR2 → IR2 be a continuous function and B ⊆ IR2 a compact subset. Define
med∗∗
B (u)
µ
:=
med B u 1
med[x∈B:u 1 (x)=med B u 1 ] u 2 (x)
¶
(23)
In contrast with the previous definitions, we here
considered also the median of the second component,
restricted to the positions where the first component
achieved its own median value.
The asymptotic expansion (17) in Proposition 5 is
of course general. Replacing sup by median at the end
of the proof we obtain that the expression analogous to
(18) and (19) for med∗∗ is
¢
¡
π2 med∗∗
D(x0 ,t) u
¶
µ
Du 1 (x, t)
= u 2 (x0 ) + t 2 Du 2 (x, t) · κu 1 (x, t)
|Du 1 (x, t)|
+ o(t 2 ).
(24)
Note that the time scale of this expression is t 2 , as in
the scalar case (Theorem 1), and then as in the asymptotic expansion of the first component of the vector.
This is in contrast with a time scale of t for the expressions having inf or sup in the second component
(equations (18) and (19)).
The PDE corresponding to the expression above, and
therefore to the second component of the vector, is
µ
¶
Du 1 (x, t)
Du 2 (x, t)
∂u 2 (x, t)
= κu 1 (x, t)
·
∂t
|Du 1 (x, t)| |Du 2 (x, t)|
× |Du 2 (x, t)|.
(25)
This equation shows that the level-sets of the second
component u 2 are moving with the same geometric velocity as those of the first one, u 1 , meaning mean curvature motion (the projection reflects the well known fact
that tangential velocities do not affect the geometry of
the motion). Under certain smoothness assumptions,
short term existence of this flow can be derived from
the results in [3].
Recapping, the second component of the vector can
be obtained via inf, sup, or median operations over
a restricted set. In all the cases, the filter is a morphological inf-sup operation, computed over different
structuring elements sets, and in all the cases a corresponding asymptotic behavior and PDE interpretation
can be given. In the case of the median operation, the
asymptotic expansion of the vector components have
all the same scale, and the second component levelsets are just moving with the geometric velocity of the
first component ones. In the other cases, the level-sets
of the second component move toward those of the
first component. In all the cases then, the level-sets of
the second component follow those of the first one, as
expected from a lexicographic order. Figure 1 shows
an example of the theoretical results presented in this
paper.
4.
Concluding Remarks
In this paper, we have extended to vector-valued images
the relation between median filtering, inf-sup morphological operators, and PDE’s based interpretations. In
order to obtain the results here reported, we have assumed a lexicographic order that permits to compare
between vectors (in addition to assuming a coupling
of the channels, which is common in the literature).
If we do not want to use this assumption, we will not
have an order, and then an infimum-supremum type
of operation. Therefore, both the positive and negative
results reported in this paper are a direct consequence
of imposing and order in IR N . In order to “avoid” this,
we need to follow a different approach to compute the
median filter, for example, Eq. (1). We should note that
for continuous signals, minimizing the L 1 norm of a
vector is equivalent to the independent minimization of
Vector Median Filters
117
Figure 1. Examples of the theoretical results presented in this paper. The original image is on the top left. The top right shows the result of
alternating (12) and (15) for 1 step with a 3 × 3 discrete support (since these equations correspond to erosion and dilation respectively, alternating
them constitutes an opening filter). The bottom figures show results of the vectorial PDE derived from the mean curvature motion for the first
component and projected mean curvature motion for the rest (after 2 and 20 iterations respectively). All computations were performed on the
Lab color space. (Images reproduced here without color).
each one of its components, reducing then the problem
to the scalar case, where, for example, each plane is independently enhanced via mean curvature motion, see
Section 2.6 Therefore, in order to have equations that
are coupled, we need to look for a different approach,
like the one presented in this paper. Inspired by the work
on median filtering of angles and directions, in [12–14]
we propose a different alternative based on minimizing
the norms of the gradient of the chromaticity vectors,
following the theory of harmonic maps.
In addition to the study of direction diffusion, the theory introduced in this paper leads to another interesting
flow:
µ
¶
∇u
∂u(x, t)
=
· vE(x, t) k∇uk,
∂t
k∇uk
where u : IR2 → IR is the deforming image and vE(x, t)
is a given vector field. This flow is inspired on Eq. (22),
but, since there is no absolute value, when the regularity
of the vector field can be controlled, the equation can
be well-defined. This PDE is basically deforming the
level-sets to follow certain direction. The theoretical
and practical results regarding this flow will be reported
elsewhere.
Acknowledgments
GS thanks Prof. R. Kohn from the Courant Institute,
NYU, for motivating him to think again about filtering vectorial images. Part of this work was performed
while GS was visiting the University of Illes Balears.
This work was partially supported by the Spanish
118
Caselles, Sapiro and Chung
DGICYT, Project PB94-1174, European Network
PAVR FMRXCT960036, the Office of Naval Research
ONR-N00014-97-1-0509, the Office of Naval Research Young Investigator Award to GS, the Presidential Early Career Awards for Scientists and Engineers
(PECASE) to GS, the National Science Foundation
CAREER Award to GS, by the National Science
Foundation Learning and Intelligent Systems Program (LIS), and NSF-IRI-9306155 (Geometry Driven
Diffusion).
Notes
1. Although edges are not completely preserved with a median filter,
they are indeed much better preserved than with ordinary linear
filters.
∇u
2. κ = div( k∇uk
).
3. Lexicographic order has recently been used in vector-valued morphology as well; see [5] for the most recent published results.
4. Du := ∇u and hDu, Du ⊥ i = 0, while kDuk = kDu ⊥ k.
5. The Beltrami flow [7] also has the property that the level-sets tend
to follow each other [8].
6. In the classical discrete case, since the median belongs to the finite
set of vectors in the window, the vectorial case is not reduced to
a collection of scalar cases.
11. G. Sapiro and D. Ringach, “Anisotropic diffusion of multivalued
images with applications to color filtering,” IEEE Trans. Image
Processing Vol. 5, pp. 1582–1586, 1996.
12. B. Tang, G. Sapiro, and V. Caselles, “Direction diffusion,” ECE
Department Technical Report, University of Minnesota, Feb.
1999.
13. B. Tang, G. Sapiro, and V. Caselles, “Direction diffusion,” in
Proc. Int. Conference Comp. Vision, Greece, Sept. 1999.
14. B. Tang, G. Sapiro, and V. Caselles, “Color image enhancement
via chromaticity diffusion,” ECE Department Technical Report,
University of Minnesota, March 1999.
15. P.E. Trahanias and A.N. Venetsanopoulos, “Vector directional
filters—A new class of multichannel image processing filters,” IEEE Trans. Image Processing, Vol. 2, pp. 528–534,
1993.
16. P.E. Trahanias, D. Karakos, and A.N. Venetsanopoulos, “Directional processing of color images: Theory and experimental
results,” IEEE Trans. Image Processing, Vol. 5, pp. 868–880,
1996.
17. R.T. Whitaker and G. Gerig, “Vector-valued diffusion,” in Geometry Driven Diffusion in Computer Vision, B. ter Haar Romeny
(Ed.), Kluwer: Boston, MA, 1994.
References
1. L. Alvarez, P.L. Lions, and J.M. Morel, “Image selective smoothing and edge detection by nonlinear diffusion,” SIAM J. Numer.
Anal., Vol. 29, pp. 845–866, 1992.
2. V. Caselles, J.M. Morel, G. Sapiro, and A. Tannenbaum (Eds.),
“Special issue on partial differential equations and geometrydriven diffusion in image processing and analysis,” IEEE Trans.
Image Processing, Vol. 7, pp. 269–273, 1998.
3. L.C. Evans and J. Spruck, “Motion of level-sets by mean curvature II,” in Trans. American Mathematical Society, Vol. 30,
No. 1, pp. 321–332, 1992.
4. F. Guichard and J.M. Morel, Introduction to Partial Differential
Equations in Image Processing. Tutorial Notes, IEEE Int. Conf.
Image Proc., Washington, DC, Oct. 1995.
5. H.J.A.M. Heijmans and J.B.T.M. Roerdink (Eds.), Mathematical Morphology and Its Applications to Image and Signal Processing, Kluwer: Dordrecht, The Netherlands, 1998.
6. D.G. Karakos and P.E. Trahanias, “Generalized multichannel image-filtering structures,” IEEE Trans. Image Processing,
Vol. 6, pp. 1038–1045, 1997.
7. R. Kimmel, R. Malladi, and N. Sochen, “Image processing via
the Beltrami operator,” in Proc. of 3rd Asian Conf. on Computer
Vision, Hong Kong, Jan. 8–11, 1998.
8. R. Kimmel, Personal communication.
9. P. Maragos and R.W. Schafer, “Morphological systems for multidimensional image processing,” Proc. IEEE, Vol. 78, pp. 690–
710, 1990.
10. P. Perona and J. Malik, “Scale-space and edge detection using
anisotropic diffusion,” IEEE Trans. Pattern. Anal. Machine Intell., Vol. 12, pp. 629–639, 1990.
Vicent Caselles received the Licenciatura and Ph.D. degrees in mathematics from Valencia University, Spain, in 1982 and 1985, respectively. Currently, he is an associate professor at the University of Illes
Balears in Spain. He is an associate member of IEEE. His research
interests include image processing, computer vision, and the applications of geometry and partial differential equations to both previous
fields.
Guillermo Sapiro was born in Montevideo, Uruguay, on April 3,
1966. He received his B.Sc. (summa cum laude), M.Sc., and
Ph.D. from the Department of Electrical Engineering at the Technion,
Israel Institute of Technology, in 1989, 1991, and 1993 respectively.
After post-doctoral research at MIT, Dr. Sapiro became Member of
Technical Staff at the research facilities of HP Labs in Palo Alto,
California. He is currently with the Department of Electrical and
Computer Engineering at the University of Minnesota.
G. Sapiro works on differential geometry and geometric partial
differential equations, both in theory and applications in computer
Vector Median Filters
vision and image analysis. He recently co-edited a special issue of
IEEE Image Processing in this topic.
G. Sapiro was awarded the Gutwirth Scholarship for Special Excellence in Graduate Studies in 1991, the Ollendorff Fellowship for
Excellence in Vision and Image Understanding Work in 1992, the
Rothschild Fellowship for Post-Doctoral Studies in 1993, the Office
of Naval Research Young Investigator Award in 1998, the Presidential Early Career Awards for Scientist and Engineers (PECASE) in
1988, and the National Science Foundation Career Award in 1999.
G. Sapiro is a member of IEEE.
Do Hyun Chung received his BS in Eng. and MS in Eng. from
the Department of Control & Instrumentation Engineering, Seoul
119
National University, Seoul, Korea in 1994 and 1996 respectively,
Currently, he is a Ph.D. candidate at the Department of Electrical
& Computer Engineering, University of Minnesota, Minneapolis,
MN. His research interests include 3-D computer vision and partial
differential equation based image processing.