MATH 590: Meshfree Methods Chapter 3: Positive Definite Functions Greg Fasshauer Department of Applied Mathematics Illinois Institute of Technology Fall 2010 [email protected] MATH 590 – Chapter 3 1 Outline 1 Positive Definite Matrices and Functions 2 Integral Characterizations for (Strictly) Positive Definite Functions 3 Positive Definite Radial Functions [email protected] MATH 590 – Chapter 3 2 We know that the solution of the scattered data interpolation problem with RBFs and/or kernels amounts to solving a linear system Ac = y, where Ajk = ϕ(kx j − x k k2 ), j, k = 1, . . . , N. Linear algebra tells us that this system will have a unique solution whenever A is non-singular. Necessary and sufficient conditions to characterize this general non-singular case are still open. We will focus on basic functions ϕ that generate positive definite matrices. [email protected] MATH 590 – Chapter 3 3 Positive Definite Matrices and Functions Definition A real symmetric matrix A is called positive semi-definite if its associated quadratic form is non-negative, i.e., N N X X cj ck Ajk ≥ 0 (1) j=1 k =1 for c = [c1 , . . . , cN ]T ∈ RN . If the quadratic form (1) is zero only for c ≡ 0, then A is called positive definite. Since the eigenvalues of a positive definite matrix are all positive a positive definite matrix is non-singular. We therefore look for basis functions Bk that generate a positive definite interpolation matrix. This leads to positive definite functions. [email protected] MATH 590 – Chapter 3 5 Positive Definite Matrices and Functions Definition A complex-valued continuous function Φ : Rs → C is called positive definite on Rs if N X N X cj ck Φ(x j − x k ) ≥ 0 (2) j=1 k =1 for any N pairwise different points x 1 , . . . , x N ∈ Rs , and c = [c1 , . . . , cN ]T ∈ CN . The function Φ is called strictly positive definite on Rs if the quadratic form (2) is zero only for c ≡ 0. Remark For theoretical reasons the definition employs complex-valued functions. All of our applications will only be real-valued. Because of the inequality in (2) we know that any (strictly) positive definite function must have a real-valued quadratic form. [email protected] MATH 590 – Chapter 3 6 Positive Definite Matrices and Functions Remark History (i.e., analysts in the 1920s and 30s) defined positive definite functions in analogy to positive semi-definite matrices. This concept is not strong enough to guarantee a non-singular interpolation matrix and so we need to define strictly positive definite functions as in [Micchelli (1986)]. This leads to an unfortunate difference in terminology used in the context of matrices and functions. Sometimes the literature is confusing about this and you might find papers that use the term positive definite function in the strict sense, and others that use it in the way we defined it above. Hopefully, the authors are clear about their use of terminology. [email protected] MATH 590 – Chapter 3 7 Positive Definite Matrices and Functions Example The function ΦB (x) = eix·y , y ∈ Rs fixed, is positive definite on Rs since its quadratic form is N X N X cj ck ΦB (x j − x k ) = j=1 k =1 N X N X cj ck ei(x j −x k )·y j=1 k =1 = N X j=1 cj eix j ·y N X ck e−ix k ·y k =1 2 X N ix j ·y = cj e ≥ 0. j=1 [email protected] MATH 590 – Chapter 3 8 Positive Definite Matrices and Functions Definition 2 suggests that we should use strictly positive definite functions as basis functions for our scattered data interpolation problem, i.e., Pf (x) = N X ck Φ(x − x k ), x ∈ Rs . (3) k =1 Remark At this point we don’t require Φ to be radial. In fact, the interpolant obtained via Pf of (3) is translation invariant. We will later specialize to strictly positive definite functions that are also radial on Rs . One can also generalize this setup by introducing (strictly) positive definite kernels (a bit more later). Then we may even give up translation invariance. [email protected] MATH 590 – Chapter 3 9 Positive Definite Matrices and Functions Theorem (Properties of (strictly) positive definite functions) (1) Non-negative finite linear combinations of positive definite functions are positive definite. If Φ1 , . . . , Φn are positive definite on Rs and cj ≥ 0, j = 1, . . . , n, then Φ(x) = n X cj Φj (x), x ∈ Rs , j=1 is also positive definite. Moreover, if at least one of the Φj is strictly positive definite and the corresponding cj > 0, then Φ is strictly positive definite. (2) Φ(0) ≥ 0. (3) Φ(−x) = Φ(x). (4) Any positive definite function is bounded. In fact, |Φ(x)| ≤ Φ(0). (5) If Φ is positive definite with Φ(0) = 0 then Φ ≡ 0. (6) The product of (strictly) positive definite functions is (strictly) positive definite. [email protected] MATH 590 – Chapter 3 10 Positive Definite Matrices and Functions Proof. Properties (1) and (2) follow immediately from Definition 2. Property (5) follows immediately from (4). Property (6) follows from Schur’s theorem in linear algebra which ensures that the elementwise (or Hadamard) product of positive (semi-)definite matrices is positive (semi-)definite. We will now give details for Properties (3) and (4). For more details see [Cheney and Light (1999)] or [Wendland (2005a)]. [email protected] MATH 590 – Chapter 3 11 Positive Definite Matrices and Functions Proof (cont.) To show (3) we let N = 2, x 1 = 0, x 2 = x, and choose c1 = 1 and c2 = c. Then we have the quadratic form 2 2 X X cj ck Φ(x j − x k ) = (1 + |c|2 )Φ(0) + cΦ(x) + cΦ(−x) ≥ 0 j=1 k =1 for every c ∈ C. Now we can take c = 1 and c = i. These, respectively, imply that both Φ(x) + Φ(−x) and i (Φ(x) − Φ(−x)) must be real. This, however, is only possible if Φ(−x) = Φ(x). [email protected] MATH 590 – Chapter 3 12 Positive Definite Matrices and Functions Proof (cont.) For the proof of (4) we let N = 2, x 1 = 0, x 2 = x, and choose c1 = |Φ(x)| and c2 = −Φ(x). Then we get the quadratic form 2 X 2 X cj ck Φ(x j − x k ) = 2Φ(0)|Φ(x)|2 − Φ(−x)Φ(x)|Φ(x)| − Φ2 (x)|Φ(x)| j=1 k =1 ≥ 0. Since Φ(−x) = Φ(x) by Property (3), this gives 2Φ(0)|Φ(x)|2 − 2|Φ(x)|3 ≥ 0. If |Φ(x)| > 0, we divide by |Φ(x)|2 and get |Φ(x)| ≤ Φ(0) as claimed. If |Φ(x)| ≡ 0, then the statement holds trivially. [email protected] MATH 590 – Chapter 3 13 Positive Definite Matrices and Functions Example The cosine function is positive definite on R. First we remember that for any x ∈ R cos x = 1 ix e + e−ix . 2 By our earlier example the function ΦB (x) = eix·y is positive definite for any fixed y ∈ Rs . Therefore, Φ1 (x) = eix and Φ2 (x) = e−ix are positive definite, and the cosine function is positive definite by Property (1). [email protected] MATH 590 – Chapter 3 14 Positive Definite Matrices and Functions Property (3) shows that any real-valued (strictly) positive definite function has to be even. We even have Theorem A real-valued continuous function Φ is positive definite on Rs if and only if it is even and N X N X cj ck Φ(x j − x k ) ≥ 0 (4) j=1 k =1 for any N pairwise different points x 1 , . . . , x N ∈ Rs , and c = [c1 , . . . , cN ]T ∈ RN . The function Φ is strictly positive definite on Rs if the quadratic form (4) is zero only for c ≡ 0. See [Wendland (2005a)] for a proof. [email protected] MATH 590 – Chapter 3 15 Integral Characterizations for (Strictly) Positive Definite Functions We summarize some facts about integral characterizations of positive definite functions. They were established in the 1930s by Bochner and Schoenberg. We will also mention more recent extensions to strictly positive definite functions that are needed for the scattered data interpolation problem. Many more details in [Wendland (2005a)]. We start with a very brief discussion of concepts from measure theory and integral transforms that appear in the results later on. [email protected] MATH 590 – Chapter 3 17 Integral Characterizations for (Strictly) Positive Definite Functions Some Important Concepts from Measure Theory Bochner’s theorem below is formulated in terms of Borel measures. Definition Let X be an arbitrary set and denote by P(x) the set of all subsets of X . A subset A of P(X ) is called a σ-algebra on X if (1) X ∈ A, (2) A ∈ A implies that its complement (in X ) is also contained in A, (3) Ai ∈ A, i ∈ N, implies that the union of these sets belongs to A. Definition Given an arbitrary set X and a σ-algebra A of subsets of X , a measure on A is a function µ : A → [0, ∞] such that (1) µ(∅) = 0, (2) for any sequence {Ai } of disjoint sets in A we have µ( ∞ [ i=1 [email protected] Ai ) = ∞ X µ(Ai ). i=1 MATH 590 – Chapter 3 18 Integral Characterizations for (Strictly) Positive Definite Functions Some Important Concepts from Measure Theory Definition If X is a topological space, and O is the collection of open sets in X , then the σ-algebra generated by O is called the Borel σ-algebra and denoted by B(X ). If in addition X is a Hausdorff spacea , then a measure µ defined on B(X ) that satisfies µ(K ) < ∞ for all compact sets K ⊆ X is called a Borel measure. The carrier of a Borel measure is given by the set X \ {O : O ∈ O and µ(O) = 0}. a X is a Hausdorff space if any two distinct points of X can be separated by open sets [email protected] MATH 590 – Chapter 3 19 Integral Characterizations for (Strictly) Positive Definite Functions A few important integral transforms Definition The Fourier transform of f ∈ L1 (Rs ) is given by Z 1 f̂ (ω) = p f (x)e−iω·x dx, (2π)s Rs and its inverse Fourier transform is given by Z 1 f (ω)eix·ω dω, f̌ (x) = p s s (2π) R ω ∈ Rs , x ∈ Rs . This definition from [Rudin (1973)] is based on angular frequency and is used in [Wendland (2005a), Schölkopf and Smola (2002)]. Another, just as common, definition from [Stein and Weiss (1971)] uses ordinary frequency, i.e., Z p f̂SW (ω) = f (x)e−2πiω·x dx = (2π)s f̂ (2πω). Rs It appears in [Buhmann (2003), Cheney and Light (1999)]. [email protected] MATH 590 – Chapter 3 20 Integral Characterizations for (Strictly) Positive Definite Functions A few important integral transforms Similarly, we can define the Fourier transform of a finite (signed) measure µ on Rs by Z 1 µ̂(ω) = p e−iω·x dµ(x), ω ∈ Rs . (2π)s Rs [email protected] MATH 590 – Chapter 3 21 Integral Characterizations for (Strictly) Positive Definite Functions A few important integral transforms Since we are mostly interested in positive definite radial functions, we note that the Fourier transform of a radial function is again radial: Theorem Let Φ ∈ L1 (Rs ) be continuous and radial, i.e., Φ(x) = ϕ(kxk). Then its Fourier transform Φ̂ is also radial, i.e., Φ̂(ω) = Fs ϕ(kωk) with Z ∞ s 1 Fs ϕ(r ) = √ ϕ(t)t 2 J s−2 (rt)dt, 2 r s−2 0 where J s−2 is the classical Bessel function of the first kind of order 2 s−2 2 . Proof in [Wendland (2005a)]. This transform is also called Fourier-Bessel transform or Hankel transform. [email protected] MATH 590 – Chapter 3 22 Integral Characterizations for (Strictly) Positive Definite Functions A few important integral transforms Remark The Hankel inversion theorem [Sneddon (1972)] ensures that the Fourier transform for radial functions is its own inverse, i.e., for radial functions ϕ we have Fs [Fs ϕ] = ϕ. [email protected] MATH 590 – Chapter 3 23 Integral Characterizations for (Strictly) Positive Definite Functions Bochner’s Theorem One of the most celebrated results on positive definite functions is their characterization in terms of Fourier transforms established by Bochner in 1932 (for s = 1) and 1933 (for general s). Theorem (Bochner) A (complex-valued) function Φ ∈ C(Rs ) is positive definite on Rs if and only if it is the Fourier transform of a finite non-negative Borel measure µ on Rs , i.e., Z 1 Φ(x) = µ̂(x) = p e−ix·y dµ(y), x ∈ Rs . (2π)s Rs [email protected] MATH 590 – Chapter 3 24 Integral Characterizations for (Strictly) Positive Definite Functions Bochner’s Theorem Proof. There are many proofs of this theorem. Bochner’s original proof can be found in [Bochner (1933)]. Other proofs can be found, e.g., in [Cuppens (1975)] or [Gel’fand and Vilenkin (1964)]. A proof using the Riesz representation theorem to interpret the Borel measure as a distribution, and then taking advantage of distributional Fourier transforms can be found in [Wendland (2005a)]. We will prove only the one (easy) direction that is important for the application to scattered data interpolation. [email protected] MATH 590 – Chapter 3 25 Integral Characterizations for (Strictly) Positive Definite Functions Bochner’s Theorem Proof (cont.) We assume Φ is the Fourier transform of a finite non-negative Borel measure and show Φ is positive definite. Thus, N N X X 1 N N X X Z −i(x j −x k )·y cj ck Φ(x j − x k ) = p cj ck e dµ(y) (2π)s j=1 k =1 Rs Z N N X X 1 = p cj e−ix j ·y ck eix k ·y dµ(y) s s (2π) R j=1 k =1 2 Z X N 1 −ix j ·y = p cj e dµ(y) ≥ 0. s (2π) Rs j=1 j=1 k =1 The last inequality holds because of the conditions imposed on the measure µ. [email protected] MATH 590 – Chapter 3 26 Integral Characterizations for (Strictly) Positive Definite Functions Bochner’s Theorem Remark Bochner’s theorem shows that for any fixed y ∈ Rs the function ΦB (x) = eix·y , x ∈ Rs , of our earlier example can be considered as the fundamental positive definite function since all other positive definite functions are obtained as (infinite) linear combinations of this function. While Property (1) of Theorem 4 implies that linear combinations of ΦB will again be positive definite, the remarkable content of Bochner’s Theorem is the fact that indeed all positive definite functions are generated by ΦB . [email protected] MATH 590 – Chapter 3 27 Integral Characterizations for (Strictly) Positive Definite Functions Extensions to Strictly Positive Definite Functions In order to guarantee a well-posed interpolation problem we have to extend (if possible) Bochner’s characterization to strictly positive definite functions. We give a sufficient condition for a function to be strictly positive definite on Rs . Theorem Let µ be a non-negative finite Borel measure on Rs whose carrier is a set of nonzero Lebesgue measure. Then the Fourier transform of µ is strictly positive definite on Rs . [email protected] MATH 590 – Chapter 3 28 Integral Characterizations for (Strictly) Positive Definite Functions Extensions to Strictly Positive Definite Functions Proof. As in the proof of Bochner’s theorem we have N N X X 1 N X N X Z −i(x j −x k )·y cj ck Φ(x j − x k ) = p e dµ(y) cj ck (2π)s j=1 k =1 Rs Z N N X X 1 cj e−ix j ·y = p ck eix k ·y dµ(y) s (2π) Rs j=1 k =1 2 Z X N 1 −ix j ·y = p cj e dµ(y) ≥ 0. (2π)s Rs j=1 j=1 k =1 Now let g(y) = N X cj e−ix j ·y , j=1 and assume that the points x j are all distinct and c 6= 0. [email protected] MATH 590 – Chapter 3 29 Integral Characterizations for (Strictly) Positive Definite Functions Extensions to Strictly Positive Definite Functions Proof (cont.) Since the x j are distinct the functions y 7→ e−ix j ·y are linearly independent, and so g 6= 0. Since g is an entire function its zero set, i.e., {y ∈ Rs : g(y) = 0} can have no accumulation point and therefore it has Lebesgue measure zero (see, e.g., [Cheney and Light (1999)]). Now, the only remaining way to make the inequality 1 p (2π)s 2 N Z X −ix ·y 1 j p |g(y)|2 dµ(y) ≥ 0 c e dµ(y) = j s s s (2π) R R j=1 Z an equality is if the carrier of µ is contained in the zero set of g, i.e., has Lebesgue measure zero. The hypothesis of the theorem does not allow this. [email protected] MATH 590 – Chapter 3 30 Integral Characterizations for (Strictly) Positive Definite Functions Extensions to Strictly Positive Definite Functions Remark Another sufficient condition is given in [zu Castell et al. (2005)]. A complete integral characterization of functions that are strictly positive definite on Rs is to my knowledge still missing. The case s = 1 is covered in [Chang (1996)]. Complete characterizations of strictly positive definite functions on compact spaces (such as spheres) do exist in the literature. [email protected] MATH 590 – Chapter 3 31 Integral Characterizations for (Strictly) Positive Definite Functions Extensions to Strictly Positive Definite Functions The following corollary gives us a way to construct strictly positive definite functions. Corollary Let f be a continuous non-negative function in L1 (Rs ) which is not identically zero. Then the Fourier transform of f is strictly positive definite on Rs . Proof. This is a special case of the previous theorem in which the measure µ has Lebesgue density f . Thus, we use the measure µ defined for any Borel set B by Z f (x)dx. µ(B) = B Then the carrier of µ is equal to the (closed) support of f . However, since f is non-negative and not identically equal to zero, its support has positive Lebesgue measure, and hence the Fourier transform of f is strictly positive definite by the preceding theorem. [email protected] MATH 590 – Chapter 3 32 Integral Characterizations for (Strictly) Positive Definite Functions Extensions to Strictly Positive Definite Functions A very useful criterion to check whether a given function is strictly positive definite is given by Theorem Let Φ be a continuous function in L1 (Rs ). Φ is strictly positive definite if and only if Φ is bounded and its Fourier transform is non-negative and not identically equal to zero. Proof in [Wendland (2005a)]. If Φ 6≡ 0 (which implies that then also Φ̂ 6≡ 0) we need to ensure only that Φ̂ be non-negative in order for Φ to be strictly positive definite. [email protected] MATH 590 – Chapter 3 33 Positive Definite Radial Functions We now focus on positive definite radial functions. Earlier we characterized (strictly) positive definite functions in terms of multivariate functions Φ. When working with radial functions Φ(x) = ϕ(kxk) it is convenient to call the univariate function ϕ a positive definite radial function. This is a small abuse of our terminology for positive definite functions, but commonly done in the literature. If follows immediately that Lemma If Φ = ϕ(k · k) is (strictly) positive definite and radial on Rs then Φ is also (strictly) positive definite and radial on Rσ for any σ ≤ s. [email protected] MATH 590 – Chapter 3 35 Positive Definite Radial Functions We now return to integral characterizations and begin with a theorem due to Schoenberg (see, e.g., [Schoenberg (1938a), p.816], or [Wells and Williams (1975), p.27]). Theorem A continuous function ϕ : [0, ∞) → R is positive definite and radial on Rs if and only if it is the Bessel transform of a finite non-negative Borel measure µ on [0, ∞), i.e., Z ∞ ϕ(r ) = Ωs (rt)dµ(t). 0 Here cos r Ωs (r ) = Γ s 2 for s = 1, s−2 2 2 r J s−2 (r ) for s ≥ 2, 2 and J s−2 is the classical Bessel function of the first kind of order 2 [email protected] MATH 590 – Chapter 3 s−2 2 . 36 Positive Definite Radial Functions Remark Now we can view the function Φ(x) = cos(x) as the fundamental positive definite radial function on R. We will see in the next chapter that the characterization of the previous theorem immediately suggests a class of (even strictly) positive definite radial functions. [email protected] MATH 590 – Chapter 3 37 Positive Definite Radial Functions A Fourier transform characterization of strictly positive definite radial functions on Rs can be found in [Wendland (2005a)]. This theorem is based on the Fourier transform formula of radial functions given earlier and the check for strictly positive definite functions. Theorem A continuous function ϕ : [0, ∞) → R such that r 7→ r s−1 ϕ(r ) ∈ L1 [0, ∞) is strictly positive definite and radial on Rs if and only if the s-dimensional Fourier transform Z ∞ s 1 Fs ϕ(r ) = √ ϕ(t)t 2 J(s−2)/2 (rt)dt s−2 r 0 is non-negative and not identically equal to zero. [email protected] MATH 590 – Chapter 3 38 Positive Definite Radial Functions Above we saw that any function that is (strictly) positive definite and radial on Rs is also (strictly) positive definite and radial on Rσ for any σ ≤ s. Therefore, we are interested in those functions which are (strictly) positive definite and radial on Rs for all s. The characterization for positive definite functions is from [Schoenberg (1938a), pp. 817–821] and the strictly positive definite case from [Micchelli (1986)]: Theorem (Schoenberg) A continuous function ϕ : [0, ∞) → R is strictly positive definite and radial on Rs for all s if and only if it is of the form Z ∞ 2 2 ϕ(r ) = e−r t dµ(t), 0 where µ is a finite non-negative Borel measure on [0, ∞) not concentrated at the origin. [email protected] MATH 590 – Chapter 3 39 Positive Definite Radial Functions Letting µ be a point evaluation measure in Schoenberg’s theorem shows us that the Gaussian ϕ(r ) = e−ε 2r 2 is strictly positive definite and radial on Rs for all s. Moreover, we can view the Gaussian as the fundamental member of the family of functions that are strictly positive definite and radial on Rs for all s. By Schoenberg’s theorem all such functions are given as infinite linear combinations of Gaussians. [email protected] MATH 590 – Chapter 3 40 Positive Definite Radial Functions Schoenberg’s theorem employs a finite non-negative Borel measure µ on [0, ∞) such that Z ∞ ϕ(r ) = e−r 2t 2 dµ(t). 0 If we want to find a zero r0 of ϕ then we have to solve Z ∞ 2 2 ϕ(r0 ) = e−r0 t dµ(t) = 0. 0 Since the exponential function is positive and the measure is non-negative, it follows that µ must be the zero measure. However, then ϕ is identically equal to zero. Therefore, a non-trivial function ϕ that is positive definite and radial on Rs for all s can have no zeros. [email protected] MATH 590 – Chapter 3 41 Positive Definite Radial Functions The discussion on the previous slide implies in particular that Theorem There are no oscillatory univariate continuous functions that are strictly positive definite and radial on Rs for all s. There are no compactly supported univariate continuous functions that are strictly positive definite and radial on Rs for all s. [email protected] MATH 590 – Chapter 3 42 Appendix References References I Buhmann, M. D. (2003). Radial Basis Functions: Theory and Implementations. Cambridge University Press. Cheney, E. W. and Light, W. A. (1999). A Course in Approximation Theory. Brooks/Cole (Pacific Grove, CA). Cuppens, R. (1975). Decomposition of Multivariate Probability. Academic Press (New York). Fasshauer, G. E. (2007). Meshfree Approximation Methods with M ATLAB. World Scientific Publishers. Gel’fand, I. M. and Vilenkin, N. Ya. (1964). Generalized Functions Vol. 4. Academic Press (New York). [email protected] MATH 590 – Chapter 3 43 Appendix References References II Iske, A. (2004). Multiresolution Methods in Scattered Data Modelling. Lecture Notes in Computational Science and Engineering 37, Springer Verlag (Berlin). Rudin, W. (1973). Functional Analysis. McGraw-Hill (New York). Schölkopf, B. and Smola, A. J. (2002). Learning with kernels: Support vector machines, regularization, optimization, and beyond. MIT Press, Cambridge, Massachussetts. Sneddon, I. H. (1972). The Use of Integral Transforms. McGraw-Hill (New York). [email protected] MATH 590 – Chapter 3 44 Appendix References References III Stein, E. M. and Weiss, G. (1971). Introduction to Fourier Analysis in Euclidean Spaces. Princeton University Press (Princeton). Wells, J. H. and Williams, R. L. (1975). Embeddings and Extensions in Analysis. Springer (Berlin). Wendland, H. (2005a). Scattered Data Approximation. Cambridge University Press (Cambridge). Bochner, S. (1933). Monotone Funktionen, Stieltjes Integrale und harmonische Analyse. Math. Ann. 108, pp. 378–410. zu Castell, W., Filbir, F. Szwarc, R. (2005). Strictly positive definite functions in Rd . J. Approx. Theory 137 2, pp. 277–280. [email protected] MATH 590 – Chapter 3 45 Appendix References References IV Chang, K. F. (1996). Strictly positive definite functions. J. Approx. Theory 87, pp. 148–158. Addendum: J. Approx. Theory 88 1997, p. 384. Micchelli, C. A. (1986). Interpolation of scattered data: distance matrices and conditionally positive definite functions. Constr. Approx. 2, pp. 11–22. Schoenberg, I. J. (1938a). Metric spaces and completely monotone functions. Ann. of Math. 39, pp. 811–841. [email protected] MATH 590 – Chapter 3 46
© Copyright 2026 Paperzz