Empirical Distribution and Beyond
Influence Function and Functional Delta Method
Empirical Distribution Function, Statistical
Functionals, and Influence Function
Tianchen Qian
Advanced Theory Class, Dec 9, 2014
Empirical Distribution and Beyond
Influence Function and Functional Delta Method
1
Empirical Distribution and Beyond
2
Influence Function and Functional Delta Method
Plug-in estimator
Functional Differentiation
Influence Function and Functional Delta Method
Empirical Distribution and Beyond
Influence Function and Functional Delta Method
Empirical Distribution Functions
Assume X1 , . . . , Xn are i.i.d. random variables with cdf F .
Their empirical distribution is defined as
F̂n (x ) =
n
1X
I (Xi ≤ x ) ,
n i=1
x ∈ R.
By WLLN and CLT, we have (for fixed x )
P
F̂n (x ) → F (x ) ,
o
√ n
D
n F̂n (x ) − F (x ) → N (0, F (x ) (1 − F (x ))) .
Empirical Distribution and Beyond
Influence Function and Functional Delta Method
(Pointwise) Confidence Interval
Based on
o
√ n
D
n F̂n (x ) − F (x ) → N (0, F (x ) (1 − F (x ))) ,
we can construct pointwise confidence interval C (x ) for
F (x ), for fixed x . I.e., we have
P {F (x ) ∈ C (x )} ≥ 1 − α.
Empirical Distribution and Beyond
Influence Function and Functional Delta Method
Confidence Band
To find a confidence band for F (·), we want to find C (·) such
that
P {F (x ) ∈ C (x ) ∀x ∈ R} ≥ 1 − α.
This is asking more than a pointwise confidence interval.
We need some control of F on the entire R.
Empirical Distribution and Beyond
Influence Function and Functional Delta Method
Confidence Band (continued)
Glivenko-Cantelli Theorem tells us that
a.s.
sup F̂n (x ) − F (x ) → 0,
as n → ∞.
x ∈R
A stronger result called the Dvoretzky-Kiefer-Wolfowitz
inequality, or DKW inequality:
(
P
)
2
sup F̂n (x ) − F (x ) > ≤ 2e −2n .
x ∈R
Note that this is a finite-sample result (holds for finite n).
Empirical Distribution and Beyond
Influence Function and Functional Delta Method
Confidence Band (continued)
(
P
)
2
sup F̂n (x ) − F (x ) > ≤ 2e −2n .
x ∈R
Setting the right hand side of the DKW inequality equal to α,
and we have
s
2
1
=
.
log
2n
α
So the confidence band for F is
s
F̂n (x ) −
1
2
log
, F̂n (x ) +
2n
α
s
!
1
2
log
2n
α
,
x ∈R
Empirical Distribution and Beyond
Influence Function and Functional Delta Method
Confidence Interval vs Confidence Band
http://web.as.uky.edu/statistics/users/pbreheny/621/F12/notes/823.pdf
Empirical Distribution and Beyond
Influence Function and Functional Delta Method
Other Confidence Band
There are other types of confidence bands, for example,
(
P
)
2
sup F̂n (x ) − F (x ) > ≤ 8 (n + 1) e −n /32 .
x ∈R
This is derived from Vapnik-Chervonenkis Theorem.
See Theorem 2.43 and Example 2.45 of
Wasserman, L. (2006). All of nonparametric statistics.
Springer.
Empirical Distribution and Beyond
Influence Function and Functional Delta Method
Plug-in estimator
1
Empirical Distribution and Beyond
2
Influence Function and Functional Delta Method
Plug-in estimator
Functional Differentiation
Influence Function and Functional Delta Method
Empirical Distribution and Beyond
Influence Function and Functional Delta Method
Plug-in estimator
Parameter as functionals
We can treat the parameter of interest θ as a functional of the
distribution function F :
θ = T (F ) .
Examples:
R
Mean: T (F ) = xdF (x )
R
2
Variance: T (F ) = (x − µ) dF (x )
−1
Quantiles: T (F ) = F (p)
Mann-Whitney functional:
R
T (F , G) = P (X ≤ Y | X ∼ F , Y ∼ G) = F (x ) dG (x )
Empirical Distribution and Beyond
Influence Function and Functional Delta Method
Plug-in estimator
Plug-in Principle
Calculate the quantity based on empirical distribution function
F̂ :
θ̂ = T F̂ .
Examples:
Mean: X̄n
2
P
Variance: n−1
Xi − X̄n
Quantile: Sample quantile
Mann-Whitney
Rstatistic:
T F̂m , Ĝn = F̂m (x ) d Ĝn (x ) =
1
mn
P
i,j
I (Xi ≤ Yj )
Empirical Distribution and Beyond
Influence Function and Functional Delta Method
Plug-in estimator
Is the plug-in estimator a good estimator?
A natural question: Is the plug-in estimator a good estimator?
a.s.
The Glivenko-Cantelli
Theorem says that F̂ → F ; does this
a.s.
mean that T F̂ → T (F )?
Answer: sometimes.
Empirical Distribution and Beyond
Influence Function and Functional Delta Method
Plug-in estimator
Plug-in density estimation
As an example where the plug-in estimator is not consistent,
consider density estimation, where T (F ) = F 0 (x0 ), for some
x0 .
http://web.as.uky.edu/statistics/users/pbreheny/621/F12/notes/823.pdf
Empirical Distribution and Beyond
Influence Function and Functional Delta Method
Plug-in estimator
When is the plug-in estimator consistent?
When is the plug-in estimator consistent?
R
Mean: T (F ) = xdF (x )
Density at x0 : T (F ) = F 0 (x0 )
Change F to (1 − )F + δx , how much does T (F ) change?
It requires certain conditions on the smoothness
(differentiability) of T (F ).
To address this more clearly, we need to expand our notion of
the derivative.
Empirical Distribution and Beyond
Influence Function and Functional Delta Method
Functional Differentiation
1
Empirical Distribution and Beyond
2
Influence Function and Functional Delta Method
Plug-in estimator
Functional Differentiation
Influence Function and Functional Delta Method
Empirical Distribution and Beyond
Influence Function and Functional Delta Method
Functional Differentiation
The Gateaux derivative
T {F } − T (F )
dT (F )
” = ” lim
→0
dF
The Gateaux derivative of T at F in the direction of G is
defined by
T {(1 − ) F + G} − T (F )
LF (T ; G) = lim
→0
An equivalent definition: let D = G − F , then
LF (T ; D) = lim
→0
T {F + D} − T (F )
Empirical Distribution and Beyond
Influence Function and Functional Delta Method
Functional Differentiation
Interpretation
T {(1 − ) F + G} − T (F )
LF (T ; G) = lim
→0
From a mathematical perspective, the Gateaux derivative is a
generalization of the concept of directional derivative to
functional analysis
From a statistical perspective, it represents the rate of change
in a statistical functional upon a small amount of
contamination by another distribution G.
Empirical Distribution and Beyond
Influence Function and Functional Delta Method
Functional Differentiation
Gateaux derivative of T (F ) = f (x0 )
As an example of how the Gateaux derivative works, suppose
F is CDF of a continuous variable, and G is the distribution
that places all of its mass at point x0 .
T {(1 − ) F + G} − T (F )
→0
"
#
d
d
dx {(1 − ) F (x ) + G (x )} |x =x0 − dx F (x ) |x =x0
= lim
→0
LF (T ; G) = lim
(1 − ) f (x0 ) + g (x0 ) − f (x0 )
= lim
→0
=∞
Empirical Distribution and Beyond
Influence Function and Functional Delta Method
Functional Differentiation
Glivenko-Cantelli does not imply convergence of estimators
So, even though F and F differ from each other only
infinitesimally, T (F ) and T (F ) differ from each other by an
infinite amount
Thus, the Glivenko-Cantelli
theorem does not
help us here:
a.s.
supx F̂ (x ) − F (x ) → 0 doesn’t imply T F̂ → T (F ).
Empirical Distribution and Beyond
Influence Function and Functional Delta Method
Functional Differentiation
Hadamard differentiability
It turns
out
that even Gateaux differentiability is not enough
for T F̂ → T (F ).
We need a stronger definition of differentiability.
A functional T is Hadamard differentiable if for a
D (= G − F ), there exists some LF (T ; D), such that for any
sequence n → 0 and Dn satisfying supx |Dn (x ) − D (x )| → 0,
we have
T (F + n Dn ) − T (F )
→ LF (T ; D)
n
Empirical Distribution and Beyond
Influence Function and Functional Delta Method
Functional Differentiation
Compare Gateaux and Hadamard differentiability
Gateaux:
T {F + D} − T (F )
→ LF (T ; D)
Hadamard:
T (F + n Dn ) − T (F )
→ LF (T ; D)
n
as long as n → 0 and supx |Dn (x ) − D (x )| → 0
Empirical Distribution and Beyond
Influence Function and Functional Delta Method
Functional Differentiation
a.s.
Sufficient conditions for T F̂ → T (F )
If
T is Hadamard differentiable;
Or, T is a bounded functional (at F ): ∃C > 0, such that
|T (F ) − T (G)| ≤ C sup |F (x ) − G (x )|
∀G
x
a.s.
Then T F̂ → T (F ).
Now we have consistency. What about asymptotic normality?
Functional Delta Method.
Empirical Distribution and Beyond
Influence Function and Functional Delta Method
Influence Function and Functional Delta Method
1
Empirical Distribution and Beyond
2
Influence Function and Functional Delta Method
Plug-in estimator
Functional Differentiation
Influence Function and Functional Delta Method
Empirical Distribution and Beyond
Influence Function and Functional Delta Method
Influence Function and Functional Delta Method
Influence function and empirical influence function
T {(1 − ) F + G} − T (F )
LF (T ; G) = lim
→0
The influence function of the functional T at point x is
defined as
LF (x ) ≡ LF (T ; δx ) = lim
→0
T {(1 − ) F + δx } − T (F )
,
where δx is a point mass at x .
A closely related concept is the empirical influence function:

L̂ (x ) = lim 
→0
n
o

T (1 − ) F̂ + δx − T F̂
.
Empirical Distribution and Beyond
Influence Function and Functional Delta Method
Influence Function and Functional Delta Method
Functional Delta Method (linear case)
First consider a simple case: T (F ) is a linear functional:
T (αF + RβG) = αT (F ) + βT (G). E.g.
T (F ) = h (x ) dF (x ) for some function h.
In this case,
T {(1 − ) F + δx } − T (F )
LF (x ) = lim
→0
(1 − ) T (F ) + T (δx ) − T (F )
= lim
→0
= T (δx ) − T (F )
so for any G,
Z
Z
LF (x ) dG (x ) =
Z
=
{T (δx ) − T (F )} dG (x )
T (δx ) dG (x ) −
= T (G) − T (F )
Z
T (F ) dG (x )
Empirical Distribution and Beyond
Influence Function and Functional Delta Method
Influence Function and Functional Delta Method
Functional Delta Method (linear case, continued)
Lemma 1: Assume T is a linear functional. For any G,
Z
LF (x ) dG (x ) = T (G) − T (F ) .
This is similar to the fundamental theorem of calculus.
Corollary:
Z
LF (x ) dF (x ) = 0
Empirical Distribution and Beyond
Influence Function and Functional Delta Method
Influence Function and Functional Delta Method
Functional Delta Method (linear case, continued)
Lemma
2: Assume T is a linear functional. Let
R
τ 2 = L2F (x ) dF (x ). If τ 2 < ∞, then
o
√ n D
n T F̂ − T (F ) → N 0, τ 2
Proof: By Lemma 1,
o √ Z
√ n n T F̂ − T (F ) = n LF (x ) d F̂ (x )
n
1 X
=√
LF (Xi )
n i=1
D
→ N 0, τ 2
Empirical Distribution and Beyond
Influence Function and Functional Delta Method
Influence Function and Functional Delta Method
Functional Delta Method (linear case, continued)
Lemma 3: Let τ̂ 2 =
1
n
P
i
L̂2 (Xi ). Then
P
τ̂ 2 → τ 2
Lemma 4: Assume T is a linear functional.
o
√ n n T F̂ − T (F ) D
→ N (0, 1)
τ̂ 2
Empirical Distribution and Beyond
Influence Function and Functional Delta Method
Influence Function and Functional Delta Method
Functional Delta Method (general case)
In the linear case, our result depends on the expression
n
o
√ n 1 X
n T F̂ − T (F ) = √
LF (Xi )
n i=1
We can prove the Delta Method for general T if we have
n
o
√ n 1 X
n T F̂ − T (F ) = √
LF (Xi ) + oP (1)
n i=1
This is a special case of the von Mises expansion, and the
oP (1) requires some restrictions on T .
Empirical Distribution and Beyond
Influence Function and Functional Delta Method
Influence Function and Functional Delta Method
An intuitive derivation
For distributions F and G, consider the map
φ : t 7→ T (F + tG). Assume the ordinary derivatives of φ (t)
at t = 0 exist for k = 1, 2, . . . , m. Denoting them by φ(k) (0).
By Taylor’s theorem, we have
φ (t) − φ (0) = tφ0 (0) + . . . +
1 m (m)
t φ (0) + o (t m )
m!
√
√ Substituting t = 1/ n and G = n F̂ − F , we obstain
1
T F̂ − T (F ) = √ φ0 (0) + . . .
n
To have control on the remainder term, some bounds need to
be established. See von Mises (1947).
Empirical Distribution and Beyond
Influence Function and Functional Delta Method
Influence Function and Functional Delta Method
Functional Delta Method (Theorem)
Theorem: If T is Hadamard differentiable at F , then
o
√ n n T F̂ − T (F )
τ̂ 2
where τ̂ 2 =
1
n
P
i
L̂2 (Xi ).
D
→ N (0, 1) ,
Empirical Distribution and Beyond
Influence Function and Functional Delta Method
Influence Function and Functional Delta Method
Take home message
For a functional T , its Gateaux derivative w.r.t a point mass
δx is the influence function for the plug-in estimator T F̂ :
n
o
√ n 1 X
√
n T F̂ − T (F ) =
LF (Xi ) + oP (1)
n i=1
The Delta Method
applies for plug-in estimators viewed as
functionals: T F̂ , and this requires some smoothness on the
functional T .
Empirical Distribution and Beyond
Influence Function and Functional Delta Method
Influence Function and Functional Delta Method
References for Functional Delta Method
Van der Vaart, A. W. (2000). Asymptotic statistics (Vol. 3).
Cambridge university press.
Chapter 20 for numerous examples
Section 20.1 on von Mises calculus (intuitive)
Section 20.2 on Hadamard derivatives
Theorem 20.8 on an even more general Delta Method (about
normed vector spaces)
Mises, R. V. (1947). On the asymptotic distribution of
differentiable statistical functions. The annals of
mathematical statistics, 309-348.
Empirical Distribution and Beyond
Influence Function and Functional Delta Method
Influence Function and Functional Delta Method
Bibliography
Van der Vaart, A. W. (2000). Asymptotic statistics (Vol. 3).
Cambridge university press. Chapter 20
Lecture notes by Jon Wellner:
http://www.stat.washington.edu/jaw/COURSES/580s/
581/LECTNOTES/ch7.pdf
Wasserman, L. (2006). All of nonparametric statistics.
Springer. Chapter 2, 3.1
Lecture notes by Patrick Breheny:
http://web.as.uky.edu/statistics/users/pbreheny/
621/F12/notes.html
Gill, R. D., Wellner, J. A., & Præstgaard, J. (1989 & 1993).
Non-and semi-parametric maximum likelihood estimators and
the von mises method. Scandinavian Journal of Statistics,
97-128.