Estimation of Mutual Information: A Survey
Janett Walters-Williams1, 2* and Yan Li2
1
School of Computing and Information Technology, University of Technology, Jamaica,
Kingston 6, Jamaica W. I.
2
Department of Mathematics and Computing, Centre for Systems Biology,
University of Southern Queensland, QLD 4350, Australia
[email protected]; [email protected]
Abstract. A common problem found in statistics, signal processing, data
analysis and image processing research is the estimation of mutual information,
which tends to be difficult. The aim of this survey is threefold: an introduction
for those new to the field, an overview for those working in the field and a
reference for those searching for literature on different estimation methods. In
this paper comparison studies on mutual information estimation is considered.
The paper starts with a description of entropy and mutual information and it
closes with a discussion on the performance of different estimation methods
and some future challenges.
Keywords: Mutual Information, Entropy, Density Estimation.
1 Introduction
Mutual Information (MI) is a nonlinear measure used to measure both linear and
nonlinear correlations. It is well-known that any estimation of MI is difficult but it is a
natural measure of the dependence between random variables thereby taking into
account the whole dependence structure of the variables.
There has been work on the estimation of MI but up to 2008 to the best of our
knowledge there has been no research on the comparison of the different categories of
commonly used estimators. Given the varying results in the literature on these
estimators this paper seeks to draw conclusion on their performance up to 2008.
We aim to introduce and explain MI and to give an overview of the literature on
different mutual information estimators. We start at the basics, with the definition of
entropy and its interpretation. We then turn to mutual information, presenting its
multiple forms of definition, its properties and applications to which it is applied.
The survey classifies the estimators into two main categories: nonparametric
density and parametric density. Each category looks at the commonly used types of
estimators in that area. Finally, having considered a number of comparison studies,
we discuss the results of years of research and also some challenges that still lie
ahead.
*
Corresponding author.
The paper is organized as follows. In Section 2, the concept of entropy is
introduced. Section 3 highlights the MI in general. In Section 4 different methods for
the estimation of MI are presented. Section 5 describes comparison studies of
different estimation methods then finally Section 6 discusses the conclusion.
2 Entropy
The concept of entropy was developed in response to the observation that a certain
amount of functional energy released from combustion reactions was always lost to
dissipation or friction and thus not transformed into useful work. In 1948, while
working at Bell Telephone Laboratories electrical engineer Claude Shannon set out to
mathematically quantify the statistical nature of “lost information” in phone-line
signals. To do this, Shannon developed the very general concept of information
entropy, a fundamental cornerstone of information theory. He published his famous
paper “A Mathematical Theory of Communication”, containing the section to what he
calls Choice, Uncertainty, and Entropy. Here he introduced an “H function” as:
k
H = −K ∑ p(i)log p(i)
(1)
i =1
where K is a positive constant. Entropy works well when describing the order,
uncertainty or variability of a unique variable, however it cannot work properly for
more than one variable. This is where joint entropy, mutual information and
conditional entropy come in.
(a) Joint entropy. The joint entropy of a pair of discrete random variables X and Y is
defined as
H ( X ,Y ) = −∑
∑
p ( x , y ) lo g p ( x , y )
(2)
where p(x,y) is the joint distribution of the variables. The chain rule for joint
entropy states that the total uncertainty about the value of X and Y is equal to the
uncertainty about X plus the (average) uncertainty about Y once you know X.
x
y
H ( X , Y ) = H ( X ) + H (Y | X )
(3)
(b) Conditional Entropy (Equivocation). Conditional entropy measures how much
entropy are random variable X has remaining if the value of a second random variable
Y is known. It is referred to as the entropy of X conditional on Y, written H(X | Y) and
defined as:
H ( X | Y ) = ∑∑ p( x | y)log p( x | y) p( y)
x
(4)
y
The chain rule for conditional entropy is
H (Y | X ) = H ( X , Y ) − H ( X )
(c) Marginal Entropy. Marginal or absolute entropy is defined as:
(5)
n
H ( X ) = ∑ p(xi ) log p( xi )
(6)
i =1
where n represents the number of events xi with probabilities p(xi) (i=1,2..n)
These joint entropy, conditional entropy and marginal entropy then create the
Chain rule for Entropy which can be defined as
H ( X , Y ) = H ( X ) + H (Y | X ) = H (Y ) + H ( X | Y )
(7)
3 Mutual Information
Mutual Information (MI), also known as transinformation, was first introduced in
classical information theory by Shannon in 1948. It is considered to be a non
parametric measure of relevance [1] that measures the mutual dependence of two
variables, both linear and non linear for which it has a natural generalization. It
therefore looks at the amount of uncertainty that is lost from one variable when the
other is known. MI represented as I(X:Y), in truth measures the reduction in
uncertainty in X which results from knowing Y, i.e. it indicates how much
information Y conveys about X. Mutual information has the following properties
(i)
I ( X : Y ) = I (Y : X )
It is symmetric
(ii) I ( X : Y ) ≥ 0
It is always non-negative between X and Y; the uncertainty of X cannot be
increased by learning of Y.
It also has the following properties:
(iii) I ( X : X ) = H ( X )
The information X contains about itself is the entropy of X
(iv) I ( X : Y ) ≤ H ( X )
I ( X : Y ) ≤ H (Y )
The information variables contain about each other can never be greater than the
information in the variables themselves.
The information in X is in no way related to Y, i.e. no knowledge is gained about
X when Y is given and visa versa. X and Y are, therefore, strictly independent.
Mutual information can be calculated using Entropy, considered to be the best way,
using Probability Density and using Kullback-Leibler Divergence.
4 Estimating Mutual Information
MI is considered to be very powerful yet it is difficult to estimate [2]. Estimation can
therefore be unreliable, noisy and even bias. To use the definition of entropy, the
density has to be estimated. This problem has severely restricted the use of mutual
information in ICA estimation and many other applications. In recent years
researchers are designed different ways of estimating MI. Some researchers have used
approximations of mutual information based on polynomial density expansions,
which led to the use of higher-order cumulants. The approximation is valid, however,
only when it is not far from the Gaussian density function. More sophisticated
approximations of mutual information have been constructed. Some have estimated
MI by binning the coordinate axes, the use of histograms as well as wavelets. All
have, however, sought to estimate a density P(x) given a finite number of data points
xN drawn from that density function. There are two basic approaches to estimation –
parametric and nonparametric and this paper seeks to survey some of these methods.
Nonparametric estimation is a statistical method that allows the functional form
of the regression function to be flexible. As a result, the procedures of nonparametric
estimation have no meaningful associated parameters. Parametric estimation, by
contrast, makes assumptions about the functional form of the regression and the
estimate is of those parameters that are free. Estimating MI techniques include
histogram based, adaptive partitioning, spline, kernel density and nearest neighbour.
The choices of the parameters in these techniques are often made “blindly”, i.e. no
reliable measure used for the choice. The estimation is very sensitive to those
parameters however especially in small noisy sample conditions [1]. Nonparametric
density estimators are histogram based estimator, adaptive partitioning of the XY
plane, kernel density estimator (KDE), B-Spline estimator, nearest neighbor (KNN)
estimator and wavelet density estimator (WDE).
Parametric density estimation is normally referred to as Parameter Estimation. It
is a given form for the density function which assumes that the data are from a known
family of distributions, such as the normal, log-normal, exponential, and Weibull (i.e.,
Gaussian) and the parameters of the function (i.e., mean and variance) are then
optimized by fitting the model to the data set. Parametric density estimators are
Bayesian estimator, Edgeworth estimators, maximum likelihood (ML) estimator, and
least square estimator.
5 Comparison Studies of Mutual Information Estimation
By comparison studies we mean all papers written with the intention of comparing
several different estimators of MI as well as to present a new method.
5.1 Parametric vs. Nonparametric Methods
It's worth to note that it's not one hundred percent right when we say that nonparametric methods are "model-free" or free of distribution assumptions. For
example, some kinds of distance measures have to be used to identify the "nearest"
neighbour. Although the methods did not assume a specific distribution, the distance
measure is distribution-related in some sense (Euclidean and Mahalanobis distances
are closely related to multivariate Gaussian distribution). Compared to parametric
methods, non-parametric ones are only "vaguely" or "remotely" related to specific
distributions and, therefore, are more flexible and less sensitive to violation of
distribution assumptions.
Another characteristic found to be helpful in discriminating the two is that: the
number of parameters in parametric models is fixed a priori and independent of the
size of the dataset, while the number of statistics used for non-parametric models are
usually dependent on the size of the dataset (e.g. more statistics for larger datasets)
5.2 Comparison of Estimators
(a)
Types of Histograms. Daub et. al [2] compared the two types of histogram
based estimators -equidistant and equiprobable and found that the equiprobable
histogram-based estimator is more accurate.
(b)
Histogram vs AP. Trappenberg et al. [3] compared the equidistant histogrambased method, the adaptive histogram-based method of Darbellay and Vajda and the
Gram-Charlier polynomial expansion and concluded that all three estimators gave
reasonable estimates of the theoretical mutual information, but the adaptive
histogram-based method converged faster with the sample size.
(c)
Histogram vs KDE. Histogram based methods and kernel density estimations
are the two principal differentiable estimator of Mutual Information [4] however
histogram-based is the simplest non-parametric density estimator and the one that is
mostly frequently encountered. This is because it is easy to calculate and understand.
(d)
B-Spline vs KDE. MI calculated from KDE does not show a linear behavior
but rather an asymptotic one with a linear tail for large data sets. Values are slightly
greater than those produces when using the B-Spline method. According to Daub et.
al. [2] the B-Spline method is computationally faster than the kernel density method
and also improves the simple binning method. It was found that B-Spline also avoided
the time-consuming numerical integration steps for which kernel density estimators
are noted.
(e)
B-Spline vs Histogram. In the classical histogram based method data points
close to bin boundaries can cross over to a neighboring bin due to noise or
fluctuations; in this way they introduce an additional variance into the computed
estimate [5]. Even for sets of a moderate size, this variance is not negligible. To
overcome this problem, Daub et al. [2] proposed a generalized histogram method,
which uses B-Spline functions to assign data points to bins. B-Spline also somewhat
alleviates the choice-of-origin problem for the histogram based methods by
smoothing the effect of transition of data points between bins due to shifts in origin.
(f)
B-Spline vs KNN. Rossi et al. [6] stated that B-Splines estimated MI reduces
feature selection. When compared to KNN, it was found that KNN has a total
complexity of O(n3p2) (because estimation is linear in the dimension n and quadratic
in the number of data points p), while B-Spline worst-case complexity is still less at
O(n3p) thus having a smaller computation time.
(g)
WDE vs other Nonparametric Estimators. In statistics, amongst other
applications, wavelets [7] have been used to build suitable non parametric density
estimators. A major drawback of classical series estimators is that they appear to be
poor in estimating local properties of the density. This is due to the fact that
orthogonal systems, like the Fourier one, have poor time/frequency localization
properties. On the contrary wavelets are localized both in time and in frequency
making wavelet estimators well able to capture local features. Indeed it has been
shown that KDE tend to be inferior to WDE [8].
(h)
KDE vs KNN. A practical drawback of the KNN-based approach is that the
estimation accuracy depends on the value of k and there seems no systematic strategy
to choose the value of k appropriately. Kraskov et. al [9] created a KNN estimator and
found that for Gaussian distributions KNN performed better. This was reinforced
when Papana et. al. [10] compared the two along with the histogram based method.
They found that KNN was computationally more effective when fine partitions were
sought, due to the use of effective data structures in the search for neighbors. They
concluded that KNN was the better choice as fine partitions capture the fine structure
of chaotic data and because KNN is not significantly corrupted with noise. They
found therefore that the k-nearest neighbor estimator is the more stable and less
affected by the method-specific parameter.
(i)
AP vs ML. Being a parametric technique, ML estimation is applicable only if
the distribution which governs the data is known, the mutual information of that
distribution is known analytically and the maximum likelihood equations can be
solved for that particular distribution. It is clear that ML estimators have an ‘unfair
advantage’ over any nonparametric estimator which would be applied to data coming
from a distribution. Darbellay et. al. [11] compared AP with ML. They found that
adaptive partitioning appears to be asymptotically unbiased and efficient. They also
found that unlike ML it is in principle applicable to any distribution and intuitively
easy to understand.
(j)
ML vs KDE. Suzuki et. al. [12] considers KDE to be a naive approach to
estimating MI, since the densities pxy(x, y), px(x), and py(y) are separately estimated
from samples and the estimated densities are used for computing MI. After
evaluations they stated that the bandwidth of the kernel functions could be optimized
based on likelihood cross-validation, so there remains no open tuning parameters in
this approach. However, density estimation is known to be a hard problem and
division by estimated densities is involved when approximating MI, which tends to
expand the estimation error. ML does not have this estimation error.
(k) ML vs KNN. Using KNN as an estimator for MI means that there is no simply
replacement of entropies with their estimates, but it is designed to cancel the error of
individual entropy estimation. It has systematic strategy to choose the value of k
appropriately. Suzuki et. al. [12] found that KNN works well if the value of k is
optimally chosen. This means that there is no model selection method for determining
the number of nearest neighbors. ML, however, does not have this limitation.
(l) EDGE vs ML. Suzuki et. al. [12] found that if the underlying distribution is close
to the normal distribution, approximation is quite accurate and the EDGE method
works very well. However, if the distribution is far from the normal distribution, the
approximation error gets large and therefore the EDGE method performs poorly and
may be unreliable. In contrast, ML performs reasonably well for both distributions.
(m) EDGE vs KDE and Histogram. Research has shown that differential entropy by
the Edgeworth expansion avoids the density estimation problems although it makes
sense only for "close"-to-Gaussian distributions. Further research shows that the order
of Edgeworth approximation of differential entropy is O(N-3/2), while KDE
approximation is of order O(N-1/2) where N is the size of processed sample. This
means that KDE cannot be used for differential entropy while the Edgeworth
expansion of neg-entropy produces very good approximations also for moredimensional Gaussian distributions [5].
(n)
ML vs Bayesian. ML is prone to over fitting [13]. This can occur when the size
of the data set is not large enough to compare to the number of degrees of freedom of
the chosen model. The Bayesian method fixes the problem of ML in that it deals with
how to determine the best number of model parameters. It is, therefore, vey useful
where there large data sets are hard to come by e.g. neuroscience.
(o) LSMI vs KNN and KDE. Suzuki et. al. [14] found that when KDE, KNN and
LSMI are compared they found density estimation to be a hard problem and therefore
the KDE-based method may not be so effective in practice. Although KNN performed
better than KDE there was a problem when choosing the number k appropriately.
Their research showed that LSMI overcame the limitations of both KDE and KNN.
(p)
LSMI vs EDGE. Suzuki et. al. [14] found that although EDGE was quite
accurate and works well if the underlying distribution is close to normal distribution;
however when the distribution is far the approximation error gets large and EDGE
becomes unreliable. LSMI is distribution-free and therefore does not suffer from these
problems.
6 Discussion and Conclusion
There have been many comparisons of different estimation methods. TABLE I
shows the order of performances of those discussed within this paper. It can be shown
that both KNN and KDE converge to the true probability density as N→∞, provided
that V shrinks with N, and k grows with N appropriately. It can be seen, therefore, that
KNN and KDE truly outperform the histogram methods.
TABLE I. Estimators based on performances in Category
Nonparametric Density
Equidistant Histogram
Equiprobable Histogram
Adaptive Partitioning
Kernel Density
K-Nearest Neighbor
B-Spline
Wavelet
Parametric Density
Edgeworth
Least Square
Maximum Likelihood
Bayesian
From conclusions of the comparison studies it can be inferred that estimating MI
by parametric density produces a more effect methodology. This is supported by
researchers [5, 11, 12, 14] who have casted nonparametric methods into parametric
frameworks, such as WDE or KDE into a ML framework - in doing so, moving the
problem into the parametric realm. When both methods are therefore combined the
performances of the methods used in this paper are (i) Equidistant Histogram, (ii)
Equiprobable Histogram , (iii) Adaptive Partitioning, (iv) Kernel Density, (v) KNearest Neighbor, (vi) B-Spline, (vii) Wavelet, (viii) Edgeworth, (ix) Least-Square,
(x) Maximum Likelihood and (xi) Bayesian. Since parametric methods are better the
question remains as why the nonparametric methods are still the methods of the
choice for most estimators.
To date there is still research into the development of new ways to estimate MI.
Research will continue on: (i) their performances, (ii) optimal parameters
investigation, (iii) linear and non-linear datasets and (iv) applications.
The challenge is how to create an estimation method that covers both parametric
and non-parametric density methodologies and still be applied to most if not all
applications effectively. From the continuing interest in the measurement it can be
deduced that mutual information will still be popular in the near future. It is already a
successful measure for many applications and it can indoubtedly be adapted and
extended to aid in many more problems.
References
1. Francois, D., Wertz, V., Verleysen, M.: In Proceedings of the European
Symposium on Artificial Neural Networks (ESANN), pp. 239-244 (2006)
2. Daub, C., O., Steuer, R., Selbig, J.,Kloska, S.: BMC Bioinformatics 5 (2004)
3. Trappenberg, T., Ouyang, J., Back, A.: Journal of Latex Class Files 1, 8, (2002)
4. Fransens, R., Strecha, C., Van Gool, L.: In Proceedings of Theory and Applications
of Knowledge-Driven Image Information Mining with Focus on Earth Observation
(ESA-EUSC) (2004)
5. Hlaváčková-Schindler, K.. Information Theory and Statistical Learning, Springer
Science & Business Media LLC (2009).
6. Rossi, F., Francois, D., Wertz, V., Meurens, M., Verleysen,M.: In Chemometrics
and Intelligent Laboratory Systems 86, (2007)
7. Aumônier, S.: Generalized Correlation Power Analysis in the workshop ECRYPT
“Tools for Cryptanalysis" (2007)
8. Vamucci, M.: Technical Reports, Duke University. 1998.
9. Kraskov, A., Stögbauer, H., Andrzejak, R. G., Grassberger, P.:
http://arxiv.org/abs/q-bio/0311039.
10. Papana, A., Kugiumtzis, D. Nonlinear Phenomena in Complex Systems 11, 2
(2008)
11. Darbellay, G. A., Vajda, I.: IEEE Transaction on Information Theory 45, 4,
(1999)
12. Suzuki, T., Sugiyama, M, Sese, J., Kanamori, T.: In Proceedings of JMLR:
Workshop and Conference 4. (2008)
13. Endres, D., Földiák, P.: IEEE Transactions on Information Theory 51, 11 (2005)
14. Suzuki, T., Sugiyama, M., Sese, J., Kanamori, T.: In Proceedings of the Seventh AsiaPacific Bioinformatics Conference (APBC), (2009)