IMPROVING THE PERFORMANCE STABILITY OF INDUCTIVE EXPERT
SYSTEMS UNDER INPUT NOISE
Vijay S. Mookerjee, Michael V. Mannino, and Robert Gilson
Department of Management Science, Box 353200
University of Washington, Seattle, WA 98195-3200
[email protected], [email protected],
[email protected]
This paper appears in Information Systems Research 6, 4 (December 1995), 328-356.
IMPROVING THE PERFORMANCE STABILITY OF INDUCTIVE EXPERT
SYSTEMS UNDER INPUT NOISE
Abstract
Inductive expert systems typically operate with imperfect or noisy input attributes.
We study design differences in inductive expert systems arising from implicit
versus explicit handling of input noise. Most previous approaches use an implicit
approach wherein inductive expert systems are constructed using input data of
quality comparable to problems the system will be called upon to solve. We
develop an explicit algorithm (ID3ecp) that uses a clean (without input errors)
training set and an explicit measure of the input noise level and compare it to a
traditional implicit algorithm, ID3p (the ID3 algorithm with the pessimistic pruning
procedure). The novel feature of the explicit algorithm is that it injects noise in a
controlled rather than random manner in order to reduce the performance variance
due to noise. We show analytically that the implicit algorithm has the same
expected partitioning behavior as the explicit algorithm. In contrast, however, the
partitioning behavior of the explicit algorithm is shown to be more stable (i.e.,
lower variance) than the implicit algorithm. To extend the analysis to the
predictive performance of the algorithms, a set of simulation experiments is
described in which the average performance and coefficient of variation of
performance of both algorithms are studied on real and artificial data sets. The
experimental results confirm the analytical results and demonstrate substantial
differences in stability of performance between the algorithms especially as the
noise level increases.
1. Introduction
Inductive expert systems have become an important decision support tool as evidenced by
considerable attention in the academic literature and business press, and a number of commercial
products to develop such systems. Inductive expert systems are typically developed to support
classification tasks, i.e., systems that attempt to classify an object as one of n categories [Quinlan
1986a]. Examples of classification in business decision making include fault diagnosis in
semiconductor manufacturing [Irani et al. 1993], bank failure prediction [Tam and Kiang 1990],
and industry and occupation code prediction [Creecy et al. 1992]. The primary goal of an
inductive expert system is to perform at the same level of human experts. Such systems can
provide many benefits to an organization [Holsapple and Whinston 1987] such as reducing
decision making time, improving the consistency of decisions, and reducing dependence on scarce
human experts.
An inductive expert system is constructed using a learning algorithm and data set. A
learning algorithm develops classification rules that can be used to determine the class of an object
from its description, i.e., from the object's attributes. The classification rules developed by these
algorithms can be depicted as a decision tree in which the non-leaf nodes of the tree prescribe
inputs that must be observed and the arcs represent states that the input variables can take. Leaf
nodes in the tree indicate how an object is to be classified. Induction algorithms build such a tree
from a set of pre-classified cases referred to as the training set. Another part of the data set
known as the test set is used to study the performance of a decision tree on novel cases. Inductive
expert systems are typically developed to maximize solution accuracy; i.e., maximize the number
of cases in which the output (decision, recommendation) provided by the system is similar to that
provided by human experts. Economic considerations, for example, costs of observing inputs,
Page 2
benefits from system outputs, and other factors that may contribute to system value are rarely
factored into system design [Mookerjee and Dos Santos 1993].
The subject of this paper is the design and performance evaluation of inductive expert
systems using noisy input attributes. The presence of input noise can have a significant impact on
the performance of an inductive expert system. We only consider noise that affects the input
1
values used by the system to make classification decisions, not other forms of noise. This
definition includes errors from such causes as incorrectly measuring an input, wrongly reporting
the state of an input, relying on stale values, and using imprecise measurement devices. Input
errors in a training set can cause a learning algorithm to form a rule with an incorrect state for an
input, while input errors in cases to be classified can cause the wrong rule to be used.
The specific issue studied here is how to account for the level of input noise: (i) implicitly
through a training set with a representative level of noise or (ii) explicitly through a noise
parameter and a clean training set. Figure 1 graphically depicts the explicit and implicit
approaches. In common practice, the implicit approach is used because it is cost effective and has
been carefully studied. However, high variance of performance is a key disadvantage of the
implicit approach that has not been widely discussed or studied. Our most important finding here
is that an induction algorithm using an explicit noise parameter can have more stable performance
than a comparable implicit algorithm.
Variation in performance is an outcome variable of interest in a wide variety of systems.
For example, the performance of a manufacturing process is judged in terms of its mean behavior
1
It must be noted however, that researchers in the past have discussed other dimensions of noise such as
conflicts in the classification, classification errors, missing input data, etc. In this paper, we will use the
term noise in a restricted sense, only with reference to input measurement accuracy. Thus, for our
purposes, noise rises if the input measurement accuracy falls. Conversely, noise reduces if the input
measurement accuracy rises.
Page 3
as well as by the variation in its behavior. Reducing variance is routinely used as a performance
objective in survey research where errors may be introduced by interviewers, respondents,
questionnaires, processing of forms, and so on. In the context of inductive expert systems
however, past research has largely ignored this important design objective. Variation in
performance could often be an extremely important aspect of an inductive expert system. For
example, if a loan granting inductive expert system was to make very good decisions in one set of
cases, but extremely poor ones in another set, it may cause the bank to fail in the period of poor
performance. Thus, managers are likely to prefer a more stable system to a highly variable one
even though the mean performance of the two systems is the same.
Clean Training Set
Noisy Training Set
Noise
Parameter
Explicit Algorithm
Decision Tree
Implicit Algorithm
Decision Tree
Figure 1: Explicit and Implicit Noise Handling
In addition to stable performance, explicit noise handling is interesting to study because
there are a number of situations in which an explicit approach is more practical than an implicit
Page 4
approach. One such situation is when the training set is obtained from experts rather than from
historical data. Here, an implicit approach would require that the input states in the training set be
deliberately corrupted. In addition, if there are multiple ways to measure an input or if the level of
noise in historical data is not representative of current practices, an explicit approach may be
preferable.
Because explicit algorithms have not been carefully studied, the major topics presented are
the design and performance evaluation of explicit algorithms. We design an explicit algorithm
that injects noise according to a specified noise parameter as it partitions a data set. The novel
aspect of the algorithm is that it injects noise in a controlled rather than random manner in order
to reduce the variance due to noise. We show analytically that the expected partitioning behavior
of the implicit and explicit algorithms is the same. However, we demonstrate analytically that the
explicit algorithm has more stable partitions than the implicit algorithm. To extend the analytical
results to the classification accuracy of the algorithms, we conduct a set of simulation experiments
on several real and artificial data sets with a range of values for the number of classes and
skewness in the class distributions. Our experimental results reveal that the expected accuracy
and variance of accuracy of the algorithms are consistent with our analytical results.
This rest of this paper is organized as follows. In Section 2, we review related work on
noise handling approaches used in inductive expert systems. In Section 3, we provide background
on decision tree induction and analytically evaluate the behavior of the implicit and explicit
algorithms: ID3p and ID3ecp. In Section 4, we describe the hypotheses investigated, experimental
designs used, and results obtained from a set of simulation experiments. A summary and
conclusions are provided in Section 5.
Page 5
2. Related Work
In this section, we summarize a theoretical study of input noise, various pruning
procedures, and models to cope with measurement errors in surveys. Laird [Laird 1988] studied
the Bernoulli Noise Process (BNP) as an extension of the theory of Probably, Approximately
Correct (PAC) learning. The goal of PAC theory is to derive an upper bound on the number of
examples needed to approximately learn a concept within a given error bound with a specified
confidence level. A BNP is characterized by independent parameters for the classification error
rate and the input error rate. Laird's basic result is that input error rate alone is not sufficient to
determine the maximum number of examples needed. There must be an additional parameter that
indicates the sensitivity of the true concept definition to input errors. This theoretical result about
the relationship between the importance of an attribute and the impact of noise has been
empirically demonstrated in other studies.
In more applied studies, researchers have developed pruning procedures (post
construction techniques) to refine the rule set generated by a learning algorithm. Learning
algorithms generally find a perfect set of rules for a training set, but the rules are usually too
specialized leading to poor performance on unseen cases. Pruning techniques reduce
specialization by eliminating rules in whole or part. Similarly, pruning techniques have also been
found useful to handle noise because noise in a training set can lead to extra rules and highly
specialized rules. For example, Quinlan [Quinlan 1987] demonstrated that four pruning methods
significantly reduced the complexity of induced decision trees without adversely affecting
accuracy. He later found on a study of the chi-square pruning technique [Quinlan 1986b] that the
performance of a learning algorithm is better using a noisy training set than a perfect test set. He
also found that increases in predictive performance from noise reduction depends on the
importance of an input.
Page 6
Two important themes in the development of pruning procedures are the use of an extra
test set and parameters to control the amount of pruning. A number of techniques use an extra
test set to choose among multiple collections of rules (critical value pruning [Mingers 1989] and
error complexity pruning [Breiman et al. 1984]) or to reduce the complexity of the rules
(minimum error pruning [Quinlan 1987]). These techniques require more data than techniques
that prune using the training set alone (pessimistic pruning [Quinlan 1987], minimum error
pruning [Niblett and Bratko 1986], and Laplace pruning [Christie 1993]). Mingers [Mingers
1989] found that pruning techniques using an extra test set achieved higher accuracy than
techniques not using an extra test set. However, Quinlan [Quinlan 1987], in an earlier study, did
not find higher accuracy as a result of extra test cases. Several techniques have been developed
that use a single parameter to control the amount of pruning (error complexity pruning [Breiman
et al. 1984] and m-probability-estimate pruning [Cestnik and Bratko 1991]). The parameters are
rather coarse applying to the entire data set, not individual inputs. In addition, there are no
guidelines for setting parameter values except that high values should be used when there is a
large amount of noise. Moulet [Moulet 1991] developed input noise parameters for the
ABACUS discovery system and demonstrated their effectiveness in learning simple laws of
physics. However, he did not apply his technique to classification problems.
Unlike the work on pruning techniques, the area of measurement errors in surveys
[Groves 1991] includes a rich stream of research on techniques to measure and reduce the level of
input noise and models to compensate for the effect of input noise on prediction tasks. This area
of research has developed a detailed classification of input noise beginning with systematic errors
that introduce bias and random errors that cause variance. Beyond this division, the source of
errors (interviewer, respondent, process, questionnaire) and the cause of errors (e.g., memory loss
and non-response) are often identified. When the level of input noise is not known, it can be
estimated using a re interview technique [Hill 1991, Rao and Thomas 1991] where error-prone
measurements are made on a sample and then more expensive and relatively error-free
Page 7
measurements are made on a sub sample. The re interview technique is similar to the idea of
using a clean training set with an explicit estimate of the noise level. Many models have been
developed to compensate for the effect of input noise on regression [Fuller 1991], analysis of
variance [Biemer and Stokes 1991], and estimation of survey statistics of categorical data [Biemer
and Stokes 1991]. However, there is no reported research on classification tasks.
3. Algorithm Design and Analysis
In this section, we present the explicit noise algorithm used in our simulation experiments
and analyze the mean and variance of its partitioning behavior. Before presenting the explicit
noise algorithm, we review the induction process underlying the baseline algorithm, ID3p, (the
ID3 algorithm [Quinlan 1986a] with the pessimistic pruning procedure).
3.1. Decision Tree Induction
Induction algorithms develop a decision tree by recursively creating non-leaf nodes and
leaf nodes in the tree. Non-leaf nodes are labeled by input names. The input chosen to label a nonleaf node is determined using an input selection criterion and a set of cases. Traditionally, inputs
are selected by their information content, measured by the reduction in information entropy
achieved as a result of observing the input. After an input has been chosen to label a non-leaf
node, each of the q outgoing arcs are labeled by a possible state of the selected input where q is
the number of possible states. The set of cases used to compute the label of a node (for the root
node this is the entire training set) is then partitioned into q subsets such that the state of the input
used to label the node is the same within each subset. The tree can grow along each outgoing arc
using the subset of cases corresponding to the state of the input used to label the arc. Creation of
non-leaf nodes continues along each path of the tree until a stopping condition is reached, at
which stage a leaf node is created. Leaf nodes are labeled using a classification function.
Page 8
An important factor that must be considered in the design of an induction algorithm is the
input selection criterion. The input selection criterion determines how, from a set of candidate
inputs, an input is chosen to label a non-leaf node. We describe the input selection criterion for
the ID3 algorithm in the remainder of this subsection.
Let,
D N
be a randomly drawn training set of size N,
X 1 , X 2 ,..., X p
be p observable input variables that may be used to classify an object,
xk1, xk2, .., xkq
be q possible states for input X k ,
ψ
be the random variable for the class of an object,
c1, c2, .., cm
~
Z
be m possible states for the class variable ψ ,
~
L( Z , π )
be a partition of the training set (Z ⊆ D N ),
~
~
be an input state conjunction, e.g., X 1 = x14 ∧ X 2 = x 23 ,
~
be a partition of Z such that π is true,
k ( Z )
be a classification function that determines how a leaf node is labeled,
g ( X k | Z )
be the expected gain for input X k given Z .
π
The input selection criterion in the ID3 algorithm chooses the input with the maximum
information content (gain) measured by the reduction in information entropy [Shannon and
Weaver 1949] as a result of observing the input. Formally, the gain of input X k is defined as:
g ( X k | Z ) = ∆EN ( X k | Z ) = EN ( Z ) − EN ( X k | Z )
where
~
EN ( Z ) = −
m
∑ P[ψ~ = cr | Z ]log2 P[ψ~ = cr | Z ] , is the initial expected entropy,
~
~
(1)
(2)
r =1
~ ~
EN ( X k | Z ) =
q
∑ P[ X k = xkj | Z ]EN ( L( Z , X k = xkj )) ,
j =1
~
~
~ ~
~
is the expected entropy after observing X k , and
(3)
Page 9
EN ( L ( Z , X k = xkj )) =
m
−
∑ P[ψ = cr | L( Z , X k = xkj )]log2 P[ ψ = cr | L( Z , X k = xkj )]
(4)
r =1
is the expected entropy in the partition L ( Z , X k = x kj )
ID3 estimates the probabilities in equations (2) through (4) by sample proportions obtained from
the training data.
~
To demonstrate the above equations, consider a sample of 100 cases ( Z ) with 2 classes (c1 =
~
66, c2 = 34). Let input X k with states xk1, xk2, xk3 be used to partition the 100 cases. From
equation (2) the initial entropy is:
~
EN ( Z ) = −(0.66 *log 2 0.66 + 0.34 *log 2 0.34) = 0.9248
~
For X k = x k 1 , let c1 = 48 and c2 = 12 be the subpartition of cases obtained. The entropy within
this subpartition is:
~ ~
EN ( L( Z , X k = xk 1 )) = −(0.8 *log 2 0.8 + 0.2 *log 2 0.2) = 0.7219
~
~ ~
For X k = x k 2 , let c1 = 10 and c2 = 20 be the subpartition with EN ( L( Z , X k = xk 2 )) = 0.9149
~
computed the same as above. For X k = x k 3 , let c1 = 8 and c2 = 2 be the subpartition with
~ ~
~
EN ( L( Z , X k = xk 3 )) = 0.7219 . From equation (3), the entropy after observing X k is:
~ ~
EN ( X k | Z ) = (0.6 * 0.7219 + 0.3 * 9149 + 01
. * 0.7219) = 0.7798
~
Thus from equation (1) the gain from observing input X k is:
~ ~
~ ~
g ( X k | Z ) = ∆EN ( X k | Z ) = 0.9248 − 0.7798 = 01449
.
To cope with noise and overfitting, we augment ID3 with the pessimistic pruning
procedure [Quinlan 1987]. We call the augmented ID3 algorithm, ID3p. The pessimistic pruning
procedure uses a statistical measure to determine whether replacing a sub-tree by its best leaf (i.e.,
the classification that maximizes accuracy for a given set of cases) is likely to increase accuracy.
If so, the branch is replaced by its best leaf; otherwise the branch is retained. Once all branches
have been examined, the process terminates. A more detailed description can be found in
Page 10
Appendix B. We use the pessimistic pruning procedure because it is easy to implement, effective
on noisy data [Mingers 1989], and does not require an extra test set.
3.2. Model of Noise
Inductive expert systems typically operate under conditions where inputs are subject to
noise. Noise occurs when the true input state is perturbed by a measurement process. We
assume that measurement errors is independent of the time of measurement and the true state of
an input. If we ignore differences among wrong states (e.g., measuring a high value as medium or
low), noise can be characterized as a binomial process with mean C, the probability of correct
measurement. The value of C can be estimated from empirical data as follows. First, for each
noisy input, collect a representative sample of input values. Second, correct the noisy values
through various techniques such as repeated measurement [Hill 1991, Rao and Thomas 1991]
and/or improved measurement, that is, using more expensive and relatively error free devices.
Third, estimate the population error rate from the sample as the number of values that are
incorrect divided by the sample size. One minus the sample error rate is an unbiased estimate of
C.
We define the parameter W as the probability of wrong or incorrect measurement, a
measure of the likelihood of disruptions in the measurement process that lead to a particular
2
incorrect state being recorded . We assume that wrong states are equally likely and that all inputs
have the same level of noise. Thus, the relationship C + (q-1)W = 1 holds for any input since
there are q-1 incorrect states. For example, if there is a 5 state input and the probability of correct
measurement is C = 0.8, then any wrong state has a probability of W = (1 − 0.8) (5 − 1) = 0.05.
2
The noise parameters (C and W) are output rather than process measures. In some studies, the noise
parameter is the probability that the measurement process has been perturbed. A perturbation in the
measurement process may or may not lead to an incorrect measurement. We use an output measure here
because it is more convenient, and a process measure can be mapped to an output measure.
Page 11
Although the assumptions about constant C across inputs and constant W within an input’s states
can be easily relaxed, we use them to simplify our analysis. Without these assumptions, many
more noise parameters will have to be estimated. More precisely, C and W are defined as:
C = P[ X ko = xkj | X kt = xkj ] ∀k , i
~
~
W = P[ X ko = x kj | X kt = x ki ] ∀k , ∀j ≠ i
where X ko , X kt are the random variables for the observed (noisy) and true states of input
~
Xk .
To analytically describe the impact of noise on the input gain, we state Proposition 1:
Proposition 1
~ ~
∂g ( X ko | Z C )
> 0, C ∈ (1 / q ,1]
∂C
= 0, C = 1 / q
< 0, C ∈[0,1 / q )
(6.1)
(6.2)
(6.3)
~
where Z C is a noisy partition with noise level C.
With no noise, the gain is at its highest (C = 1). As noise is increased (value of C is decreased
from 1 to 1/q), the gain decreases to the point where the value of C is 1/q (6.1). When C is equal
to 1/q, there is no benefit from observing the input because it could occur in any of its states with
equal probability (6.2). As the value of C decreases below 1/q, the noise becomes so high that the
observed states become predictably incorrect. Hence, observed states of the input again begin to
provide some information (6.3).
Thus the nature of the relationship between the noise level and the gain is convex,
validating that we are dealing with a reasonable model of noise. In Appendix A, we prove
Proposition 1 for a special case. In addition, our simulations strongly suggest that the gain
monotonically decreases (i.e., less uncertainty reduction) as the correct measurement probability
increases from 0 to 1/q and then monotonically increases until the correct measurement
probability is 1.
Page 12
3.3. Impact of Noise on Expected Behavior
In this subsection, we demonstrate how the information content of a noisy input is
computed in an implicit version of ID3 (ID3p) where a noisy training set is used and in an explicit
version of ID3 (ID3ep) where a clean training set and noise parameters are used.
3.3.1.Implicit Handling of Noise
In an implicit algorithm such as ID3p, the level of noise acts implicitly on the gain of an
input. Specifically, Equation (3) uses two probability estimates: (i) the probability of a class
given a new input and the set of previously observed inputs and (ii) the probability of a new input
given the set of previously observed inputs. With noise, we need to estimate the same two
probabilities except that it is the noisy observed states rather than true states of the input that are
estimated. Equation (7) shows the expected entropy of X ko after observing a set of noisy inputs
). ID3 estimates the probabilities in equation (7) by sample proportions obtained from the
(Π
o
p
noisy training data.
q
m
) = − ∑ P[ X
] ∑ ζ log ζ
EN ( X ko | Π
=
x
|
Π
o
ko
kj o
r
2 r
j =1
r =1
(7)
] and
where ζ r = P[ ψ = cr | X ko = xkj ∧ Π
o
Π
o = Π1o ∧... ∧ Π do is the set of previous observations on the path.
3.3.2.Explicit Handling of Noise
Explicit algorithms estimate the probabilities in equation (7) using a clean training set and
noise parameters rather than with a noisy training set. Both implicit and explicit algorithms are
subject to noise processes having the same mean noise parameters (C and W in our case). The
difference is that the noise process has already occurred for implicit algorithms as opposed to the
noise process occurring as part of explicit algorithms. Thus, implicit and explicit algorithms will
compute the same expected decision tree because the expected gain calculations are the same.
Even though explicit algorithms have better information (both the noise level and clean data),
Page 13
their expected predictive ability is the same as that of implicit algorithms. However, in subsection
3.4, we demonstrate that the additional information can be used to reduce the variance in the
predictive performance of explicit algorithms.
For explicit algorithms, one way to estimate the information content of a noisy input is to
use a recursive Bayesian updating scheme [Pearl 1988]. In the Bayesian approach, we rewrite the
3
expression for ζr from equation (7) as equation (8) using the chain propagation rule ([Pearl
1988] p.154). Because we start with a clean training set, the information content of an input in its
true state can be estimated. Note that we use the property of conditional independence wherein
the true state separates the class and the observed state ([Pearl 1988] p.154]).
q
~
~
~
~
~ = c | X~ = x ∧ Π
ζ r = ∑ P[ X kt = x ki | X ko = x kj ∧ Π o ] P[ ψ
r kt
ki
o].
=
i 1
(8)
In equation (8), the probability of a class given the current true state and the history of
observed states can be written as:
~
~ = c | X~ = x ∧ Π
P[ ψ
r kt
ki
o] =
~
~ ~
~
~ = c | X~ = x ∧ Π
P[ ψ
∑
r kt
ki
t ] P[Π t | X kt = x ki ∧ Π o ]
~
Πt ∈π t
(9)
is the random vector for the combination of previously observed variables
where Π
t
(without noise) and π t is the set of true state vectors.
Equation (9) demonstrates the difficulty of a Bayesian approach. The first term of the
right hand side of (9) must be computed O((p-d)(qd)) times where d is the node or input level
(root node is level 0) in a decision tree, p is the number of inputs, and q is the average number of
input states. The cardinality of π d is qd as π d contains all possible true states of all inputs on the
path. The joint probability calculations must be repeated for all remaining inputs (p-d). Since the
3
The chain propagation rule provides the relation: P ( A| B ) =
∑ j P( A| B, C j ) P(C j | B)
Page 14
depth of a tree is partially dependent on the number of inputs, a recursive Bayesian updating
approach is impractical for computational reasons.
As an alternative to a Bayesian updating approach, noise effects can be propagated during
tree construction by introducing noise into partitions by randomly scrambling cases using the
parameters C and W. This amounts to computing the class probabilities using a partition instead
of calculating the probabilities in (9). A formal description of the random scrambling procedure
follows.
Procedure Random Scrambling
Input
Z: a partition
X: branching input with q states
C: noise level
Output
S: a ‘scrambled’ set of partitions of input X, S = {S k S k is partition of S} k = 1,2,..., q
Procedure
1. Initialize each S k ∈ S to φ .
2. Let T be the set of true partitions resulting from splitting Z into q partitions on input X.
3. For each partition Tk ∈ T do
3.1. For each case τ ∈Tk do
3.1.1. Let r be a random number in [0,1].
3.1.2. If r ≤ C then set S k : = S k ∪ τ else randomly choose S j , j ≠ k and
set S j := S j ∪ τ .
In contrast to the Bayesian updating approach, the complexity of injecting noise through
random scrambling is O((p-d)q) because q scrambling operations for each input are necessary and
only the current input needs to be scrambled. The effect of noise from all but the current input on
the path is already included in the starting noisy partition.
Figure 2 illustrates the random scrambling procedure. The root box depicts a partition of
size 100 with 66 cases expected of class c1 and 34 cases expected of c2. Input Xk is selected and
partitioned by its three states where the expected size and class frequencies of the partitions are
Page 15
4
shown in the boxes without parentheses . Noise is then introduced by scrambling all the
partitions of Xk according to the correct noise probability C=0.8. For each case in a partition, a
random number is drawn to determine if the case should be moved to a partition associated with
another state of the input. If the number is less than or equal to C, the case remains in its original
partition. Otherwise the case is randomly moved to a partition associated with another state of
the input. After scrambling, the size and class frequencies of each partition are typically more
uniform because of the impact of noise. In Figure 2, the scrambled partitions are beneath the
clean partitions. The lines indicate that cases can be moved from a clean partition to any noisy
partition.
100
c1: 66, c2: 34
Xk
xk3
xk1
xk2
60
c1 : 48, (24.96)
c2 : 12, (10.56)
30
c1 : 10, (8.92)
c2 : 20, (16.06)
10
c1 : 8, (7.36)
c2 : 2, (1.96)
Initial Partitions
52
c1: 40.2, (25.43) {16.14}
c2: 11.8 (10.85) {6.94}
31
c1: 13.6, (12.66) {6.03}
c2: 17.4 (14.88) {10.4}
17
c1: 12.2, (11.54) {4.73}
c2: 4.8 (5.05) {1.52}
Scrambled Partitions
(C = 0.8)
Figure 2: Example of the Random Scrambling Process
4
Numbers inside round and curly brackets are the variances that are explained in subsection 3.4.
Page 16
The explicit algorithm ID3ep uses the random scrambling procedure to introduce noise into
partitions. First, the usual partitioning process of the ID3 algorithm is performed. Second, the
partitions created in the first step are scrambled using the random scrambling procedure. After all
partitions of an input are scrambled, the normal formulas for the input selection, stopping rule,
and classification function are used. Note that we retain the pessimistic pruning procedure in
ID3ep. Thus, the only difference between ID3p and ID3ep is the way that noise is treated. In ID3p,
the treatment of noise is indirect through a sample of the data collection process. In ID3ep, the
treatment of noise is direct through random injection of noise in a clean training set using the
parameters C and W. Despite the differences in handling noise, the expected behavior of both
algorithms is governed by equation (7). This observation is similar to the results of previous
research comparing input selection criteria [Mantaras 1991].
3.4. Impact of Noise on Variance
As discussed in subsection 3.3, explicit algorithms have no advantage over implicit
algorithms in terms of expected predictive ability. The advantage of explicit noise handling lies in
the potential to make performance more stable. In this section, we study the impact of noise on
the variance in class frequencies between a binomial noise process and a constant noise process.
The binomial noise process represents ID3p and ID3ep in which noise is randomly introduced
either in the training set or through the random scrambling process. The constant noise process
represents a controlled variation of ID3ep (ID3ecp) in which class frequencies in noisy partitions are
set to their estimated expected values. As the analysis elucidates, the variance in size due to noise
is much less in the constant noise process than in the binomial noise process.
We begin with the sampling variance common to both noise processes limiting our focus
to a single input, state, and class. Let,
P[ X kt = x kj ] = γ j and P[ ψ = cr | X kt = x kj ] = ρ rj be population parameters and
s( L ( Zn , X kt = x kj )) be the size of the partition Zn where X kt = x kj .
Page 17
The expected value and variance of the size of a partition of size n where X kt = x kj are
defined in equations (10) and (11), respectively. For each case, the sampling process selects
X kt = x kj with probability γj. Thus, the size of a partition is binomially distributed with mean and
variance as shown in equations (10) and (11).
E ( s( L( Zn , X kt = x kj ))) = nγ j
~ ~
V ( s( L( Z n , X kt = x kj ))) = nγ j (1- γ j )
(10)
(11)
To analyze the mean and variance of the sub partition size where ψ = cr , we need to apply
Wald's theorem [Ross 1970 -- page 37]. Wald's theorem defines the expected value and variance
~
~
of the sum of K random variables where K is also a random variable. The expected value of the
~
~
~
~
sum of K independent, identically distributed random variables ( X i ) is given by E ( K ) E ( X i ) .
~
~
~
~
The variance of the sum is given by E ( K )V ( X i ) + V ( K )( E ( X i )) 2 . Here, the expected size of the
beginning partition is nγ j as defined in (10). Within this partition, the probability of ψ = cr is
ρ rj . Applying Wald's theorem, the mean and variance of the sub partition size are defined by (12)
and (13). In (13), we assume that the class random variables of the individual cases within the
partition are independent and identically distributed.
E ( s( L( Zn , ψ = cr ∧ X kt = xkj ))) = nγ j ρ rj = µ rj
~ ~
~
= cr ∧ X kt = x kj ))) = nγ j ρ rj (1 − ρ rj ) + nγ j (1 − γ j )ρ 2rj = σ 2rj
V ( s( L( Z n , ψ
(12)
(13)
Now consider the effect of the binomial noise process common to ID3p and ID3ep. This
process introduces more variance in the class frequency because cases are assigned at random to
partitions based on the parameter C. The expected value and variance of the size of the above
partition after noise is given in equations (14) and (15) where the noise is a binomial process with
mean and variance (C, (1-C)C). Note that these formulations involve another application of
Wald's theorem because once again the size of the beginning partitions is uncertain as defined in
~
(12) and (13). In (14) and (15), nγ j ρ rj replaces E ( K ) for correct input states and
~
nγ i ρ ri replaces E ( K ) for q-1 incorrect input states. For correct input states, C replaces
Page 18
~
~
~
E ( X i ) and W replaces E ( X i ) for q-1 incorrect states. In (15), σ 2rj from (13) replaces V ( K ) for
~
correct input states and σ 2ri replaces V ( K ) for q-1 incorrect states. For correct input states, C(1~
~
C) replaces V ( X i ) and W(1-W) replaces V ( X i ) for q-1 incorrect states.
~ ~
~
E ( s( L ( Z n , ψ
= cr ∧ X ko = xkj ))) = nγ j ρ rj C +
q
∑ nγ i ρ riW
i≠ j
q
= Cµ rj + W
~ ~
~
V ( s( L( Z n , ψ
= cr ∧ X ko = x kj ))) =
(14)
∑ µ ri
i≠ j
(15)
q
2
C (1 − C )µ rj + C 2 σ rj
+ W (1 − W )
∑
i≠ j
q
µ ri + W 2
∑ σ ri2
i≠ j
Now consider the effect of the constant noise process associated with the controlled
scrambling algorithm ID3ecp. Here, the class frequency is set to its expected value. Therefore, the
variance due to noise disappears. However, the variance of the sampling process remains.
Equation (16) for the partition size with a constant noise process is derived using the variance of a
constant times a function of a random variable. Alternatively, the variance can be derived by
dropping the first and third terms in (15) because the constant noise process has zero variance.
Note that the expected value remains as defined in equation (14).
~ ~
~
V ( s( L( Z n , ψ
= cr | X ko = x kj ))) = C 2 σ 2rj + W 2
q
∑ σ 2ri
(16)
i≠ j
To further depict the difference between the binomial noise process and the constant noise
process, consider a numerical example based on Figure 2 where the population statistics are given
below.
~
~
~
P[ X kt = x k 1 ] = 0.6 , P[ X kt = x k 2 ] = 0.3 , P[ X kt = x k 3 ] = 01
.
~
~
~
~
~ = c | X~ = x ] = 0.8
P[ ψ = c1 | X kt = x k 1 ] = 08
. , P[ ψ = c1 | X kt = x k 2 ] = 0.33, P[ ψ
kt
k3
1
~
~
~
~
~
~
P[ ψ = c2 | X kt = x k 1 ] = 0.2 , P[ ψ = c2 | X kt = x k 2 ] = 0.67 , P[ ψ = c2 | X kt = x k 3 ] = 0.2
The labeling of the boxes in Figure 2 shows the expected values for the input state and
class in the clean and noisy partitions. The variances for the binomial (equation (15)) and
Page 19
constant noise (equation (16)) processes are shown in round and curly brackets, respectively, next
to the expected values (equation (14)) in the noisy partitions. Note the reductions for the
constant noise process.
Figure 3 shows a graphical view of the difference between the constant and binomial noise
processes for a particular class. Here there are two curves corresponding to partitions at level 4
in a decision tree. The curves for the binomial noise process are based on (15), while the curve
for the constant noise process is based on (16). For the binomial noise process, Figure 3
demonstrates that the coefficient of variation increases as the noise level increases with the peak
about 1/q. Although not shown here, the difference between the coefficient of variation of the
two processes increases as the depth of the tree increases because observing more noisy inputs
introduces additional uncertainty.
1.8
Binomial Noise
Process
1.6
1.4
1
Constant Noise
Process
0.8
0.6
0.4
0.2
C
Figure 3: Graphical Comparison of Noise Processes of a Node at Level 4
1
0.95
0.9
0.85
0.8
0.75
0.7
0.65
0.6
0.55
0.5
0.45
0.4
0.35
0.3
0.25
0.2
0.15
0.1
0.05
0
0
CV Size
1.2
Page 20
The goal of reducing the variance of the sub-partition sizes is to make the input selection
process more stable and ultimately to reduce the variance in performance on a set of unseen cases.
Analytically, it is rather difficult to measure the variance in the gain because distribution
assumptions are necessary and the mathematics is complicated. Since any distribution
assumptions would rarely be met, it seems better to make a strong statement about the subpartition sizes rather than a weak statement about the gain. Even with a strong statement about
the variance of the gain, simulation experiments are still necessary to link the theoretical behavior
with performance on unseen cases.
3.5. Controlled Scrambling Procedure
The controlled scrambling procedure is designed to behave as close as possible to the
constant noise process and thereby to reduce the variance in the gain and ultimately, the
performance on unseen cases. The differences between the controlled scrambling procedure and
the constant noise processes are due only to rounding of fractional values and conserving the
number of cases in various partitions of the training set. The controlled scrambling procedure first
computes the class and true input state frequencies as close as possible to their estimated values
and then randomly assigns cases to match the computed frequencies. Control of class frequencies
follows from the discussion in Section 3.4.
The reason for controlling the true state frequency is a little subtle, however. The true
state distribution shows the fraction of each true state within a noisy partition. For example in
Figure 2, the true state frequencies in the noisy partition xk1 are 48 (60*0.8) for true state xk1, 3
(30*0.1) for true state xk2, and 1 (10*0.1) for true state xk3. Controlling the true state frequency
does not affect the variance of the class frequency in the current node, but rather it potentially
impacts the class frequencies in descendant nodes. The true state frequency is important to
control when the next input to select in the decision tree is conditionally dependent on the current
input state. When there is a strong dependence, reducing the variance in true state frequencies
Page 21
will reduce variance in the class frequencies of the next input. For example, the constant noise
curve in Figure 3 was generated with a strong dependence between the current input state and the
next input state. Thus, controlling the true state frequencies is a matter of reducing variance in
descendant nodes rather than in the current node.
Computation of the class and true state frequencies is accomplished by solving two
optimization models. First, the controlled scrambling procedure assigns class frequencies such
that the assignment minimizes the distance between the assigned class frequencies and estimated
class frequencies in a set of partitions (all the partitions of an input) subject to integer, nonnegativity, and case conservation constraints. The latter constraints ensure that the total number
of cases is the same before and after allocation by the controlled scrambling procedure. Second,
the controlled scrambling procedure assigns true state frequencies such that the assignment
minimizes the distance between the assigned true state frequencies and the estimated true state
frequencies in a subset of a partition (all cases of the same class) subject to integer, nonnegativity, and case conservation constraints. The latter constraints are based on the assignment
made by the class frequency optimization model.
Let
d ( cr , j ) be the assigned frequency of class cr in the noisy partition where X ko = xkj .
s( cr , L ( Z , X kt = x kj )) be the frequency of class cr in the partition Z where X kt = x kj .
e(cr , j ) be the estimated frequency of class cr in the noisy partition where X ko = xkj .
q
e(cr , j ) = C * s(cr , L( Z , X kt = x kj )) + W
∑ s(cr , L( Z , X kt = x ki ))
i≠ j
d ( xki , r , j )be the assigned frequency of true state xki in the noisy partition where
X ko = xkj and the class is cr.
e( x ki , r , j ) be the estimated frequency of true state xki in the noisy partition where
X ko = xkj and the class is cr.
e( x ki , r , j ) = C * s(cr , L( Z , X kt = x ki )) if i = j
e( x ki , r , j ) = W * s(cr , L( Z , X kt = x ki )) if i ≠ j
Page 22
The class frequency optimization model solves m optimization problems CFr, r=1,...,m.
In each problem, there are q decision variables, d ( cr , j ) , j=1,...,q.
Problem CFr :
 q

2

Min
(d (cr , j ) − e(cr , j)) 

 j =1

s. t. d (cr , j ) ≥ 0 and integer, ∀ j = 1,..., q
∑
q
∑
(17)
q
d ( cr , j ) =
j =1
∑ s(cr , L( Z , X kt = x kj ))
- - case conservation constraint
j =1
The true state frequency optimization model solves mq optimization problems TSFrj,
r=1,...,m, j=1,...,q (one for each combination of class and observed state). In each problem, there
are q decision variables, d ( xki , r , j ), i=1,...,q.
Problem TSFrj :
 q
2
Min (d ( x ki , r , j ) − e( x ki , r , j )) 
 i =1

s. t. d ( x ki , r , j ) ≥ 0 and integer ∀i = 1,..., q
∑
(18)
q
∑ d ( x ki , r , j) = d * (cr , j)
i =1
- - case conservation constraint
where d * (cr , j ) is the optimal value of d (cr , j ) in CFr
The controlled scrambling procedure initially assigns class frequencies (d ( cr , j ) ) by
rounding the estimated class frequencies to the nearest integer value. If the case conservation
constraint is not satisfied for a set of assigned class frequencies, the assigned class frequencies are
adjusted in the order that minimizes the distance from the estimated class frequency until the
constraint is satisfied. A similar procedure is used for the assigned true state frequencies. More
precisely, the algorithm to compute the assigned class frequencies is shown in procedure Assign
Class Frequencies.
Page 23
Procedure Assign Class Frequencies
Input
Z : a partition
X k : branching input with q states
C: noise level
Output
D: a set of assigned class frequencies for Z and Xk,
D = {d (cr , j ) d (cr , j ) is an assigned class frequency r = 1,2,..., m; j = 1,2,..., q}
Procedure
1.
For each class cr and each state xkj of Xk, set d ( cr , j ) = rounded ( e( cr , j )) .
2.
For each set of assigned frequencies of a class ({d ( cr , j ) | j = 1... q} ), adjust them such that
the case conservation constraint is satisfied. Stop when the case conservation constraints
are satisfied for all sets of assigned class frequencies.
2.1 If
q
q
j =1
j =1
q
q
j =1
j =1
∑ d (cr , j ) > ∑ s(cr , L( Z , X kt = xkj )) , then adjust by taking away from some
d ( cr , j ) . Sort d ( cr , j ) by descending order of (d (cr , j ) − e (cr , j )) . Starting from the
d ( cr , j ) with the largest difference, compute new d ( cr , j ) = old d ( cr , j ) - 1 until the
case conservation constraint is satisfied.
2.2 If
∑ d (cr , j ) < ∑ s(cr , L( Z , X kt = xkj )) , then adjust by adding to some d (cr , j ) .
Sort d ( cr , j ) by ascending order of (d (cr , j ) − e (cr , j )) . Starting from the d ( cr , j )
with the smallest difference, compute new d ( cr , j ) = old d ( cr , j ) + 1 until the case
conservation constraint is satisfied.
The procedures to assign the class and true state frequencies are optimal and polynomial in
the number of states and classes. The procedures start with the best assignment (rounded
estimated values) and adjust the best assignment until a feasible value is obtained. The
adjustments are always the smallest deviation from the best assignment. Because there is no
interaction among the decision variables, the resulting assignment minimizes the sum of squared
deviations subject to case conservation constraints. The worst case complexity of the class
frequency assignment algorithm is O(mqlog(q)) for each input because there are m lists of
decision variables where each list must be sorted (qlog(q)). The operations of computing the
rounded estimates and adjusting the estimates can be performed in time linear to the number of
Page 24
states. Similarly, the worst case complexity for the true state frequency assignment algorithm is
O(mq2log(q)) because there are mq decision variables. The total worst case complexity is the sum
of the above worst case complexities because the two procedures are independently performed.
The controlled scrambling procedure uses the algorithms to assign class and true state
frequencies. After the class and true state frequency assignments are made, the controlled
scrambling procedure randomly selects sets of cases from the true partitions and assigns them to
scrambled partitions to satisfy the class and true state frequencies. In contrast, the random
scrambling procedure assigns individual cases to scrambled partitions without the constraints of
the class and true state frequencies. Formally, the algorithm to introduce noise in a controlled
manner is presented in procedure Controlled Scrambling.
Procedure Controlled Scrambling
Input
Z : a partition
X k : branching input with q states
Output
S: a ‘scrambled’ set of partitions of input X, S = {S j S j is partition of S} j = 1,2,..., q
Procedure
1. Perform procedure Assign Class Frequencies.
2. Perform procedure Assign True State Frequencies.
3. Let T be the set of true partitions resulting from splitting Z into q partitions on input X k .
T = {Ti Ti is a partition of T where X k = x ki } .
4. For each j = 1,2,...,q do
4.1. For each i = 1,2,...,q do
4.1.1 For each r = 1,2,...,m do
4.1.1.1 Define Tir , such that Tir is a partition of Ti where the class is cr
4.1.1.2. Let S jr be d ( xki , r , j ) randomly selected cases without replacement from
Tir . S j = S j ∪ S jr
The explicit algorithm ID3ecp uses the controlled scrambling procedure to introduce noise
into partitions. First, the usual partitioning process of the ID3 algorithm is performed. Second,
the partitions created in the first step are scrambled using the controlled scrambling procedure.
Page 25
After all partitions of an input are scrambled, the normal formulas for the input selection, stopping
rule, and classification function are used. Note that we retain the pessimistic pruning procedure in
ID3ecp.
4. Comparison of Algorithm Performance
In this section, we describe simulation experiments that investigate the performance of the
implicit (ID3p) and explicit algorithms (ID3ecp) in terms of average accuracy and coefficient of
variation (CV) of accuracy. We describe the hypotheses, performance measures, data sets,
methodology, and results.
4.1. Hypotheses
The hypotheses extend the analytical results of Section 3 to the performance of the
implicit and explicit algorithms. Section 3 established several relationships between the implicit
and explicit noise handling approaches: (i) same expected behavior on the gain, (ii) lower variance
in the class frequency of a partition for a constant noise process than a binomial noise process,
and (iii) difference in variance increases as noise level increases. Because it is difficult to
analytically link the performance of the algorithms to the noise handling approach, simulation
experiments are necessary. We feel that these three results will extend to the performance of the
implicit and explicit algorithms as stated below.
Hypothesis 1: There is no difference in expected performance between the implicit algorithm
ID3p and the explicit algorithm ID3ecp.
Hypothesis 2: The explicit algorithm (ID3ecp) has a smaller coefficient of variation in
performance than the implicit algorithm (ID3p).
Hypothesis 3: The difference in the coefficient of variation in performance of the explicit
algorithm (ID3ecp) minus the implicit algorithm (ID3p) increases as the noise level increases
over a range of reasonable noise levels.
Page 26
4.2. Performance Measurement
We use two measures of performance: classification accuracy, a traditional measure and
relative information score, a more refined measure. Classification accuracy (ratio of correctly
classified cases to total cases) does not account for the effects of the number of classes and the
prior probabilities of each class. The Relative Information Score (RIS) [Kononenko and Bratko
1991] measures the percentage of the uncertainty of the data set that is explained by the learning
algorithm. Thus, a high RIS value is preferred to a low value. In equation (19), RIS is computed
as the amount (i.e., the number of bits) of uncertainty removed by the classification process (Ia)
divided by the amount of uncertainty in the data set before classification (the entropy of the
distribution E).
I
RIS = a * 100%
E
(19)
where Ia is the Average Information Score computed as
1 n
Ia = * ∑ I j
T 1
where T is the size of the test set and Ij is the information score of case j where
 − log 2 P(C ) if a correct classification is made
Ij =
 log 2 (1 − P(C )) if an incorrect classification is made
where P(C) is the prior probability of class C (determined from the data set).
4.3. Data Sets
To provide a range of case distributions, we executed the experiments with five data sets,
two real and three artificially generated. The Bankruptcy data set [Liang 1992] contains 50 cases,
8 inputs, and 2 equally distributed classes (bankrupt or healthy). The Lymphography data set
[Murphy and Aha 1991] was developed through data collection at the University Medical Centre,
Institute of Oncology in Ljubljana, Yugoslavia. To reduce the effects of spurious noise, we
removed cases with missing values, removed redundant cases, and removed all but one among
Page 27
conflicting cases. After this cleansing activity, the Lymphography data set contains 148 cases
distributed among 4 classes where two classes are very sparse (2 and 4 cases, respectively). The
Lymphography data set contains 18 inputs with an average of 3.3 states per input where the
number of states ranges from 2 to 8.
The artificial data sets were generated by a program based on the specifications described
in [Bisson 1991]. The data set generator can control the number of cases, classes, inputs, states
per input, the distribution of cases among classes, and the complexity of the true rule sets for each
class. Data set 1 contains 4 equally distributed classes with 10 inputs. Data set 2 contains 8
moderately skewed classes and 15 inputs. Data set 3 contains 12 highly skewed classes with 20
inputs. In data set 2, two classes have 50% of the cases and the remainder of the cases are
uniformly distributed among the other 6 classes. In data set 3, 80% of the cases are distributed to
3 classes and the remainder are uniformly distributed to the other 9 classes. In the artificial data
sets, the number of cases was 200, the average number of states per input was 3, and the average
size of the true rule sets was 2 rules per class and 3 conjunctive terms per rule.
4.4. Experimental Design
5
Table 1 shows a 2 X 3 factorial design for the algorithm and noise level factors. The
numbers in the cells show the observations for each treatment. As shown in Table 1, we choose 3
levels of noise: high (C = 0.65), moderate (C = 0.80), and low (C = 0.95). The low level of noise
is close to perfect measurement. The moderate level of noise causes a significant decrease in
predictive performance. The high noise level causes a further significant decrease in predictive
performance. Further decline in predictive performance from higher levels of noise is not as
5
The algorithms and simulation experiment were implemented using Microsoft C on a 486 personal
computer.
Page 28
significant. In separate experiments, the entire range of noise levels is studied to graphically
depict the functional relationship between noise and the mean and variance of performance.
There are two experiments corresponding to the dependent variable average RIS and
coefficient of variation (CV) of RIS. The method used to estimate performance is the standard
test-sample method [Breiman et al. 1984]. Each observation is the average performance of the
same 30 splits of a data set where the data set is divided roughly 70% for training and 30% for
testing. Each cell of both experiments uses the same set of 100 noise perturbations. In a
perturbation, the data set is randomly changed using the given C and W values. ID3p is given a
perturbed training set while ID3ecp is given a clean training set. Both algorithms use the same test
set and the same perturbed training set in the pessimistic pruning procedure. In the average
experiment, all observations are used. In the CV experiment, each cell contains the same 30
6
random samples of size 30 from the 100 observations. An observation is the CV computed from
the given sample.
Algorithm
ID3p
ID3ecp
Table 1: Experimental Design
Noise Level (C)
L (.95)
M (.80)
100 (AVG)
100 (AVG)
30 (CV)
30 (CV)
100 (AVG)
100 (AVG)
30 (CV)
30 (CV)
H (.65)
100 (AVG)
30 (CV)
100 (AVG)
30 (CV)
4.5. Results
Analysis of variance was used to determine whether there are performance differences
between the algorithms across the 5 data sets. Tables 2 - 5 report the analysis of variance results
for the Lymphography data set and data set 1 (GD1). The ANOVA tables show that the
6
A sample is a random selection of 30 perturbations. Each cell of the CV experiment contained the same
30 perturbations.
Page 29
simulation results are consistent with the theoretical analysis in Section 3. The noise level (NL)
affects both dependent variables (AVG RIS and CV RIS), but the algorithm (ALG) and the
interaction term (NL*ALG) are not significant for AVG RIS (Tables 2 and 4). However, ALG
and the interaction term are significant for CV RIS (Tables 3 and 5). Thus, the ANOVA results
confirm Hypothesis 1 and support Hypotheses 2 and 3. Similar results were obtained for the
other data sets (see Table 6).
Tables 7 and 8 list mean CV differences between the algorithms (first order differences) at
each noise level and mean differences between the algorithms at adjacent noise levels (second
order differences). Because both the first order and second order differences are significant,
Hypotheses 2 and 3 are confirmed.
Table 2: AVG Lymphography ANOVA Results
SOURCE
SS
df
MS
F
NL
ALG
NL*ALG
WITHIN
TOTAL
73785.8149
39.7113688
31.9080444
17310.1319
91167.5661
2
1
2
594
599
36892.9074
39.7113688
15.9540222
29.1416361
P-value
1265.98614
1.3627021
0.54746488
SOURCE
Table 3: CV Lymphography ANOVA Results
SS
df
MS
F
P-value
NL
ALG
NL*ALG
WITHIN
TOTAL
1.72273569
0.52293785
0.16348104
0.22173633
2.63089091
SOURCE
NL
ALG
NL*ALG
WITHIN
TOTAL
2
1
2
174
179
0.86136785
0.52293785
0.08174052
0.00127435
675.928939 9.1635E-83
410.357584 1.212E-47
64.1430741 1.3533E-21
Table 4: AVG GD1 ANOVA Results
SS
df
MS
F
230805.371
0.74741892
4.5854718
8725.97472
239536.678
2
1
2
594
599
115402.685
0.74741892
2.2927359
14.6901931
6.368E-215
0.24353811
0.57870587
7855.76366
0.05087877
0.15607255
P-value
0
0.82161873
0.85553219
Page 30
SOURCE SS
NL
ALG
NL*ALG
WITHIN
TOTAL
Table 5: CV GD1 ANOVA Results
df
MS
F
0.26582049
0.0891291
0.01144605
0.04442473
0.41082037
2
1
2
174
179
0.13291025
0.0891291
0.00572303
0.00025531
520.574572
349.095311
22.4155911
P-value
3.6729E-74
1.9006E-43
2.1791E-09
Table 6: P-Values for Other Data Sets
Bankruptcy
GD2
NL
ALG
NL*ALG
AVG
1.5053E-62
0.6006
0.9231
CV
5.7346E-56
2.2906E-68
1.7721E-33
AVG
0.0000
0.8216
0.8555
CV
2.3041E-95
9.0516E-64
2.3986E-22
GD3
AVG
0.0000
0.4669
0.0225
CV
4.6106E-76
2.5253E-39
3.0017E-18
Table 7: Lymphography CV Mean Differences
ID3p
ID3ecp
t-value
p-value
H
M
L
M-H
L-M
H
M
L
M-H
L-M
0.429047734
0.268555164
0.11611841
-0.160492571
-0.152436757
0.251359557
0.153436962
0.08552477
-0.097922595
-0.067912194
11.53771308
15.87069539
6.540714774
-3.638112396
-10.34340997
1.16936E-12
3.88396E-16
1.8278E-07
0.000529124
1.52555E-11
Table 8: GD1 CV Mean Differences
ID3p
ID3ecp
t-value
p-value
0.1812388
0.1197556
0.0678872
-0.061483251
-0.051868367
8.071E-13
2.786E-16
3.47781E-14
0.009954073
2.19337E-07
0.1180832
0.0735932
0.043692
-0.044490058
-0.029901399
11.717007
16.073779
13.31519161
-2.464027269
-6.473220833
4.6. Discussion
To depict the magnitudes of the differences in performance, we generated simulation data
to graphically compare the performances. In a simulation run, each observation was computed
from the same 20 splits and 20 perturbations. In Figures 4 and 5, the average performance graphs
of GD1 almost coincide as expected. In Figures 6 and 7, the average performance graphs for the
Lymphography data set are more erratic as the graphs cross numerous times. This slightly erratic
Page 31
behavior is probably due to the increased level of residual variation in the Lymphography data set
as its maximum RIS is slightly less than 50% compared to more than 80% for GD1. The shape of
the average performance graphs provides evidence to support Proposition 1 in Section 3.2. Note
that the average RIS and accuracy is minimized near 1/q and the bowl-like shape of the curves.
For GD1, the average number of states is just below 3. For the Lymphography data set, the
average number of states is 3.3 and there is a larger variation in the number of states (2 of the
inputs have 8 states). Because ID3 favors inputs with many states, the inputs with 8 states
probably appear on most paths in the decision tree. This explains why the Lymphography graph is
minimized below the simple average number of states from the data set.
As for the CV of performance, Figures 4 through 7 show a strong separation between the
implicit and explicit algorithms. In addition, the CV of performance increases as noise increases
and the difference between the explicit and implicit algorithms increases as the noise level
increases. The shape of the curves and their extreme point is consistent with the theoretical
graphs shown in Figure 3. There is larger separation between the explicit and implicit graphs in
the Lymphography data set (Figures 6 and 7) than in the GD1 (Figure 4 and 5). The difference at
low levels of noise is obscured in Figures 6 and 7 because the scale is wider. However, the
difference between the plotted points is larger for the Lymphography data set than GD1 even at
low levels of noise. Note also that the shape of the CV graphs in Figures 6 and 7 differ because
the CV RIS graph (Figure 6) is affected by some negative RIS values.
To probe the sensitivity of ID3ecp, we generated additional simulation data for GD1.
Here, the true noise level (C) used in the test set and in the training set of ID3p differed from the
false noise parameters given to ID3ecp (C'). In the constant error sensitivity runs, the false noise
parameter C' was misstated by 0.10 (either all high or all low in a run) except near the end points
(C = 0 or 1). In varying error sensitivity runs, the mean of the false noise level C' was equal to
the true noise level C but variance in a range of ±0.10 or ±0.05 of the true C value was
Page 32
introduced. Somewhat surprisingly, the average performance of the algorithms in each sensitivity
run shows little difference than in Figure 4. In Figure 8, the CV of performance continues to
demonstrate a large advantage for ID3ecp for both constant error runs (high and low). In Figure
9, the CV advantage of ID3ecp has disappeared at high values of C in the ±0.10 case. In the ±0.05
case, however, there is still a marked advantage for ID3ecp. Figure 9 demonstrates that CV of
performance of ID3ecp is relatively sensitive to accurate noise level assessments as compared to an
implicit algorithm using a training set drawn from the same noise process as unseen cases. Thus,
the stable behavior of explicit algorithms may deteriorate due to the variance caused by incorrect
specification of the noise level. When the variance in incorrect specification is high, the noise
variance acting on implicit algorithms may be balanced by the incorrect specification variance
acting on explicit algorithms.
5. Summary and Conclusions
We compared decision tree induction algorithms under conditions of noisy input data
where the level of noise was either implicitly known through a sample of the noise process or
explicitly known through an external parameter. Explicit measurement of noise is cost effective
when training data is provided directly by experts or when the cost of directly estimating the level
of noise is less than the cost of sampling a representative noisy process. In addition, the appeal of
explicit noise measurement broadens when there is an associated performance benefit. Here, we
demonstrated that explicit noise measurement can be accompanied by more stable performance on
unseen cases.
Our primary contributions were to develop an explicit noise handling algorithm (ID3ecp)
and demonstrate its advantages as compared to a standard implicit noise algorithm (ID3 with the
pessimistic pruning procedure). ID3ecp injects random noise into partitions but controls the class
and true state frequencies in a partition as close as possible to their estimated values. The aim of
Page 33
the controlled scrambling procedure is to reduce the variance in the partitioning behavior. We
demonstrated that the implicit and explicit algorithms have the same expected behavior and
performance, but the explicit algorithm has more stable behavior and performance. The
behavioral results were demonstrated by an analytical comparison of a constant noise process as
the best case for a controlled partitioning process and a binomial noise process for implicit noise
measurement. The performance of the algorithms was demonstrated by simulation experiments to
compare the average performance and coefficient of variation of performance for the two
algorithms.
This research is part of our long term interest concerning the economics of expert systems.
Two direct extensions of this work are treating the noise level as a decision variable rather than a
constraint and developing induction algorithms that combine mean and variance of performance.
In the former topic, explicit measurement of noise is required to make a tradeoff among the cost
of removing noise with the benefit of improved decision making. Other topics not directly related
to this work are optimizing expert system performance over a multi-period horizon and
developing cost-benefit objectives for other approaches such as Bayesian reasoning networks.
Page 34
Appendix A
Convexity of Gain with Respect to Noise Level
Proposition 1
~ ~
∂g ( X ko | Z C )
> 0, C ∈ (1 / q ,1]
∂C
= 0, C = 1 / q
< 0, C ∈[0,1 / q )
(6.1)
(6.2)
(6.3)
We prove (6.2) and demonstrate the truth of (6.1) and (6.3) for a special case. Let us
begin by showing the second condition, namely, the slope is zero at C = 1/q. With some algebra,
first derivative of the gain function with respect to C (A1) can be derived.
m q
pr + α ( qp j prj − pr )
∂g ( X ko | Z C )
1
=
( qp j prj − pr ) log 2
∂C
q − 1 r =1 j =1
1 + α ( qp j − 1)
where ,
p j = P ( X kt = x kj | Z C )
pr = P ( ψ = cr | Z C )
prj = P( ψ = cr | Z C ∧ X kt = x kj )
α = (1 − Wq )
∑∑
(A1)
For C = 1/q, α = 0. Substituting α = 0 in the above equation for the slope we obtain:
∂g ( X ko | Z C )
1
=
∂C
1− q
q
m
∑
r =1
log 2 ( pr )
∑
q
( qp j prj − pr ) = 0 as
j =1
The second condition in the proposition is proved.
∑ (qp j prj − pr ) = qpr − qpr = 0
j =1
†
For (6.1) and (6.3), we show that the second derivative is greater than zero in a special
case. With some algebra, the second derivative of the gain with respect to W can be derived as
(A2). We must show that the second derivative is greater than 0 as stated in (A3).
Page 35
∂ 2 g ( X ko | Z C )
∂ 2W
where W
δ rj
δj
q
q
=−
3
δ rj pr + qδ rj
m
q
δ j (1 − qδ j )
∑ ∑ Wp r + (1 − Wq )δ rj + ∑ W + (1 − Wq )δ j
j =1 r =1
= q1-−C1
= qp j prj − pr
= qp j − 1
δ j (1 − qδ j )
8
q
m
j =1
3
δ rj pr + qδ rj
8
∑ W + (1 − Wq )δ j ∑ ∑ Wp r + (1 − Wq )δ rj
>
j =1
(A2)
(A3)
j =1 r =1
Because
∑r δ rj = δ j ,we assume that one of the δrj terms equals δj and the other m-1
terms be divided into pairs such that the sum of each pair is zero and each pair member is the
same absolute value. In addition, we assume that pr (= µ) is constant for all r. Using these
assumptions and some algebra, the right hand side of (A3) can be reduced to a quantity less than
(A4).
q
δ j (µ − qδ j )
∑ W µ + (1 − Wq )δ j
(A4)
j =1
Substituting (A4) into (A3) and using some further algebra, (A3) can be reduced to (A5).
q
δ2 j
>
D
j =1
∑
q
δ2 jµ
where D = (W µ + (1 − Wq )δ j )(W + (1 − Wq ) δ j )
D
j =1
∑
(A5)
(A5) is true because δ 2j > 0 and µ < 1 and D > 0. This proves (6.1) and (6.3) for the
special case stated above.
†
Page 36
Appendix B
Pessimistic Pruning Procedure
Appendix B describes the pruning procedure used in the algorithms ID3p and ID3ecp. This
description has been adapted from [Quinlan 1987].
For any given tree T, generated using a training set of N cases, let some leaf in the tree
account for K of these cases with J of them misclassified. The ratio J/K does not provide a
reasonable estimate of the error rate when classifying unseen cases [Quinlan 1987]. A more
reasonable estimate is obtained using the continuity correction factor for the binomial distribution,
wherein J is replaced by J+0.5 [Snedecor and Cochran 1980].
Let S be a subtree of T with LS leaves, and let JS and KS be the corresponding sums of
errors and cases classified over S. Using the continuity correction factor, the expected number of
cases (MS) misclassified by S out of KS unseen cases should be:
M S = J S + 0.5 LS
The standard error of MS, Se(MS), is given by:
Se( M S ) =
M S * ( KS − M S )
KS
Let E be the number of cases misclassified out of KS if the subtree S is replaced by its best
leaf. The pessimistic pruning procedure replaces S by its best leaf if:
E + 0.5 ≤ M S + Se( M S )
In pessimistic pruning, all non-leaf subtrees are examined only once and subtrees of
pruned subtrees need not be examined at all.
Page 37
References
Bisson, H. "Evaluation of Learning Systems: An Artificial Data-Based Approach," in Proceedings
of the European Working Session on Machine Learning, Y. Kodratoff (ed.), SpringerVerlag, Berlin, F.R.G., 1991.
Breiman, L., Friedman, J., Olshen, R., and Stone, C. Classification and Regression Trees,
Wadsworth Publishing, Belmont, CA, 1984.
Biemer, P. and Stokes, L. "Approaches to the Modeling of Measurement Error," in Measurement
Errors in Surveys, Chapter 24, Biemer, P., Groves, Lyberg, L., Mathiowetz, N., and
Sudman, S. (eds.), John Wiley & Sons, New York, 1991, pp. 487-516.
Christie, A. "Induction of Decision Trees from Noisy Examples," AI Expert, May 1993, 16-21.
Cestnik, B. and Bratko, I. "On Estimating Probabilities in Tree Pruning," in Proceedings of the
European Working Session on Machine Learning, Porto, Portugal, Springer-Verlag,
March 1991, pp. 138-150.
Creecy, R., Masand, B., Smith, S., and Waltz, D. "Trading MIPS and Memory of Knowledge
Engineering," Communications of the ACM 35, 8 (August 1992), 48-64.
Fuller, W. "Regression Estimation in the Presence of Measurement Error," in Measurement
Errors in Surveys, Chapter 30, Biemer, P., Groves, Lyberg, L., Mathiowetz, N., and
Sudman, S. (eds.), John Wiley & Sons, New York, 1991, pp. 617-636.
Groves, R. "Measurement Error Across Disciplines," in Measurement Errors in Surveys, Chapter
1, Biemer, P., Groves, Lyberg, L., Mathiowetz, N., and Sudman, S. (eds.), John Wiley &
Sons, New York, 1991, pp. 1-28.
Hill, D. "Interviewer, Respondent, and Regional Office Effects," in Measurement Errors in
Surveys, Chapter 23, Biemer, P., Groves, Lyberg, L., Mathiowetz, N., and Sudman, S.
(eds.), John Wiley & Sons, New York, 1991, pp. 463-486.
Holsapple, C. and Whinston, A. "Business Expert Systems, Irwin, Homewood, Illinois, 1987.
Irani, K., Cheng, J., Fayyad, U., and Qian, Z. "Applying Machine Learning to Semiconductor
Manufacturing," IEEE Expert 8, 1 (February 1993), 41-47.
Kononenko, I. and Bratko, I. "Information-Based Evaluation Criterion for Classifier's
Performance," Machine Learning, 6, 1991, 67-80.
Laird, P. Learning from Good and Bad Data, Kluwer Academic Publishers, Norwell, MA, 1988.
Liang, T. "A Composite Approach to Inducing Knowledge for Expert System Design,"
Management Science 38, 1 (1992), 1-17.
Mantaras, R.. "A Distance-Based Attribute Selection Measure for Decision Tree Induction,
Machine Learning 6, 1991, 81-92.
Mingers, J. "An Empirical Comparison of Pruning Methods for Decision Tree Induction, Machine
Learning 4, 2, 1989, 227-243.
Page 38
Moulet, M. "Using Accuracy in Scientific Discovery," in Proceedings of the European Working
Session on Machine Learning, Porto, Portugal, Springer-Verlag, March 1991, pp. 118136.
Murphy, P. and Aha, D. UCI Repository of Machine Learning Databases, University of
California, Irvine, Department of Information and Computer Science, 1991.
Mookerjee, V. and Dos Santos, B. "Inductive Expert System Design: Maximizing System Value,"
Information Systems Research (in press), 1993.
Niblett, T. and Bratko, I. "Learning Decision Rules in Noisy Domains," in Research and
Development in Expert Systems, (Proceedings of the Sixth Technical Conference of the
BCS Specialist Group on Expert Systems, Brighton, U.K., 1986.
Pearl, J. Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference,
Morgan Kauffman Publishers, San Mateo, CA, 1988.
Quinlan, J. "Induction of Decision Trees," Machine Learning, Vol. 1, 1986, 81-106.
Quinlan, J. "The Effect of Noise on Concept Learning," in Machine Learning, Vol. 2, Eds. R.
Michalski, J. Carbonnell, and T. Mitchel, Palo Alto, CA, Tioga Press, 1986, pp. 149-166.
Quinlan, J. "Simplifying Decision Trees," International Journal of Man Machine Studies, Vol.
27, 1987, 221-234.
Ross, S. Applied Probability Models, Holden-Day, San Francisco, CA, 1970.
Rao, J. and Thomas, R. "Chi-Squared Tests with Complex Survey Data Subject to
Misclassification Error," in Measurement Errors in Surveys, Chapter 31, Biemer, P.,
Groves, Lyberg, L., Mathiowetz, N., and Sudman, S. (eds.), John Wiley & Sons, New
York, 1991, pp. 637-664.
Shannon, C. and Weaver, W.. The Mathematical Theory of Communication, University of Illinois
Press 1949, (published in 1964).
Snedecor, C. and Cochran, W. Statistical Methods, 7th edition, Iowa State University Press,
1980.
Tam, K. and Kiang, M. "Predicting Bank Failures: A Neural Network Approach," Applied
Artificial Intelligence, 4, 1990, 265-282.
Page 39
Generated Data Set 1 - Average Performance (RIS)
Avg RIS
90
80
70
60
50
ID3p
ID3ecp
40
30
20
10
0
0
0.2
0.4
0.6
0.8
1
C
CV RIS
Generated Data Set 1 - Variance of Performance (RIS)
0.5
0.45
0.4
0.35
0.3
0.25
0.2
0.15
0.1
0.05
0
ID3p
ID3ecp
0
0.2
0.4
0.6
0.8
C
Figure 4: RIS Performance Graphs for Generated Data Set 1
1
Page 40
Generated Data Set 1 - Average Performance (Accuracy)
0.7
0.6
0.5
ID3p
ID3ecp
0.4
0.3
0.2
0.1
0
0
0.2
0.4
0.6
0.8
1
C
Generated Data Set 1 - Variance of Performance (Accuracy)
CV ACC
Avg ACC
0.9
0.8
0.5
0.45
0.4
0.35
0.3
0.25
0.2
0.15
0.1
0.05
0
ID3p
ID3ecp
0
0.2
0.4
0.6
0.8
C
Figure 5: Accuracy Performance Graphs for Generated Data Set 1
1
Page 41
50
45
40
35
30
25
20
15
10
5
0
ID3p
ID3ecp
0
0.2
0.4
0.6
0.8
1
C
Lymphography Data Set - Variance of Performance (RIS)
2.5
2
CV RIS
Avg RIS
Lymphography Data Set - Average Performance (RIS)
1.5
ID3p
ID3ecp
1
0.5
0
0
0.2
0.4
0.6
0.8
C
Figure 6: RIS Performance Graphs for the Lymphography Data Set
1
Page 42
Lymphography Data Set - Average Performance (Accuracy)
0.75
0.7
0.6
ID3p
ID3ecp
0.55
0.5
0.45
0.4
0
0.2
0.4
0.6
0.8
1
C
Lymphography Data Set - Variance of Performance (Accuracy)
CV ACC
Avg ACC
0.65
0.1
0.09
0.08
0.07
0.06
0.05
0.04
0.03
0.02
0.01
0
ID3p
ID3ecp
0
0.2
0.4
0.6
0.8
1
C
Figure 7: Performance Graphs (Accuracy) for the Lymphography Data Set
Page 43
ID3ecp Sensitivity in Generated Data Set 1 - Constant Low ( -0.1)
0.4
0.35
0.25
ID3p
ID3ecp
0.2
0.15
0.1
0.05
0
0
0.2
0.4
0.6
0.8
1
C
ID3ecp Sensitivity in Generated Data Set 1 - Constant High (+0.1)
0.4
0.35
0.3
CV RIS
CV RIS
0.3
0.25
ID3p
ID3ecp
0.2
0.15
0.1
0.05
0
0
0.2
0.4
0.6
0.8
C
Figure 8: Constant Sensitivity Graphs for Generated Data Set 1
1
Page 44
ID3ecp Sensitivity in Generated Data Set 1 - Varying [-0.1, +0.1]
0.45
0.4
0.35
0.25
ID3p
ID3ecp
0.2
0.15
0.1
0.05
0
0
0.2
0.4
0.6
0.8
1
C
ID3ecp Sensitivity in Generated Data Set 1 - Varying [-0.05, +0.05]
0.45
0.4
0.35
0.3
CV RIS
CV RIS
0.3
0.25
ID3p
ID3ecp
0.2
0.15
0.1
0.05
0
0
0.2
0.4
0.6
0.8
C
Figure 9: Varying Sensitivity Graphs for Generated Data Set 1
1