A FRAMEWORK FOR GENERATING DATA TO SIMULATE CHANGING
ENVIRONMENTS
Anand Narasimhamurthy
School of Electronics and Computer Science,
University of Wales, Bangor,
LL57 1UT, United Kingdom
[email protected]
Ludmila I. Kuncheva
School of Electronics and Computer Science,
University of Wales, Bangor,
LL57 1UT, United Kingdom
[email protected]
ABSTRACT
A fundamental assumption often made in supervised classification is that the problem is static, i.e. the description
of the classes does not change with time. However many
practical classification tasks involve changing environments. Thus designing and testing classifiers for changing
environments are of increasing interest and importance. A
number of benchmark data sets are available for static classification tasks. For example, the UCI machine learning
repository is extensively used by researchers to compare
algorithms across various domains. No such benchmark
datasets are available for changing environments. Also,
while generating data for static environments is relatively
straightforward, this is not so for changing environments.
The reason is that an infinite amount of changes can be
simulated, and it is difficult to define which ones will
be realistic and hence useful. In this paper we propose
a general framework for generating data to simulate
changing environments. The paper gives illustrations of
how the framework encompasses various types of changes
observed in real data and also how the two most popular
simulation models (STAGGER and moving hyperplane)
are represented within.
“spam” and “non-spam” evolve with time. They are userspecific, and users’ preferences change with time. Also
spammers are active opponents who devise newer methods to evade detection thereby altering the definition of
class “spam” within the current feature space. A change
in the environment is referred to as concept drift in machine learning. The changes could include minor fluctuations of the underlying probability distributions, steady
trends, rapid substitution of one classification task with another and so on [12, 33]. These categories are by no means
mutually exclusive and various combinations are possible.
Bespoke online classification models [5, 6, 32], as well as
classifier ensembles [21,29] have been proposed for changing environments [22].
In the light of the growing importance of classification in changing environments, it is striking how little attention has been paid to putting together a collection of
benchmark data sets or formulating generic procedures for
simulating data. As a result researchers use their own
simulated datasets or specific real data domains such as
finance [8, 12, 32], text categorisation [17–19], spam filtering [4] and web searches [11]. In spite of some critiques [27], the creation of the UCI Machine Learning
Repository [3] in 1987 has had a significant positive impact on Machine Learning and Pattern Recognition Communities [1]. A KDD (Knowledge Discovery in Databases)
archive was subsequently developed in 1999 [1] containing large data sets of diverse types and domains (http:
//kdd.ics.uci.edu). Data with a variety of known
changes are needed for evaluating classification methods
in changing environments. In this study we propose a general framework for generating artificial data which simulate
changing environments. The data may contain one or more
types of drift as desired by the user.
KEY WORDS
Machine Learning, Changing environments, Concept drift,
Population drift, Artificial data, Simulated data.
1 Introduction
Online learning is increasingly occupying centre stage in
data mining as more and more applications generate massive streams of data [5, 6, 29]. In these cases it is often
impractical to apply conventional classification methodologies since these typically are batch algorithms often requiring more than one pass through the entire data. Many online versions of conventional classification methods have
been developed [23–25].
A fundamental assumption in supervised classification, including many of the online classification methods, is
that the problem is static, i.e. the description of the classes
does not change with time. In many practical problems this
is most likely to be false. An example is detecting and filtering out spam-email. The descriptions of the two classes
549-167
The rest of the paper is organised as follows. In Section 2 we review the terminology used in the literature on
changing environments. The proposed framework is described in Section 3. Illustrations of how the proposed
framework accommodates the defined types of environmental changes are given in Section 4. An implementation
algorithm is given in Section 5. Section 6 concludes the
study.
384
2 Terminology
because authors make distinctions.
The most used term in the ML literature on the subject is
‘concept drift’. However, there is a variety of other terms,
some of them not precisely defined, taken to mean the same
thing. This lack of uniformity of terminology prompted
us to try to summarise and explain the terms found in the
literature before proceeding with the general framework.
The term ’concept’ is used to mean slightly different
things. It is sometimes used to refer to the class of interest.
A concept change in this case would mean a deviation of
the description of the class from the original description.
Consider for example the STAGGER dataset described by
Widmer and Kubat [33] (this is discussed further in section
4.2). In this dataset the feature space of a simple blocks
world is described by three features (attributes) namely,
size, color and shape. Examples are labelled according to a
target concept (this is not given to the classifier explicitly)
which changes over time. For example the target concept
for the first set of instances is ”size = small AND color =
red”, the target concept for a subsequent set of instances is
”color = green OR shape = circular”. An analogous ’target
concept’ or a ’concept of interest’ in a real world domain
such as information filtering, may be the class ‘relevant articles’ from a set of documents (e.g. Klinkenberg [14, 18]).
In other articles, the authors use ‘concept’ to refer to
the whole distribution of the problem at time moment , including all classes. Then concept drift is the substitution of
. To avoid
one concept at time with another at time
confusion, in this study we will try to avoid the term altogether. Another frequently used term is population drift,
this refers to the changes in the underlying distributions.
Kelly et al. [12] use the term population drift to mean a
changes either in the priors or in the probabilities probabilities of class membership, conditional on feature vectors.
They point out that the term concept drift has been used
in the machine learning literature both for this (population
drift) as well as other changes.
While in a natural system we can expect gradual drifts
(e.g., seasonal, demographic, habitual, etc.), sometimes the
class description may change rapidly due to so called hidden contexts. Such contexts may be, for example, illumination in image recognition and accents in speech recognition [34]. The context might instantly become highly relevant. Consider a system trained on images with similar
illumination. If images with the same type of content but
a different illumination are fed to the system, the class descriptions might change so as to make the system worse
than a random guess.
The type of changes can be roughly summarised as
follows. They are briefly discussed in subsequent sections.
These categories are by no means mutually exclusive and
various combinations are possible. For example, recurring
trends and population drift are both subsumed by the other
categories, namely gradual changes and concept substitution. These are listed separately however, since they are
changes which occur in real world applications and also
Gradual changes (gradual drift, evolutionary changes,
concept drift) [2, 10, 13, 16, 30]
Substitutions (abrupt changes, revolutionary changes,
concept substitution, concept shift) [10, 16, 33]
Recurring trends (recurring contexts) [31, 33]
Population drift [12]
3 A general framework for simulating
changing environments
Every classification problem, however complex it might
be, may be described completely by the following. Let
be an object in the -dimensional feature space
of the problem and be the set of class
labels. The knowledge of the prior probabilities for the
classes, and the class-conditional probability den
sity functions (pdf) , , determine completely the problem and the optimal (minimum error) classifier for it. Viewed in this probabilistic sense, a classification problem may change due to the changes in the and/or . Posterior probabilities and the unconditional pdf may also change but these changes
can be re-expressed in terms of and .
Without loss of generality assume that all data live in
the -dimensional real space, . Consider a set of data sources with known distributions. The distribution for
source is characterised by the class-conditional probability density functions , and the prior probabilities
, , for this source, . At
any time we have one or more “active” data sources. Let
specify the extent of the influence of data
source at time . We shall treat the influences as mixing
proportions
and the resultant distribution as a mixture, i.e.
! for any . At time the data distribution " #$ %
is characterised by prior probabilities
&
'
( and class-conditional pdfs
&
'
( As the distributions at the sources are fixed, the data distribution at moment , " #$ % is specified through . Thus
we can equivalently define " #$ % as
&
" #$ % To be able to cater for all possible scenarios of changes we
will allow the set of sources to be as large as necessary. For
example, if 1000 time instances are needed in a population
drift data, the set of sources might need to be of cardinality
1000, one source for each time instance.
385
transition. At time instance , " #$ % will be a set of binary
mixing proportions, with for the data source cor
. To make
responding to , and all other the model plausible, the consecutive data sources should be
can
closely related. For example, data source for time
be obtained from data source for time by tweaking a parameter. In this example, the starting and the final distributions are the same as in the first example. However, at each
time , only one source is sampled, so the two classes in the
example remain separable at any transition time moment .
Figure 3 shows the scatterplot of the data. Note that this
time there are four data sources with their corresponding
.
This second example of gradual change illustrates a
popular artificial data type used by many authors to test
their classification methods for changing environments.
This data simulation idea is often called the moving hyperplane [7, 11, 21].
The proposed framework is meant to be a versatile
tool to combine the scenarios of the two examples by letting
many data sources to be sampled at a time with mixing
proportions evolving with time. It is hoped that in this way
the generated data will represent a more realistic model of
real-life changing environments.
1
1
0.5
2
time
0
Figure 1. Mixing proportions and gradual change between two data sources
for a linear
4 Illustration of the types of changes within
the proposed framework
4.1 Gradual drift
Two examples of gradual drift are given below. The first
example represents only two sources with gradual changes
from one source to the other, as proposed by Widmer and
Kubat [33] and Black and Hickey [2]. This is a scenario
where one concept fades gradually while the other takes
over. The real-world example given by Widmer and Kubat
is that of a device which begins to malfunction. At first
only small number of data points will come from the stable
failure state. Finally the failure will take over completely.
Figure 1 shows and as functions of for this
case.
Figure 2 (a) represents the starting data where the distribution is equivalent to that of source 1, i.e., ,
if
otherwise
if
otherwise
4.2 Substitutions
This is the type of change most often simulated in order to
assess the performance of a classifier in changing environment. The classical example is the STAGGER concept first
proposed by Schlimmer and Granger [28]. It has received
the status of a benchmark artificial data by being used by
many authors since then [4, 7, 20–22, 33].
The feature space is described by three features (attributes): size small (1), medium (2), large (3) , color
red (a), green (b), blue (c) and shape
square (A),
circular (B), triangular (C) . There are three data sources
(called ‘target concepts’)
The end distribution is that of source 2, shown in Figure 2 (d). Subplots (b) and (c) plot the transition of the
data distribution for equal to (0.67,0.33) and
(0.33,0.67), respectively.
The second example of gradual change is when there
are more sources so the mixture components at any time
of the transition are not just the start and the end ones.
This example mimics population drift whereby the parameters of the distributions change gradually with time.
An example from real-life would be the change of category “high risk” in loan approval. The criteria for a
money lender to approve a loan evolve with time to respond to new socio-economic circumstances; the gross annual salaries increase; the market is getting saturated with
loan providers. All this will alter the class distributions,
so class “high risk” from a few years ago will have a new
profile now. To model gradual changes in the parameters
of the distributions, we assume that there are as many data
sources as there are time instances to be recorded in the
Target Concept 1 : size = small AND color = red
Target Concept 2 : color = green OR shape = circular
Target Concept 3 : size = medium OR size = large
In their experiment, Widmer and Kubat [33] generate
randomly (from a uniform distribution across the feature
space) 120 training instances and label each instance according to the current target concept. After processing each
instance, the predictive accuracy is tested on an independent test set of 100 instances, also generated randomly. One
target concept is active at a time. Concept 1 is active from
instance 1 to instance 40, Concept 2 is active from instance
41 to 80 and Concept 3 from instance 81 to 120.
This example is modelled as follows. Consider three
data sources each related to one of the target concepts. The
distributions at the three data sources are
386
1.5
1.5
1.5
1
1
1
1.5
1
0.5
0.5
0.5
0.5
0
0
0
0
−0.5
−0.5
−0.5
−0.5
−1
−1
−1
−1
−1.5
−1.5
−1.5
−1.5
−1
0
1
−1
(a)
0
1
−1
(b)
0
1
−1
(c)
0
1
(d)
Figure 2. Scatterplots of the data illustrating a gradual (linear) change from distribution of source 1 (a) to that of source 2 (e).
Class is depicted with grey dots, and class with black dots.
1.5
1.5
1.5
1.5
1
1
1
1
0.5
0.5
0.5
0.5
0
0
0
0
−0.5
−0.5
−0.5
−0.5
−1
−1
−1
−1
−1.5
−1
0
−1.5
1
−1
0
1
−1.5
(a)
−1
0
1
−1.5
(b)
−1
0
1
(c)
(d)
Figure 3. Scatterplots of the data illustrating a gradual (linear) change using 4 data sources. Class is depicted with grey dots,
and class with black dots.
, ,
for instances 1aA, 1aB and 1aC and
0 for all other instances. for instances
1aA, 1aB and 1aC and 1/24 for all other instances.
4.3 Recurring trends
Data source 1.
, ,
for instances xby and xzB, where x
is any of 1,2,3 y is any of A,B,C and z is any of a,b,c.
for all other instances. for all other
instances. for xby and xzB and 1/12 for
all remaining instances.
Data source 2.
In many applications contexts tend to recur either cyclically
or in an unordered fashion. Biological and economic systems tend to go through cycles of development with recurring patterns [33]. Recurring contexts may be due to irregular phenomena such as inflation rates or market mood [9]
or due to cyclic phenomena such as seasonal changes. For
example, Tsymbal [31] deal with antibiotic resistance data
for nosocomial infections. They discovered seasonal context recurring with winter and spring models, these corresponded to yearly spring and winter infection outbreaks.
With respect to our framework, the phenomenon of a
periodically recurring context can be expressed as
, ,
for instances for 2zy and 3zy and 0
for all remaining instances. for 2zy and
3zy and 1/9 for all remaining instances.
Data source 3.
" #$ % " #$ %
For the STAGGER data,
for
for for
where is the period of repetition.
and 0 for any other
and 0 for any other
and 0 for any other
5 Implementation algorithm
Figure 4 gives an algorithm which implements the proposed framework. 1
In the substitution scenario, usually one data source is
instantly replaced by another, so " #$% contain again binary
mixing proportions as in the second example above. However, this time the subsequent data sources are assumed to
be significantly different from one another.
1 MATLAB code is available at the following url.
http:
//www.informatics.bangor.ac.uk/˜kuncheva/
activities/epsrc2006/changing_environments.html
387
ods for problems with changing environments, it will not
be long before a valuable benchmark of such data will become publicly available, following in the footsteps of UCI
ML repository [3] and UCI KDD Database [1].
Input:
number of time instances required
( = number of data points)
number of data sources
a user-defined function which
samples once from data source an
array with the mixing
proportions (column contains
)
Acknowledgments
This work was supported by EPSRC grant EP/D04040X/1.
References
Data generation:
Initialise the data set For (1)
= Sample once from with
distribution
defined by
(2)
Sample a new data point
from source , ,
[1] S. D. Bay, D. Kibler, M. J. Pazzani, and P. Smyth. The
UCI KDD archive of large data sets for data mining
research and experimentation. SIGKDD Exploration,
2(2):14–18, 2000.
and add it to the current set (the class label is also added)
End Return the data set
[2] M.M Black and R.J. Hickey. Maintaining the performance of a learned classifier under concept drift.
Intelligent Data Analysis, 3(6):453–474, 1999.
[3] C. L. Blake and C. J. Merz.
UCI repository of machine learning databases, 1998.
http://www.ics.uci.edu/˜mlearn/
MLRepository.html.
Figure 4. Algorithm: Generating data to simulate changing
environments
[4] S. J. Delany, P. Cunningham, and A. Tsymbal. A
comparison of ensemble and case-based maintenance
techniques for handling concept drift in spam filtering. Technical Report TCD-CS-2005-19, Trinity College Dublin, 2005.
6 Conclusion
[5] P. Domingos and G. Hulten. Mining high-speed data
streams. In Knowledge Discovery and Data Mining,
pages 71–80, 2000.
We propose a general framework to generate data simulating changing environments. The framework is meant to
channel the effort in developing a test-bed of artificial data
to train and test classifiers under various modes of changing environments. The framework accommodates the currently favourite STAGGER and Moving Hyperplane generation strategies and offers a versatile approach towards
devising new strategies. We show that the framework covers the three main types of changes as formulated in the
literature: gradual changes, substitution and systematic (recurrent) trends. These all can be simulated individually or
in combination. An implementation algorithm is detailed
as well.
Our future work includes simulating changes on real
data. Current state-of-the-art is relatively poor in this respect, revolving around dropping classes and adding data
from new classes [15, 26] or tweaking the class labels [2].
One possible way to go is to use one of the features as hidden context and sort the data set by that feature to simulate
a trend.
For now we have looked into generating artificial data
but the main focus must be maintained on the possibility of
finding real data sets which exhibit a clear case of changing environments. Many such datasets, e.g. in finance,
are not freely available as benchmark due to privacy limitations. Given the pressing need for new tools and meth-
[6] P. Domingos and G. Hulten. A general framework for
mining massive data streams. Journal of Computational and Graphical Statistics, 12:945–949, 2003.
[7] F. Ferrer-Troyano, J. S. Aguilar-Ruiz, and J. C.
Riquelme. Incremental rule learning based on example nearness from numerical data streams. In 2005
ACM Symposium on Applied Computing, SAC 05,
Santa Fe, New Mexico,USA, 2005. ACM Press.
[8] M. Harries and K. Horn. Detecting concept drift in
financial time series prediction using symbolic machine learning. In Eighth Australian Joint Conference on Artificial Intelligence, pages 91–98, Singapore, 1995. World Scientific Publishing.
[9] M. Harries, C. Sammut, and K. Horn. Extracting hidden context. Machine Learning, 32:101–126, 1998.
[10] R.J. Hickey and M. M. Black. Refined time stamps
for concept drift detection during mining for classification rules. In Proceedings of the International
Workshop on Temporal, Spatial and Spatio-Temporal
Data Mining (TSDM2000), Lecture Notes in Artificial
388
Intelligence, volume 2007, pages 20–30. SpringerVerlag: Berlin, 2000.
on Data Mining, pages 123–130, Los Alamitos, CA,
2003. IEEE Press.
[11] G. Hulten, L. Spencer, and P. Domingos. Mining time-changing data streams. In In Proc. 7th
ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 97–106.
ACM Press, 2001.
[22] L. I. Kuncheva. Classifier ensembles for changing environments. In 5th International Workshop on Multiple Classifier Systems, MCS 04, LNCS, volume 3077,
pages 1–15. Springer-Verlag, 2004.
[23] H. K. H. Lee and M. A. Clyde. Lossless online
Bayesian bagging. Journal of Machine Learning Research, 5:143–151, 2004.
[12] Mark G. Kelly, David J. Hand, and Niall M. Adams.
The impact of changing populations on classifier performance. In KDD ’99: Proceedings of the fifth ACM
SIGKDD international conference on Knowledge discovery and data mining, pages 367–371, New York,
NY, USA, 1999. ACM Press.
[24] N. C. Oza. Online Ensemble Learning. PhD thesis,
University of California, Berkeley, 2001.
[25] N. C. Oza and S. Russell. Online bagging and boosting. In Eighth International Workshop on Artificial
Intelligence and Statistics, Key West, Florida, 2001.
[13] M. Klenner and U. Hahn. Concept versioning : A
methodology for tracking evolutionary concept drift
in dynamic concept systems. In 11th European
Conference on Artificial Intelligence, pages 473–477,
Chichester, England, 1994. Wiley.
[26] R. Polikar, L. Udpa, S. S. Udpa, and V. Honavar.
Learn++: An incremental learning algorithm for supervised neural networks. IEEE Transactions on Systems, Man and Cybernetics, 31(4):497–508, 2001.
[14] R. Klinkenberg. Learning drifting concepts with partial user feedback. In P. Perner and V. Fink, editors, Beitrge zum Treffen der GI-Fachgruppe 1.1.3
Maschinelles Lernen (FGML-99), Magdeburg, Germany, 1999.
[27] S. Salzberg. On comparing classifiers: A critique
of current research and methods. Data Mining and
Knowledge Discovery, 1:1–12, 1999.
[28] J. C. Schlimmer and R. H. Granger. Incremental learning from noisy data. Machine Learning,
1(3):317–354, 1986.
[15] R. Klinkenberg. Using labeled and unlabeled data
to learn drifting concepts. In Workshop notes of the
IJCAI-01 Workshop on Learning from Temporal and
Spatial Data, pages 16–24, Menlo Park, CA, USA,
2001. IJCAI, AAAI Press.
[29] W. N. Street and Y. S. Kim. A streaming ensemble
algorithm (SEA) for large-scale classification. In Proceedings of the 7th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining,
pages 377–382. ACM Press, 2001.
[16] R. Klinkenberg. Learning drifting concepts: Example selection vs. example weighting. Intelligent Data
Analysis, 8(3), 2004.
[30] A. Tsymbal. The problem of concept drift: definitions
and related work. Technical Report TCD-CS-200415, Trinity College Dublin, April 2004.
[17] R. Klinkenberg and T. Joachims. Detecting concept
drift with support vector machines. In Proceedings
of the Seventeenth International Conference on Machine Learning (ICML), pages 487–494, San Francisco, CA, USA, 2000. Morgan Kaufmann.
[31] A. Tsymbal, M. Pechenizkiy, P. Cunningham, and
S. Puuronen. Handling local concept drift with dynamic integration of classifiers: domain of antibiotic
resistance in nosocomial infections. In Proc 19th
IEEE Int. Symposium on Computer-Based Medical
Systems CBMS 2006, Salt Lake City, UT, June 2006.
IEEE CS Press.
[18] R. Klinkenberg and I. Renz. Adaptive information filtering: Learning in the presence of concept drifts. In
AAAI-98/ICML-98 workshop Learning for Text Categorization, Menlo Park,CA, 1998.
[19] R. Klinkenberg and S. Rping. Concept drift and the
importance of examples. In Text Mining – Theoretical Aspects and Applications, pages 55–77. PhysicaVerlag, Berlin, Germany, 2003.
[32] H. Wang, W. Fan, P. S. Yu, and J. Han. Mining
concept drifting data streams using ensemble classifiers. In Proceedings of the 9th ACM SIGKDD International Conference on Knowledge Discovery and
Data Mining, pages 226–235. ACM Press, 2003.
[20] J. Kolter. Using additive expert ensembles to cope
with concept drift - A senior thesis, April 2005. A
senior thesis.
[33] G. Widmer and M. Kubat. Learning in the presence of
concept drift and hidden contexts. Machine Learning,
23:69–101, 1996.
[34] G. Widmer and M. Kubat. Special Issue on Context
Sensitivity and Concept Drift. Machine Learning, 32,
1998.
[21] J. Z. Kolter and M. A. Maloof. Dynamic weighted
majority: A new ensemble method for tracking concept drift. In Proc 3rd International IEEE Conference
389