Document de travail du LEM
2011-06
A PROBABILITY-MAPPING ALGORITHM FOR
CALIBRATING THE POSTERIOR PROBABILITIES: A
DIRECT MARKETING APPLICATION
Kristof Coussement*, Wouter Buckinx**
*
IESEG School of Management (LEM-CNRS)
**
Python Predictions, Brussels, Belgium
A Probability-Mapping Algorithm for Calibrating the Posterior
Probabilities: A Direct Marketing Application*
Kristof Coussement£, Wouter Buckinx+
£
IESEG School of Management (LEM-CNRS), Department of Marketing, 3 Rue de la Digue,
F-59000, Lille (France).
+
Managing partner (PhD), Python Predictions,
Avenue R. Van den Driessche 9, B-1150 Brussels, Belgium.
First and corresponding author: Kristof Coussement, [email protected] ,
Tel.:+33320545892
Second author: Wouter Buckinx, [email protected]
This paper is accepted for publication in European Journal of Operational Research
1
A Probability-Mapping Algorithm for Calibrating the Posterior Probabilities:
A Direct Marketing Application
Abstract
Calibration refers to the adjustment of the posterior probabilities output by a classification
algorithm towards the true prior probability distribution of the target classes. This adjustment
is necessary to account for the difference in prior distributions between the training set and the
test set. This article proposes a new calibration method, called the probability-mapping
approach. Two types of mapping are proposed: linear and non-linear probability mapping.
These new calibration techniques are applied to 9 real-life direct marketing datasets. The
newly-proposed techniques are compared with the original, non-calibrated posterior
probabilities and the adjusted posterior probabilities obtained using the rescaling algorithm of
Saerens, Latinne, & Decaestecker (2002). The results recommend that marketing researchers
must calibrate the posterior probabilities obtained from the classifier. Moreover, it is shown
that using a ‘simple’ rescaling algorithm is not a first and workable solution, because the
results suggest applying the newly-proposed non-linear probability-mapping approach for best
calibration performance.
Keywords: data mining, direct marketing, response modeling, calibration, decision support
systems
This paper is accepted for publication in European Journal of Operational Research
2
1. Introduction
Due to recent developments in IT infrastructure and the ever-increasing trust placed in
complex computer systems, analysts are showing an increasing interest in classification
modeling in a variety of disciplines such as credit scoring (Martens et al., 2010; Paleologo et
al., 2010), medicine (Conforti & Guido, 2010), text classification (Bosio & Righini, 2007),
SMEs fund management (Kim and Sohn, 2010), revenue management (Morales & Wang,
2010), and so on. The same interests are shared by the direct marketing community. Direct
marketing analysts have an increasing interest in building prediction models that assign a
probability of response to each and every individual customer in the database (Lamb et al.,
1994). The task of classification is made even more interesting by the fact that nowadays
current marketing environments store incredible amounts of customer information at a very
low cost, including socio-demographics, transactional buying behavior, attitudinal data, etc.
(Naik et al., 2000), while at the same time there has been a tremendous increase in academic
interest in direct marketing applications (e.g. Allenby et al., 1999; Baumgartner & Hruschka,
2005; Hruschka, 2010; Lee et al., 2010; Piersma & Jonker, 2004). Therefore response models
are defined as classification models that attempt to discriminate between responders and nonresponders on a certain company mailing.
In the past, purely statistical methods like logistic regression, discriminant analysis and naive
bayes models have been proposed to discriminate between responders and non-responders in
a direct marketing context (Baesens et al., 2002; Bult, 1993; Deichmann et al., 2002).
Although these techniques may be very effective, they make a stringent assumption about the
underlying relationship between the independent variables and the dependent or response
variable. In response to this, more advanced data mining algorithms like decision treegenerating techniques, artificial neural networks and support vector machines have been
applied (Baesens et al., 2002; Bose & Chen, 2009; Crone and et al., 2006; Haughton &
Oulabi, 1997; Zahavi & Levin, 1997). All these binary classification models are used for two
reasons. First, researchers rely on them to obtain robust parameter estimates of the
independent variables by modeling the probability of response as a function of the
independent variables. Second, these models are used to obtain consistent predicted
probabilities of response, which are then used (i) to rank the customers based on their
This paper is accepted for publication in European Journal of Operational Research
3
responsiveness to the campaign, (ii) to optimize the overall campaign strategy by offering the
customer the product with the highest response probability over the different response models
and (iii) for the discrimination task of the response event itself where one classifies customers
into responders and non-responders. For (ii) and (iii), the absolute size of the posterior
response probabilities is crucial. This study focuses on the process of obtaining correct
response probabilities, where calibrating the posterior probabilities could have a positive
impact on the optimization of the overall campaign strategy and the efficiency of the
discrimination task.
In practice, a classification model is built on a training set, i.e. a set of customers where both
the independent variables and the dependent variable are present. In order to correctly
measure the discrimination power of the trained classifier, the classification model is applied
to a group of customers who have not been used for training, called the scoring or test set. The
purpose is to obtain robust and consistent predictions for the response probability of these
unseen customers. As one is interested to divide the customers into responders and nonresponders, a judicious classification based on the posterior response probabilities of the
customers is needed. In other words, customers having a response probability exceeding a
certain threshold will be classified as responders and vice versa.
However, it often happens that a classifier is trained using a dataset that does not reflect the
true prior probabilities of the target classes in the real-life population. This may have serious
negative consequences on the discrimination performance because the posterior probabilities
do not reflect the true probability of interest. This phenomenon occurs in a direct marketing
context as well where the prior probabilities between the training set and the (out-of-sample)
test set are significantly different. More specifically, the training set consists of customers
who are preselected by an earlier response model as being customers with a high response
probability, while the test set does not make any restrictions based on the customer profiles in
the database. In such a case, a large discrepancy exists between the response distributions on
the training set and the test set. The incidence, which is the percentage of responders in a data
set, is much higher in the training set as compared to the incidence of real response in the outof-sample test set. This inconsistency has a negative effect on the discrimination performance
on the test set, especially because the classifier’s decision to classify customers into
This paper is accepted for publication in European Journal of Operational Research
4
responders or non-responders is based on setting a threshold on the raw posterior probabilities
of class membership. For instance, when a classifier is trained on a dataset with a higher
incidence than the one in the test set, the posterior probabilities on the test set are inflated.
Thus making a classification decision based on the absolute value of the posterior
probabilities may significantly harm the discrimination performance. Moreover, optimizing
the campaign strategy by offering the product with the highest response probability to the
customer becomes useless because the response probabilities for different products for a
particular customer are not comparable. This paper focuses on how researchers can adjust the
posterior probabilities based on the true prior distribution of the response variable. This
process of adjustment is called calibration.
This paper proposes a new methodology to be used to calibrate the posterior probabilities
from the test set with the real-world situation, a process called probability-mapping. It maps
the posterior response probabilities obtained from the classifier onto the prior distribution of
real response. The new probability-mapping approaches using generalized linear models and
non-parametric generalized additive models are compared with the original, non-calibrated
posterior probabilities and the calibrated probabilities using the rescaling methodology of
Saerens et al. (2002).
This paper is structured as follows. Section 2 describes the methodological framework, while
Section 3 explores the different calibration approaches (rescaling approaches and probabilitymapping approaches). Section 4 explains the characteristics of empirical validation, while
Section 5 explores the results. Section 6 gives managerial recommendations, and finally
Section 7 concludes this paper.
2. Methodological framework
Figure 1 shows the methodological framework for the different calibration methods applied in
this study.
This paper is accepted for publication in European Journal of Operational Research
5
[INSERT FIGURE 1 OVER HERE]
Define a training set TRAIN M
(x i , y i )
m
i 1
consisting of m customers. Each customer
( x i , y i ) is a combination of an input vector x i representing the independent variables and a
dependent variable y i with y i
0,1 corresponding to whether or not a customer responded
on a certain mailing. TRAIN M consists of all customers who were selected by a previous
response model, thus received a direct mailing to buy the product, and therefore indicated as
customers having a high response probability. During the training phase, a classifier C maps
the input vector space onto the binary response variable using the training set observations.
For the test set TEST N
(x i )
n
i 1
consisting of n customers, the trained classifier C is applied
and for every customer in TEST N a response probability Porg is obtained. The purpose of this
paper is to adjust the posterior probabilities Porg to the real response distribution because the
trained sample TRAIN M is not representative for TEST N which corresponds to the true
population. Therefore for every observation (x i ) in TEST N , the real response is collected
and summarized in REAL N
(yi )
n
i 1
with y i
0,1 corresponding to whether or not the
customer spontaneously bought that particular product in a time window without direct
mailing actions. The real response represents a response of pure interest in the product. In
other words, REAL N is used to represent the true prior probabilities.
The purpose of the calibration phase is to adjust Porg, the non-calibrated posterior probabilities
of TEST N , in order to truly represent the probability of response. With the aim of
methodologically benchmarking the different calibration methods, a k-fold cross-validation is
applied. In a k-fold cross-validation, the dataset is randomly split into k equal parts of which
one after the other is used during the scoring phase; while the other k-1 parts are used for
training the calibration model. Note that TEST kN ( REAL kN ) represents the k-fold for TEST N
( REAL N ), while Pkorg represents the non-calibrated posterior probabilities of TEST kN .
3. Calibration approaches
This paper is accepted for publication in European Journal of Operational Research
6
Two types of calibration methods are applied: (i) the rescaling algorithm of Saerens et al.
(2002) and (ii) the newly-proposed probability-mapping approaches. The former algorithm
rescales Pkorg the posterior probabilities of TEST kN taking into account the real incidence of
REAL kN (Saerens et al., 2002), while the latter type adjusts the posterior probabilities of
TEST kN by mapping them onto the real responses of REAL kN .
3.1 Rescaling algorithm (SAERENS)
This section explains the methodology of Saerens et al. (2002). The starting point of the
Saerens et al. (2002) calibration approach is based on Bayes’ rule, i.e. the posterior
probabilities of response depend in a non-linear way on the prior probability distribution of
the target classes. The prior probability distribution of the target class is defined as the
incidence of the target class, or in this setting the percentage of responders in the dataset.
Therefore, a change in the prior probability distribution of the target classes changes the
posterior response probabilities of the classification model. Saerens et al. (2002) describe a
process that adjusts the posterior probabilities of response output by the classifier to the new
prior probability distribution of the target classes making use of a predefined rescaling
formula. In detail, the calibrated posterior probabilities of response for the customers in the
test set of fold k are obtained by weighting the non-calibrated posterior probabilities, Pkorg, by
the ratio of the response incidence of REAL kN , i.e. the new prior probability distribution, to
the response incidence in the training set, i.e. the old prior probability distribution. The
denominator is a scaling factor to make sure that the calibrated posterior probabilities sum up
to one.
In summary,
Pknew
Pk (c1 )
Pkorg
Pkt (c1 )
Pk (c0 )
Pk (c1 )
(1 Pkorg )
Pkorg
Pkt (c0 )
Pkt (c1 )
(1)
with Pknew representing the calibrated posterior response probabilities in fold k, Pk(ci) and
Pkt(ci) the new and old prior probabilities for class i with i
0,1 . A data set NEWkN is
This paper is accepted for publication in European Journal of Operational Research
7
obtained which contains Pknew, the calibrated posterior probabilities for the test data of
TEST kN .
3.2 Probability-mapping approaches
The purpose of the probability-mapping approaches is to map Pkorg, the old posterior
probabilities of TEST kN , onto the real response probabilities of REAL kN . As such, one is able
to build a classification model that maps the non-calibrated probabilities onto the real
response probabilities. This model is then used to calibrate the old probabilities with the
corrected probabilities of response. However, the real probability distribution of the target
classes is not directly available from REAL kN which only contains the real responses yi with
yi
0,1 on an individual customer level. In order to convert the real responses yi with
yi
0,1 on an individual level in REAL kN into a real response probability distribution, a
number of bins b are constructed. The incidence of response is calculated per bin and equals
the percentage of real response. This incidence is used as an approximation for the real
probability of response per bin. In practice, both TEST kN and REAL kN are split into a number
of bins b using the equal frequency binning approach based on the posterior probabilities of
TEST kN . TEST kb ( REAL kb ) represents the b-th bin in the k-fold of TEST kN ( REAL kN
respectively). TEST kb and REAL kb logically contain identical observations, while Pkborg is the
non-calibrated posterior probability average for the b-th bin in TEST kN and Pkbreal is the
percentage of real responders in the b-th bin of REAL kN . Pkbreal serves as a proxy for the true
prior probability. In order to formalize the relationship between the average posterior
probabilities of TEST kN and the approximate real probabilities obtained from REAL kN , a
formal mapping is obtained using the binned training set of fold k by
Pkbreal = fk(Pkborg)
(2)
with fk being the classifier that maps the non-calibrated posterior probabilities onto the real
probabilities in fold k. After the classifier fk is built, it is applied to the unseen test data of
This paper is accepted for publication in European Journal of Operational Research
8
TEST kN to obtain the new posterior probabilities, Pknew, for every individual in the test data
set of the k-th fold. A new data set is obtained NEWkN which contains Pknew, the calibrated
posterior probabilities.
There are several possibilities for fk, a function that links the estimated, non-calibrated
probilities of TEST kb to the approximated real probabilities of REAL kb . This study uses one
probability-mapping approach based on generalized linear models (Section 3.2.1.) and three
non-linear approaches; one based on generalized linear models with log-transformed noncalibrated probabilities (Section 3.2.2.) and two approaches based on generalized additive
models (Section 3.2.3. and Section 3.2.4.).
3.2.1 Generalized linear model (GLM)
Given yi as the dependent variable with y i
0,1 representing Pkbreal, the averaged true prior
probabilities from REAL kb and xi equal to Pkborg, the averaged posterior probabilities of
TEST kb , a generalized linear model with logit link function is employed to model fk(xi)
0,1 .
Moreover, it assumes that the relationship between Pkborg and Pkbreal is linear in the log-odds
via
logit y i
log
yi
1 yi
αk
β ki x i
(3)
or
yi ≡ fk(xi) = logit -1 (α k
β ki x i )
(4)
with α k as the intercept and β ki x i as the predictor. The parameters α k and β ki are estimated using
maximum likelihood (Tabachnick & Fidell, 1996).
3.2.2. Generalized linear model with log transformation (LOG)
This paper is accepted for publication in European Journal of Operational Research
9
Another approach is to log-transform xi in equation (3) and equation (4), because as such one
captures the non-linearity in the log-odds space between yi, Pkbreal the true prior probabilities
from REAL kb , and xi, Pkborg the posterior probabilities of TEST kb .
3.2.3. Generalized additive models
An attractive alternative to standard generalized linear models is generalized additive models
(Hastie & Tibshirani, 1986, 1987, 1990). Generalized additive models relax the linearity
constraint and apply a non-parametric non-linear fit to the data. In other words, the data
themselves decide on the functional form between the independent variable and the dependent
variable. Define yi as the dependent variable with y i
0,1 representing Pkbreal, the true
posterior probabilities from REAL kb , and xi equals to Pkborg, the posterior probabilities of
TEST kb . To model fk(xi)
0,1 , generalized additive models with logit link function are
employed. Methodologically, generalized additive models generalize the generalized linear
model principle by replacing the linear predictor β ki x i in equation (4) with an additive
component where
yi ≡ fk(xi) = logit -1 (α k
s ki ( x i ))
(5)
with s ki ( x i ) as a smooth function. This study uses penalized regression splines s ki ( x i ) to
estimate the non-parametric trend for the dependency of yi on xi (Wahba, 1990; Green and
Silverman, 1994). These smooth functions use a large number of knots leading to a model
quite insensitive to the knot locations, while the penalty term is used to avoid the danger of
over-fitting that would otherwise accompany the use of many knots. The complexity of the
model is controlled by a parameter λ and it is inversely related to the degrees of freedom (df).
If λ is small (i.e. the df are large), a very complex model that closely matches the data is
employed. When λ is large (i.e. the df are small), a smooth model is considered. In order to
optimize the generalized additive model, the fitting amounts to penalized likelihood
maximization by penalized iteratively reweighted least squares (Wood, 2000; 2004; 2008).
This paper is accepted for publication in European Journal of Operational Research
10
3.2.4. Generalized additive models with monotonicity constraint
Due to the fact that generalized additive models produce a non-linear relationship between the
independent variable Pkborg and the dependent variable Pkbreal, the original ranking of the
posterior probabilities of TEST kN and its calibrated version may change. However, marketing
analysts could argue that the mapping from TRAIN M onto TEST N and the corresponding
ranking of the customers in TEST N (and respectively TEST kN ) given by the initial classifier
C should be conserved. As such a non-decreasing monotonicity constraint on the generalized
additive models predictions is introduced to retain the original ranking of the customers.
Inspired by rule-set creation advances in the post-learning phase (e.g. pedagogical rule-based
extraction techniques as employed in Martens et al. (2007)), a rule set on the training set of
fold k is produced in the post-estimation phase of the generalized additive models to obtain a
function fk’, a non-decreasing monotone function. This ensures that the initial ranking of Pkborg
is maintained in the corresponding predictions Pkbreal of fold k. Practically, the training set is
sorted by Pkborg. Afterwards the rule-based algorithm detects all non-decreasing monotonic
inconsistencies on the prediction values fk(Pkborg) on the training set. For instance, suppose
that the prediction value for bin X+1 is lower than the prediction value for bin X than the rulebased algorithm adds a rule to the rule-base to change the prediction value of bin X+1 to the
larger prediction value of bin X. In the end, the generalized additive model and the rule-base
describe a non-decreasing monotone generalized additive model based function fk’ with
following characteristics (Denlinger, 2010) if
PkborgX ≤ PkborgX+1 => fk’(PkborgX) ≤ fk’(PkborgX+1)
(7)
with PkborgX and PkborgX+1 original non-calibrated posterior probabilities for bins X and X+1 in
the training data set, and fk’(PkborgX) and fk’(PkborgX+1) the calibrated posterior probabilities in
fold k for bins X and X+1.
This paper is accepted for publication in European Journal of Operational Research
11
4. Empirical validation
The calibration methods are employed on a test bed of 9 real-life direct marketing datasets
provided by a large European financial institution. Each of these datasets corresponds to a
typical financial product. Table 1 shows the characteristics of the response datasets.
[INSERT TABLE 1 OVER HERE]
With the aim of methodologically comparing the different algorithms, a 10-fold crossvalidation is applied. Furthermore, the classifier C which links TRAIN M and TEST N and
outputs Porg is a logistic regression with forward variable selection as it is a robust and wellknown classification technique in the marketing environment (Neslin et al., 2006). Moreover,
the calibration approaches based on generalized additive models use different levels of
degrees of freedom (df) representing the non-linearity of the model. The higher the df, the
higher the non-linearity. On the hand, the df are set manually by the researcher (userspecified), while on the other hand the df are simultaneously estimated in correspondence
with the shape of the response function (automatic). This study opts to manually set the df
equal to {3,4,5} (resulting in GAMdf and GAMdf MONO). This df range is inspired by the
recommendation and the applications in Hastie & Tibshirani (1990) and Hastie et al. (2001)
that use a relatively small number of df to account for different levels of non-linearity.
Additionally, the generalized cross-validation procedure (GCV) is employed to automatically
select the ideal number of df, resulting in GAMgcv and GAMgcv MONO (Gu & Wahba,
1991; Wood, 2000; 2004). The number of bins b for TEST kN and REAL kN is set to 200.
Furthermore, Porg, the non-calibrated posterior probabilities of TEST N , are used as a
benchmark (ORIGINAL). The different algorithms are compared on an individual customer
level using the log-likelihood (LL) by
N
LL
p( x i ) y i 1 p( x i )
ln(
i 1
1 yi
N
)
y i ln p( x i )
(1 y i ) ln 1 p( x i )
(8)
i 1
with N the number of customers, p( x i ) equal to Pknew, the calibrated posterior response
probability, and yi as the real response variable with y i
0,1 . The LL is a well-known
This paper is accepted for publication in European Journal of Operational Research
12
metric in (direct) marketing to evaluate the performance of an algorithm (e.g. Baumgartner &
Hruschka, 2005). The higher the LL, the better the calibration of the posterior probabilities to
the true response distribution is. Moreover, the non-parametric Friedman test (Demšar, 2006;
Friedman, 1937, 1940) with the Bonferroni-Dunn test (Dunn, 1961) is used in order to
significantly compare the different approaches with the best performing algorithm.
5. Results
Table 2 represents the 10-fold cross-validated log-likelihood values for the different datasets
and the different algorithms. Three panels (a,b,c) are included representing the various levels
of the user-selected degrees of freedom for the generalized additive model mappings. For
each dataset, the best performing algorithm in terms of log-likelihood is put in italics.
Moreover, the average ranking (AR) per algorithm over the different datasets is given. The
lower the ranking, the better the algorithm is shown to be. The best performing algorithm is
underlined and set in bold, while the algorithms that are not significantly different to the best
one at a 5% significance level are only set in bold.
[INSERT TABLE 2 OVER HERE]
The algorithms are split into 4 categories; the original, non-calibrated posterior probabilities
(ORIGINAL), the rescaling methodology (SAERENS), the linear probability-mapping
approach (GLM) and the non-linear probability-mapping approaches (LOG, GAMdf, GAMdf
MONO, GAMgcv and GAMgcv MONO).
Table 2 reveals that calibrating the posterior probabilities has a beneficial impact when a
discrepancy exists between the true prior probabilities of the training set and the test set:
ORIGINAL always performs worse than the other calibration approaches.
This paper is accepted for publication in European Journal of Operational Research
13
Comparing the rescaling approach (SAERENS) with the best performing calibration
approaches, one concludes that SAERENS always significantly performs less well than the
non-linear probability-mapping approaches, while SAERENS performs better than the linear
probability-mapping approach (GLM). These results show that the analyst better shifts
towards a non-linear probability-mapping approach, despite the fact that SAERENS is an easy
and workable solution to the calibration problem.
Contrasting the various probability-mapping approaches, Table 2 discloses that the non-linear
calibration approaches (LOG, GAMdf, GAMdf MONO, GAMgcv and GAMgcv MONO) are
always amongst the best performing algorithms. The linear mapping approach (GLM) is never
significantly competitive with one of its non-linear counterparts. However, the generalized
linear model with log-transformation (LOG) is competitive to the more advanced GAM
approaches (GAMdf, GAMdf MONO, GAMgcv and GAMgcv MONO). Within the nonlinear calibration setting, one concludes that GAMgcv MONO always performs best, followed
by the other non-linear calibration approaches.
Table 3 contains the performance measures for all generalized additive models approaches
(GAMdf, GAMdf MONO, GAMgcv and GAMgcv MONO), for all the levels of degrees of
freedom. On a dataset level, the best performing algorithm is put in italics. Furthermore, the
average ranking (AR) for each algorithm is given and the best performing algorithm (i.e. the
one with the lowest ranking) is underlined and set in bold, while the ones that are not
significantly different to the best at a 5% significance level are simply put in bold.
[INSERT TABLE 3 OVER HERE]
Table 3 reveals that GAM5 MONO is the best performing algorithm amongst the GAM and
GAM MONO approaches, quickly followed by GAMgcv MONO. Table 3 shows a better
performance trend for the GAM approaches when the number of df are increased. GAM3
performs less well than GAM4, while GAM4 has a less well performance than GAM5.
Furthermore, it is clear that including the monotonicity constraint has a beneficial impact on
This paper is accepted for publication in European Journal of Operational Research
14
the calibration performance of the GAM approaches. The average ranking of the GAM
approaches including the monotonicity constraint is always better than their original GAM
counterparts (i.e. GAMdf versus GAMdf MONO and GAMgcv versus GAMgcv MONO).
Moreover, the automatic smoothness parameter selection procedure proves its beneficial
impact. For the non-monotonicity models, GAMgcv has always a better ranking than the
GAMdf approaches. For the monotonicity models, GAMgcv MONO performs always better
than GAM3 MONO and GAM4 MONO, while GAMgcv MONO is very competitive to
GAM5 MONO.
6. Discussion
The results suggest that marketing analysts should calibrate the posterior probabilities when
the training set does not represent the true prior distribution. In general, calibrating the
posterior probabilities is more beneficial than using the non-calibrated posterior probabilities.
Moreover, it is shown that a ‘simple’ rescaling algorithm (SAERENS) that takes into account
the ratio of the old and the new priors is not sufficient to be a first and workable solution to
initially solve the calibration problem. SAERENS always performs significantly worse than
the more complex non-linear probability-mapping approaches. Furthermore, marketing
researchers should better not apply the linear probability-mapping approach in this specific
setting. Indeed, amongst the different probability-mapping approaches, it has been shown that
non-linear approaches are preferable over the linear mappings. The LOG approach is
competitive to the more complex GAM-based calibration approaches, and because it is based
on the common generalized linear model framework, LOG could be seen as a first and
workable approach. However if one is interested to optimize the calibration performance, the
GAM-based approaches are preferable. Moreover, one concludes that using the automatic
smoothing parameter selection procedure and imposing a monotonicity constraint on the
GAM method are the most preferred options to be employed in GAM models in order to
optimize calibration performance.
7. Conclusion
Direct marketing receives considerable attention these days in academia as well as in business
due to a serious drop in the cost of IT equipment and the ever increasing usage of response
This paper is accepted for publication in European Journal of Operational Research
15
models in a variety of business settings. In a direct marketing context, a discrepancy
sometimes exists between the prior distributions on the training set and scoring set which is
problematic. This may happen due to the fact that the training set consists entirely of
customers previously selected by a response model, and thus this dataset consists of a higher
percentage of responders. Applying a classification model built on this training set to the
complete set of customers will harm the estimation of the response probabilities. Thoroughly
adjusting the posterior probabilities to the real response probability distribution will improve
the classification performance. This study reveals that the non-linear probability-mapping
approaches are amongst the best performing algorithms and their usage is highly
recommended in a day to day business setting for following reasons. Firstly, the non-linear
probability-mapping approaches deliver a better performance compared to the other
calibration algorithms included in this research paper. This leads to the fact that the calibrated
probabilities better reflect the true probabilities of response. Secondly, there is a possibility to
visualize the relationship between Pkborg and Pkbreal. This gives managers a better and visual
understanding of the calibration process for a particular setting. For instance, the more the
calibration curve is away from the 45 degree line (,i.e. the line where Pkborg=Pkbreal or no
calibration is necessary), the higher the added value of sending a leaflet because the incidence
in TRAIN M is higher than in REAL N . Finally, the underlying techniques like generalized
linear models and generalized additive models are easily implementable in today’s business
environment due to the availability of the classifiers in traditional software packages like SAS
and R.
Whilst we are confident that our study adds significant value to the literature, valuable
directions for future research are identified. Beside the probability-mapping approaches which
map the Pkborg onto the Pkbreal, an extensive research project could be dedicated to investigate
the impact of ‘integrated’ calibration approaches, i.e. methods that integrate the calibration
process into the initial training phase of classifier C in order to come up with a new classifier
C’ which directly outputs calibrated probabilities. For instance, a workable ‘integrated’
calibration approach could be represented by a two-stage Bayesian logistic regression
approach that directly outputs calibrated posterior probabilities. In order to obtain this
‘integrated’ Bayesian calibration model, the following procedure is proposed. Under the
assumption that the commonly-used prior distribution for β ki is multivariate Gaussian, i.e.
p( β ki )~N( β 0 ,∑0), the Bayesian empirical approach could be used to specify the values of
This paper is accepted for publication in European Journal of Operational Research
16
β 0 and ∑0 by fitting a Bayesian logistic regression to TRAIN kM using non-informative priors.
Consequently, the resulting posterior mean vector and variance-covariance matrix of this
initial model could then be used for the values of β 0 and ∑0 for the second Bayesian logistic
regression on REAL kN . The resulting ‘integrated’ Bayesian logistic regression approach C’
will directly output adapted, calibrated posterior probabilities1. Furthermore, the probabilitymapping approaches are validated in a direct marketing setting, whereas future research
efforts could be spent to investigate the external validity to other operational research settings.
Acknowledgements
The authors would like to thank the anonymous company for freely distributing the datasets.
We would like to thank our friendly and journal reviewers for their fruitful comments on
earlier versions of this paper and the editor, Jesus Artalejo, for guiding this paper through the
reviewing process.
1
Nevertheless, this approach is not tested in the current version of the paper for confidentiality reasons.
This paper is accepted for publication in European Journal of Operational Research
17
References
Allenby, G. M., Leone, R. P., & Jen, L. C. (1999). A dynamic model of purchase timing with
application to direct marketing. Journal of the American Statistical Association, 94, 365-374.
Baesens, B., Viaene, S., Van den Poel, D., Vanthienen, J., & Dedene, G. (2002). Bayesian
neural network learning for repaeat purchase modeling in direct marketing. European Journal
of Operational Research, 138, 191-211.
Baumgartner, B., & Hruschka, H. (2005). Allocation of catalogs to collective customers based
on semiparametric response models. European Journal of Operational Research, 162, 839849.
Bose, I., & Chen, X. (2009). Quantitative models for direct marketing: A review from systems
perspective. European Journal of Operational Research, 195, 1-16.
Bosio, S., & Righini, G. (2007). Computational approaches to a combinatorial optimization
problem arising from text classification. Computers & Operations Research, 34, 1910-1928.
Bult, J. R. (1993). Semiparametric Versus Parametric Classification Models: An Application
to Direct Marketing. Journal of Marketing Research, 30, 380-390.
Conforti, D., & Guido, R. (2010). Kernel based support vector machine via semidefinite
programming: Application to medical diagnosis. Computers & Operations Research, 37,
1389-1394.
Crone, S. F., Lessmann, S., & Stahlbock, R. (2006). The impact of preprocessing on data
mining: An evaluation of classifier sensitivity in direct marketing. European Journal of
Operational Research, 173, 781-800.
Deichmann, J., Eshghi, A., Haughton, D., Sayek, S., & Teebagy, N. (2002). Application of
multiple adaptive regression splines (MARS) in direct response modeling. Journal of
Interactive Marketing, 16, 15-27.
This paper is accepted for publication in European Journal of Operational Research
18
Demšar, J. (2006). Statistical comparisons of classifiers over multiple data sets. Journal of
Machine Learning Research, 7, 1-30.
Denlinger, C.G. (2010). Elements of real analysis. Jones and Bartlett Publishers.
Dunn, O. J. (1961). Multiple comparisons among means. Journal of the American Statistical
Association, 56, 52-64.
Friedman, M. (1937). The use of ranks to avoid the assumption of normality implicit in the
analysis of variance. Journal of the American Statistical Association, 32, 675-701.
Friedman, M. (1940). A comparison of alternative tests of significance for the problem of m
rankings. The Annals of Mathematical Statistics, 11, 86-92.
Green, P.J. & Silverman, B.W. (1994). Nonparametric regression and generalized linear
models. Chapman and Hall/CRC Press.
Gu, C., & Wahba, G. (1991). Minimizing GCV/GML scores with multiple smoothing
parameters via the Newton method. SIAM Journal of Scientific and Statistical Computing, 12,
383-398.
Hastie, T., & Tibshirani, R. (1986). Generalized additive models. Statistical Science, 1, 297318.
Hastie, T., & Tibshirani, R. (1987). Generalized Additive Models: Some applications. Journal
of the American Statistical Association, 82, 371-386.
Hastie, T., & Tibshirani, R. (1990). Generalized Additive Models. London: Chapman and
Hall.
Hastie, T., Tibshirani, R., & Friedman, J. (2001). The Elements of Statistical Learning: Data
Mining, Inference and Prediction. New York: Springer-Verlag.
Haughton, D., & Oulabi, S. (1997). Direct marketing modeling with CART and CHAID.
Journal of Direct Marketing, 11, 42-52.
This paper is accepted for publication in European Journal of Operational Research
19
Hruschka, H. (2010). Considering endogeneity for optimal catalog allocation in direct
marketing. European Journal of Operational Research, 206, 239-247.
Kim, H.S., & Sohn, S.Y. (2010). Support vector machines for default prediction of SMEs
based on technology credit. European Journal of Operational Research, 201, 838-846.
Lamb, C. W., Hair, J. F., & McDaniel, C. (1994). Principles of marketing (second ed.).
Cincinnati: Soulh-Westem Publishing Co.
Lee, H. J., Shin, H., Hwang, S. S., Cho, S., & MacLachlan, D. (2010). Semi-Supervised
Response Modeling. Journal of Interactive Marketing, 24, 42-54.
Martens, D., Baesens, B., Van Gestel, T., & Vanthienen, J. (2007). Comprehensible credit
scoring models using rule extraction from support vector machines. European Journal of
Operational Research, 183, 1466-1476.
Martens, D., Van Gestel, T., De Backer, M., Haesen, R., Vanthienen, J., & Baesens, B.
(2010). Credit rating prediction using Ant Colony Optimization. Journal of the Operational
Research Society, 61, 561-573.
Morales, D. R., & Wang, J. B. (2010). Forecasting cancellation rates for services booking
revenue management using data mining. European Journal of Operational Research, 202,
554-562.
Naik, P. A., Hagerty, M. R., & Tsai, C. L. (2000). A new dimension reduction approach for
data-rich marketing environments: Sliced inverse regression. Journal of Marketing Research,
37, 88-101.
Neslin, S. A., Gupta, S., Kamakura, W., Lu, J. X., & Mason, C. H. (2006). Defection
detection: Measuring and understanding the predictive accuracy of customer churn models.
Journal of Marketing Research, 43, 204-211.
Paleologo, G., Elisseeff, A., & Antonini, G. (2010). Subagging for credit scoring models.
European Journal of Operational Research, 201, 490-499.
This paper is accepted for publication in European Journal of Operational Research
20
Piersma, N., & Jonker, J.J. (2004). Determining the optimal direct mailing frequency.
European Journal of Operational Research, 158, 173-182.
Saerens, M., Latinne, P., & Decaestecker, C. (2002). Adjusting the outputs of a classifier to
new a priori probabilities: A simple procedure. Neural Computation, 14, 21-41.
Tabachnick, B. G. & Fidell, L. S. (1996). Using multivariate statistics. HarperCollings
Publishers, New York.
Wahba, G. (1990). Spline models for observational data. Society for Industrial and Applied
Mathematics (SIAM) Capital City Press, Montpelier (Vermont).
Wood, S.N. (2000). Modelling and Smoothing Parameter Estimation with Multiple Quadratic
Penalties. Journal of the Royal Statistical Society B, 62, 413-428.
Wood, S.N. (2004). Stable and efficient multiple smoothing parameter estimation for
generalized additive models. Journal of the American Statistical Association, 99, 673-686.
Wood, S.N. (2008). Fast stable direct fitting and smoothness selection for generalized additive
models. Journal of the Royal Statistical Society B, 70, 495-518.
Zahavi, J., & Levin, N. (1997). Applying neural computing to target marketing. Journal of
Direct Marketing, 11, 76-93.
This paper is accepted for publication in European Journal of Operational Research
21
k = 1 to 10
TRAINM
C
TESTN
LIN | LOG|
GAM|GAM MONO
ORIGINAL | SAERENS
TESTkN
TESTk1
TESTk2
TESTk3
….
….
TESTkb
TESTkN
NEWkN
NEWN
NEWk1
NEWk2
NEWk3
….
….
NEWkN
NEW1
NEW2
NEW3
….
….
NEWN
REALkN
REALN
REALk1
REALk2
REALk3
….
….
REALkb
Figure 1: Methodological framework.
This paper is accepted for publication in European Journal of Operational Research
22
Dataset ID
1
2
3
4
5
6
7
8
9
TRAINM
TESTN
# customers
% responders
# customers
% responders
70,463
1.29%
119,329
0.18%
56,301
2.40%
119,104
0.44%
23,328
7.57%
117,7433
0.14%
9,027
11.94%
305,567
0.57%
14,946
17.11%
1,073,346
0.18%
14,586
5.04%
1,223,703
0.05%
25,660
3.10%
748,602
0.18%
12,603
0.56%
127,651
0.24%
19,190
0.95%
113,496
0.23%
Table 1: Dataset characteristics.
This paper is accepted for publication in European Journal of Operational Research
# variables used by C
10
16
19
12
22
11
14
10
18
23
RESCALING
Panel a
PROBABILITY-MAPPING
NON-LINEAR
LINEAR
DATASET
1
2
3
4
5
6
7
8
9
AR
ORIGINAL
SAERENS
GLM
LOG
GAM3
GAMgcv
-242.91
-479.55
-998.78
-223.14
-243.69
-9884.39
-3823.20
-17802.90
-5493.35
7.67
-179.07
-306.81
-1280.32
-206.79
-140.90
-1192.41
-1032.46
-1290.20
-510.03
5.00
-202.76
-323.14
-1064.30
-223.00
-246.53
-1189.09
-1025.02
-1297.41
-525.81
6.89
-178.02
-304.73
-980.99
-206.29
-140.36
-1173.78
-1016.21
-1294.45
-506.11
3.22
-177.88
-306.83
-982.94
-206.69
-142.71
-1165.40
-1017.24
-1291.47
-523.01
4.44
-180.22
-304.23
-980.74
-207.16
-145.56
-1163.68
-1016.52
-1290.74
-515.27
3.78
RESCALING
Panel b
1
2
3
4
5
6
7
8
9
AR
ORIGINAL
SAERENS
GLM
LOG
GAM4
GAMgcv
-242.91
-479.55
-998.78
-223.14
-243.69
-9884.39
-3823.20
-17802.90
-5493.35
7.66
-179.07
-306.81
-1280.32
-206.79
-140.90
-1192.41
-1032.46
-1290.20
-510.03
5.22
-202.76
-323.14
-1064.30
-223.00
-246.53
-1189.09
-1025.02
-1297.41
-525.81
6.88
-178.02
-304.73
-980.99
-206.29
-140.36
-1173.78
-1016.21
-1294.45
-506.11
3.22
-178.06
-305.68
-983.81
-206.48
-146.48
-1164.36
-1016.80
-1290.92
-522.89
4.55
-180.22
-304.23
-980.74
-207.16
-145.56
-1163.68
-1016.52
-1290.74
-515.27
3.88
RESCALING
Panel c
ORIGINAL
SAERENS
GLM
GAM4
MONO
-177.40
-305.68
-980.17
-206.37
-140.02
-1164.06
-1016.27
-1292.02
-507.05
2.77
GAMgcv
MONO
-177.40
-303.60
-979.81
-206.61
-139.96
-1163.46
-1016.04
-1291.86
-505.91
1.77
GAM5
MONO
-177.38
-304.86
-979.81
-206.33
-139.94
-1163.62
-1016.11
-1291.86
-506.63
2.33
GAMgcv
MONO
-177.40
-303.60
-979.81
-206.61
-139.96
-1163.46
-1016.04
-1291.86
-505.91
1.88
PROBABILITY-MAPPING
NON-LINEAR
LINEAR
DATASET
GAMgcv
MONO
-177.40
-303.60
-979.81
-206.61
-139.96
-1163.46
-1016.04
-1291.86
-505.91
1.56
PROBABILITY-MAPPING
NON-LINEAR
LINEAR
DATASET
GAM3
MONO
-177.58
-307.03
-981.08
-206.56
-140.35
-1165.18
-1016.53
-1292.57
-507.20
3.44
LOG
GAM5
GAMgcv
1
-242.91
-179.07
-202.76
-178.02
-178.70
-180.22
2
-479.55
-306.81
-323.14
-304.73
-305.46
-304.23
3
-998.78
-1280.32
-1064.30
-980.99
-982.74
-980.74
4
-223.14
-206.79
-223.00
-206.29
-206.52
-207.16
5
-243.69
-140.90
-246.53
-140.36
-149.81
-145.56
6
-9884.39
-1192.41
-1189.09 -1173.78
-1163.91 -1163.68
7
-3823.20
-1032.46
-1025.02 -1016.21
-1016.59 -1016.52
8
-17802.90
-1290.20
-1297.41 -1294.45
-1290.75 -1290.74
9
-5493.35
-510.03
-525.81
-506.11
-522.79
-515.27
AR
7.66
5.11
6.88
3.33
4.66
4.00
* 10-fold CV LL values, AR = average ranking
Table 2: The 10-fold cross-validated log-likelihood values.
Panel a: overview with GAM3 & GAM3 MONO.
Panel b: overview with GAM4 & GAM4 MONO.
Panel c: overview with GAM5 & GAM5 MONO.
This paper is accepted for publication in European Journal of Operational Research
24
NON-LINEAR
GAM3
GAM4
GAM5
GAM3
GAM4
GAM5
MONO
MONO
MONO
1
-177.88
-177.58
-178.06
-177.40
-178.70
-177.38
2
-306.83
-307.03
-305.68
-305.68
-305.46
-304.86
3
-982.94
-981.08
-983.81
-980.17
-982.74
-979.81
4
-206.69
-206.56
-206.48
-206.37
-206.52
-206.33
5
-142.71
-140.35
-146.48
-140.02
-149.81
-139.94
6
-1165.40
-1165.18
-1164.36
-1164.06
-1163.91 -1163.62
7
-1017.24
-1016.53
-1016.80
-1016.27
-1016.59 -1016.11
8
-1291.47
-1292.57
-1290.92
-1292.02
-1290.75 -1291.86
9
-523.01
-507.20
-522.89
-507.05
-522.79
-506.63
AR
6.55
5.55
5.88
3.66
5.22
2.11
* 10-fold CV LL values, AR = average ranking
Table 3: The 10-fold cross-validate log-likelihood values for
GAM and GAM MONO calibration models.
DATASET
This paper is accepted for publication in European Journal of Operational Research
GAMgcv
-180.22
-304.23
-980.74
-207.16
-145.56
-1163.68
-1016.52
-1290.74
-515.27
4.55
GAMgcv
MONO
-177.40
-303.60
-979.81
-206.61
-139.96
-1163.46
-1016.04
-1291.86
-505.91
2.33
25