A Bayesian Model for Sales Forecasting at Sun
Microsystems
Phillip M. Yelland, Shinji Kim, Renée Stratulate1
Sun Microsystems Laboratories, MS UMPK16-158, 16 Network Circle, Menlo Park, U.S.A.
Abstract
An accurate short-term forecast of product sales is vital for the smooth operation of modern supply chains, especially where the manufacture of complex products is outsourced internationally.
As a vendor of enterprise computing products whose business model has long emphasized extensive outsourcing, Sun Microsystems Inc. has a keen interest in the accuracy of its product sales
forecasts. Historically, the company has relied on a judgment-based forecasting process, involving
its direct sales force, marketing management, channel partners, and so on. Management recognized, however, the need to address the many heuristic and organizational distortions to which
judgment-based forecasting procedures are prey. Simply replacing the judgmental forecasts by
statistical methods with no judgmental input was unrealistic; short product life cycles and volatile
demand demonstrably confounded purely statistical approaches. This article documents a forecasting system developed in Sun’s research laboratory and currently deployed by the company
that uses Bayesian methods to combine both judgmental and statistical information. We discuss
the development and architecture of the system, including steps that were taken to ease its incorporation into the company’s existing forecasting and planning processes. We also present an
evaluation of the system’s forecasting performance, as well as possible directions for future development.
Key words: Sales forecast, Bayesian statistics, prior elicitation, Markov chain Monte Carlo
Email address: [email protected], [email protected],
[email protected].
1
The authors gratefully acknowledge the invaluable contributions of Dr. Eunice Lee (Sun Worldwide Operations) to the work described here.
1
A Bayesian Model for Sales Forecasting at Sun Microsystems
Introduction
Sun Microsystems
Founded in 1982, and currently headquartered in Santa Clara, California, Sun Microsystems Inc. is a premier supplier of enterprise computing products, employing some
33,500 people worldwide. In the fiscal year ending in June 2008, its revenues amounted to
approximately $14 billion. Sun’s sales offerings range from microprocessors to IT services,
but the bulk of its income is derived from sales of computer servers and storage systems,
which range in price from below $1,000 to more than $10 million each. 2
Throughout Sun’s history, the company’s business model has been distinguished by a
very high degree of outsourcing, with a heavy reliance on contract manufacturers and
resellers. Baldwin and Clark (1997) relate that on the whole, this strategy has proven
very advantageous to Sun, and indeed the past few years have witnessed the adoption
of similar approaches by the company’s competitors (Davis 2007, Dean 2007). It does,
however, present significant challenges for supply chain management—challenges that
are compounded by the character of computer products themselves, which have lifecycles
measured in months or even weeks, and parts that can depreciate catastrophically in inventory. Accordingly, Sun has devoted considerable effort to the engineering of its supply
chain, as exemplified by its recently-implemented One Touch program, which involves extensive use of drop shipment and cross-docking (Whiting 2006). Nonetheless, as articles
such as (de Kok, Janssen, van Doremalen, van Wachem, Clerkx, and Peters 2005) recount,
the effectiveness of even the most agile and well-coordinated supply chain rests on accurate forecasts of demand—in fact in a survey documented by Wisner and Stanley (1994),
managers of lean supply chains like Sun’s placed a greater reliance on demand forecasts
than did those operating more traditional supply chains.
Forecasting at Sun
Sun’s supply chain forecasting process, in common with many of those in industry, revolves around predictions for product sales over a quarterly time horizons—generally the
current and next two fiscal quarters, in Sun’s case. These numbers serve as input to planning and procurement processes. The quarterly sales of a small selection of Sun’s server
and storage products are displayed in figure 1. 3
2
All information current as of February 2009—c.f. (SMCI).
In the interests of confidentiality, sales data appearing in the figure are mildly disguised. A label
such as
on the horizontal axis of a graph in the figure denotes the third financial quarter of the
second year of the product’s life.
3
2Q3
2
8000
30
6000
20
4000
10
2000
0
0
Figure 1. Sample product demands (units)
3
6Q1
40
Q4
10000
Q3
12000
50
Q2
60
5Q1
0
Q4
150
Q3
200
Q2
300
Q4
Q3
6Q1
Q3
5Q1
Q3
4Q1
Q3
3Q1
0
4Q1
500
2Q1
2000
Q3
1Q1
Q3
Q4
2Q1
Q2
Q3
Q4
3Q1
Q2
Q3
Q4
4Q1
Q2
Q3
Q4
5Q1
Q2
Q3
Q4
6Q1
2500
Q3
Q4
2Q1
Q2
Q3
Q4
3Q1
Q2
Q3
Q4
4Q1
Q2
Q3
Q4
5Q1
Q2
Q3
Q4
6Q1
Q2
Q3
Q4
2Q1
Q2
Q3
Q4
3Q1
Q2
Q3
Q4
4Q1
Q2
Q3
Q4
5Q1
Q2
Q3
Q4
6Q1
3000
Q2
3Q1
Q3
Q4
2Q1
Q2
Q3
Q4
3Q1
Q2
Q3
Q4
4Q1
Q2
Q3
Q4
5Q1
Q2
Q3
Q4
6Q1
A Bayesian Model for Sales Forecasting at Sun Microsystems
2500
2000
1500
1500
1000
1000
500
0
250
8000
6000
100
4000
50
2000
0
A Bayesian Model for Sales Forecasting at Sun Microsystems
Before the rollout of the program described in this paper, Sun’s sales forecasting rested
heavily on the production of so-called judgmental forecasts (Lawrence, O’Connor, and Edmundson 2000), assembled by polling the direct sales force and channel representatives
and combining the results in committee with input from marketing, sales and senior executives. Sun is hardly alone in such a practice; judgment has long formed the basis of
commercial sales forecasting. It figures prominently, for example, in an instructional article written in the 1950s by Boulden (1958), and surveys published in recent decades—
including (Dalrymple 1987, Wisner and Stanley 1994, Sanders and Manrodt 1994, Klassen
and Flores 2001, Sanders and Manrodt 2003, Fildes, Goodwin, Lawrence, and Nikolopoulos 2009)—attest to the continuing prominence of judgment in the forecasting processes
of the vast majority of companies. According to Sanders and Manrodt (2003), for example, of 240 US corporations surveyed, almost 90% based their sales forecasts exclusively
on judgment, and of those who used quantitative techniques, some 60% routinely made
judgment-based adjustments to the forecasts produced.
Certainly, some of the reasons for the continued widespread use of judgment in industry have little to do with improved forecast accuracy: Sanders and Manrodt (1994) cite
a lack of acquaintanceship with mathematical techniques and a desire to conflate forecasting with organizational goal-setting. But Lawrence, Goodwin, O’Connor, and Önkal
(2006) note that of late, the research community has come to recognize that judgment has
an important and technically legitimate role to play in forecasting. This is especially true—
as Sanders and Ritzman (2001) observe—in highly dynamic environments such as Sun’s,
where structural changes may undermine a statistical forecasting model. Moon, Mentzer,
and Smith (2003) insist that both judgment-based and statistical elements are indispensable to the sales forecasting process in particular—a point illustrated vividly in the episode
related by Worthen (2003), when Nike Corporation was compelled to take massive writeoffs against inventory, thanks to an exclusive reliance on statistical forecasts. Our experience at Sun also led us to conclude that judgmental input was vital, in that exclusively
statistical methods that dispensed with judgmental inputs were found to be largely ineffective at forecasting company product sales.
There is, however, a wealth of literature documenting the biases and errors associated with
judgment-based forecasting—c.f. (McGlothlin 1956, Tversky and Kahneman 1974, Wright
and Ayton 1986) or (Bolger and Harvey 1998), for example. Mentzer and Bienstock (1998)
and Tyebjee (1987) point out that in addition to these problems, judgmental sales forecasts may be distorted by other factors, such as organizational pressures. That commercial
sales forecasts are indeed subject to distortion is confirmed by recent survey data from
Fildes et al. (2009). Mindful of this, supply chain managers in Sun’s “Worldwide Operations” business unit looked for statistical techniques that could be used to enhance—but
4
A Bayesian Model for Sales Forecasting at Sun Microsystems
not supplant—the company’s judgmental forecasts, and engaged the research unit of Sun
(“Sun Labs”) to this end. The ensuing joint project culminated in the Sun Labs Forecasting
System, the suite of software described in this paper.
Related Work
Many researchers and practitioners have sought to combine judgmental and statistical
forecasting techniques. Broadly speaking, such efforts may be categorized under the headings: (1) adjustment, (2) combination, (3) correction, (4) judgmental bootstrapping and
(5) Bayesian methods. Work in each of these categories is reviewed briefly below.
Adjustment
Articles such as (Sanders and Manrodt 1994, Fildes and Goodwin 2007, Franses and Legerstee 2009) and (Fildes et al. 2009) evidence the widespread manual adjustment of statistical forecasts ex post facto—frequently, by “eyeball” analysis of forecast graphs, as described
in (Bunn and Wright 1991) and (Webby and O’Connor 1996). Without doubt, therefore, this
approach has intuitive appeal. However, Armstrong and Collopy (1998) point out that if
adjustments are carried out in an unstructured or undisciplined fashion, this approach
risks simply reintroducing the distortions of judgmental forecasting. In fact the accounts
of Fildes and Goodwin, Franses and Legerstee, and Fildes et al. (ibid.) suggest that more
often than not in practice, adjustments reduce the accuracy of a statistical forecast. Bunn
and Wright (1991) also aver that without sufficient ancillary detail, it can be difficult to
see how an apparently arbitrary adjustment was justified, leading to possible contention
within the organization employing the forecast.
Combination
An alternative approach—discussed by Blattberg and Hoch (1990), Webby and O’Connor
(1996), Lawrence et al. (2006) and Franses (2008)—is to originate statistical and judgmental
forecasts independently and use a mechanical procedure to combine them. This builds
upon work on forecast combination which began with (Bates and Granger 1969), and
which is surveyed by Clemen (1989) and Timmermann (2006). Combination is generally
effected by taking a weighted sum of the values of the constituent forecasts in each period.
A number of methods have been proposed to estimate the combination weights: Granger
and Ramanathan (1984) suggest least squares regression, for example, whereas Bates and
Granger (1969) compute weights based on the forecast errors of the constituent forecasts,
5
A Bayesian Model for Sales Forecasting at Sun Microsystems
and Diebold and Pauly (1990) use Bayesian shrinkage regression. In practice, a simple
unweighted average of the component forecasts—as expounded by Blattberg and Hoch
(1990) and Armstrong (2001b) amongst others—has been found to perform consistently
well.
Timmermann (2006) lists a number of factors that recommend forecast combination: It
synthesizes the information sets used to produce the component forecasts, it dilutes bias
in individual forecasts and it increases robustness with respect to model misspecification
and structural breaks. Set against this, Timmermann also notes that estimated combination weights can be very unstable in practice—which helps explain the remarkably good
relative performance of the simple average. In principle, avers Diebold (1989), where the
information sets underlying the component forecasts are available, 4 it is always preferable to construct a single encompassing forecast model, rather than simply to combine the
forecasts themselves.
Correction
Rather than combining a statistical and a judgmental forecast, some authors—including
Theil (1971), Ahlburg (1984), Moriarty (1985), Elgers, May, and Murray (1995) and Goodwin (1996, 2000)—have explored statistical methods for correcting judgmental forecasts in
the light of observed outcomes. Generally, such methods are based on Theil’s (ibid.) “optimal linear correction,” which involves regressing observed outcomes on forecasts, using
the estimated regression coefficients to produce a revised prediction from new forecasts;
Goodwin (1997) accommodates time-varying coefficients using a weighted regression.
A technique related to statistical correction originates with Lindley (1983), and is applied
explicitly to time series forecasting by West and Harrison (1997, sec. 16.3.2). Lindley’s
methodology is an example the so-called supra-Bayesian approach to the reconciliation of
expert opinion developed by Pankoff and Roberts (1968) and Morris (1974, 1983). Here,
the value of a judgmental forecast is construed as a linear function of the actual value, and
Bayesian updating is used to produce a revised forecast.
Judgmental bootstrapping
Researchers—particularly in psychology—have long sought to capture judgmental reasoning in a tractable mathematical form. Efforts centered on linear models date back at
4
Such a set might comprise a judgmental bootstrap model of the forecaster in the case of a judgmentbased forecast—see later.
6
A Bayesian Model for Sales Forecasting at Sun Microsystems
least to (Hughes 1917); important contributions were made by Meehl (1957), Hammond,
Hursch, and Todd (1964), Hursch, Hammond, and Hursch (1964) and Tucker (1964). Surveying this work, Dawes (1971) coined the term bootstrapping to describe the process by
which an expert’s judgment is modeled by a linear expression involving the environmental factors (usually referred to as cues) that enter into the expert’s consideration. Recent authors exploring the application of such a process to judgment-based forecasting, including
as Armstrong (2001a), O’Connor, Remus, and Lim (2005) and Batchelor and Kwan (2007)
use the qualified term judgmental bootstrapping to avoid confusion with the (quite distinct)
statistical bootstrap technique developed in the late 1970s by Efron (1979). 5 Armstrong
(ibid.) also applies the term to models that go beyond simple linear combination of cues.
Evidence for the efficacy of judgmental bootstrapping in forecasting is mixed: Ashton,
Ashton, and Davis (1994) find a bootstrap model out-performed by a statistical forecasting
model, and Åstebro and Elhedhli (2006) and Batchelor and Kwan (2007) cannot conclude
that a bootstrap model forecasts more accurately than the experts it represents. Lawrence
and O’Connor (1996) and Fildes et al. (2009) assert that bootstrapping is less effective in the
context of time series extrapolation, where cue information tends to be autocorrelated; the
“error bootstrapping” technique developed in response by Fildes (1991) seeks to model
the errors in a judgmental forecast, much like the forecast correction approach described
above.
Bayesian methods
The Bayesian paradigm of statistical inference (see e.g. (Gelman, Carlin, Stern, and Rubin 2003) for an overview), with its incorporation of subjective information in the form
of prior distributions, seems a natural means of combining judgmental and statistical elements in forecasting. Indeed, a substantial number of Bayesian models have been devised
for product demand, both at the inventory level (Silver 1965, Hill 1997, Dolgui and Pashkevich 2008a,b), and in the aggregate (Lenk and Rao 1990, Montgomery 1997, Moe and Fader
2002, Neelamegham and Chintagunta 1999, van Heerde, Mela, and Manchanda 2004, Neelamegham and Chintagunta 2004, Lee, Lee, and Kim 2008)—amongst many others. Sales
applications also feature prominently in the seminal work of Pole, West, and Harrison
(1994) and West and Harrison (1997) on Bayesian forecasting.
Bunn and Wright (1991) note that despite the apparent attractions of Bayesian modeling,
and a research literature that dates back several decades, there is a dearth of Bayesian
5
Alternatives terms, such as “the statistical approach,” “actuarial modeling” (both of which appear in the clinical literature) or the (mildly grandiloquent) “paramorphic representation” of Hoffman (1960) would doubtless invite even greater confusion.
7
A Bayesian Model for Sales Forecasting at Sun Microsystems
models with judgmental priors used routinely in forecast applications. Bunn and Wright
suggest that the chief impediment is the expense (in time and effort) of repeatedly eliciting
subjective priors of sufficient quality—a point reinforced by researchers such as Wright
and Ayton (1986), who highlight the difficulties involved in obtaining reliable judgmental
priors. In fact many of the models cited in the previous paragraph circumvent the need
for informative priors by relying on a hierarchical (Gelman and Hill 2006) structure to pool
information from analogous historical situations in order to produce forecasts.
System Overview
As the title of the paper suggests, we began the Sun Labs Forecasting System with a Bayesian model of product sales. Mindful of the need to incorporate judgmental information,
we built a model (described in detail in the next section) intended to formalize the judgmental framework used by the company’s forecasters. The model included parameters
specifying the level of sales achieved by a product in its mature phase, the time required
to achieve maturity after launch, and so on. Our intention was to elicit priors for these
parameters from the forecasters, using Bayesian updating to reconcile these priors with
actual sales and to extrapolate future sales. 6 This system relied on the frequent elicitation
of priors from the forecasting staff—ideally, each time they came into possession of new
information. We tried to minimize the work required of the forecasters to produce the
priors, constructing the model so that its parameters were easy to interpret, and implementing a number of graphical tools to assist with estimation. Despite this, we fell afoul
of a “Catch-22” (Heller 1961) that affirmed the conjecture of Bunn and Wright highlighted
above: Our colleagues were unwilling to invest significant time and effort until the efficacy of the Forecasting System had been proven, but without good priors, the System’s
performance was bound to be subpar.
Faced with this problem, we attempted first to deploy a hierarchical Bayesian model that
drew priors from historical records, 7 combining the statistical forecasts produced by the
model with the judgmental forecasts prepared by company forecasters. The heterogeneity
of Sun’s product line and rapid changes in its product markets made the performance of
the hierarchical model disappointing, however, and combinations of the model’s forecasts
with judgmental ones performed inconsistently.
The solution eventually adopted for use in the Forecasting System resembles the converse
of the judgmental bootstrapping process described in the previous section. For consider
6
7
An early version of the Forecasting System based on this idea is described in (Yelland 2004).
Again, see Yelland (ibid.) for details.
8
A Bayesian Model for Sales Forecasting at Sun Microsystems
that the company’s forecasters routinely produce judgmental sales forecasts, and that the
model used in the Forecasting System embodies the way in which those forecasts are assembled from beliefs about sales at maturity, time to reach maturity and so on. Therefore
fitting the model to the judgment-based sales forecasts produces estimates of the forecasters’ that underlie them—“cues” in the parlance of judgmental bootstrapping, or parameter
priors, to a Bayesian. In the Forecasting System, these prior estimates are then corrected
using a device similar to the linear correction used by Theil and his successors: Using
historical records of forecasts and actual sales, parameter estimates derived from actual
sales are regressed on estimates for the same parameters derived from forecasts of those
sales. The coefficients of the regression are used to produce adjusted prior estimates from
the new judgmental forecast. Finally, the adjusted prior estimates are revised by Bayesian
updating in the light of actual sales to produce the System’s forecast.
A functional schematic of the Sun Labs Forecasting System is given in figure 2. Details
of the forecasting model and a full description of prior estimation and correction are provided in the next section.
System Details
Model
Dynamic Linear Model
From its inception, the Sun Labs Forecasting System centered on the use of dynamic linear models (DLMs), as described by West and Harrison (1997) (hereinafter abbreviated
“W&H”). Bayesian formulations of the structural time series models also discussed from
a classical statistical viewpoint by Harvey (1989), 8 DLMs have been employed for sales
forecasting by a number of authors, including W&H themselves, van Heerde et al. (2004)
and Neelamegham and Chintagunta (2004). For reference, the general formulation of a
DLM representing a scalar time series (W&H, p. 102) is presented below:
Observation equation:
Evolution equation:
Initial information:
yt = Ft> θt + νt ,
θt = Gt θt−1 + ωt ,
θ0 ∼ N(m0 , C0 ).
8
νt ∼ N(0, Vt ),
(1a)
ωt ∼ N(0, Wt ),
(1b)
(1c)
The more recent account of Durbin and Koopman (2001) also concentrates on a classical treatment of structural models, with some attention is given to the Bayesian perspective.
9
A Bayesian Model for Sales Forecasting at Sun Microsystems
Processing step
Prior
deduction
Data
Sales force
forecasts
Initial parameter
estimates
Forecasting
model
Parameter
correction
Historical
records
Parameter
priors
Bayesian updating
Actual
sales
Revised
forecast
Figure 2. System schematic
Here, observation equation (1a) takes the form of a regression of the time series value, yt
(which in the case of the Forecasting System is quarterly product sales), on (the transpose
of) a design vector, Ft , of known covariates. The coefficients in this regression, θt , may vary
over time; they comprise the state vector of the model. Changes in the state vector itself
are described by the evolution equation (1b): In each time period, the state is subject to
a linear transformation and randomly perturbed by the addition of the noise component
ωt . 9 The specification is completed by the provision of a multivariate normal prior for the
state vector in the period preceding that of the first observation.
9
Noise terms νt and ωt are assumed to be uncorrelated.
10
A Bayesian Model for Sales Forecasting at Sun Microsystems
The DLM framework is extraordinarily flexible: Harvey, West and Harrison, Durbin and
Koopman (ibid.) demonstrate the encoding of trend, seasonal and dynamic regression
components—singly or in combination—as DLMs, as well as providing DLM versions of
the familiar ARIMA models (Box, Jenkins, and Reinsel 1994). As Harvey points out, structural time series models are generally more intuitively accessible than alternatives such as
the ARIMA models—an important consideration in view of our original intention to parallel the reasoning of Company forecasters. Unfortunately, we struggled to represent the
important characteristics of Sun’s product demands directly in DLM form; in particular,
we found DLMs ill-suited to the distinctive life cycle curve illustrated in figure 1, and the
multiplicative seasonality evinced by many products (the DLM formulation provides only
for additive seasonality). Obvious devices, such as log transformations, produced forecasts
that were far from satisfactory, and elaborations of the basic DLM such as “mixture” or
linearized models (W&H, chp. 12, 13, resp.) quickly became unwieldy. 10
The model currently used in the Sun Labs Forecasting System is actually derived from
the most basic form of DLM, called the first-order polynomial model by West and Harrison
(p. 32), and the random walk plus noise model by Harvey (ibid. p. 19). 11 The model is defined
as follows:
yt = S(t)L(t)µt + νt ,
νt ∼ N(0, V ),
(2a)
µ t = µ t −1 + ω t ,
ωt ∼ N(0, W ),
(2b)
µ0 ∼ N(m0 , C0 ).
(2c)
Here the state vector consists of a single scalar quantity, µt —usually referred to as the level
of the process—which evolves according to the random walk set out in equation (2b). In a
further simplification, the variances of the noise terms in equations (2a) and (2b) are constant over time. The only departure from the DLM framework is in the (scalar) quantity
corresponding to the design vector Ft in equation (1a); instead of the known (if possibly time-varying) values assumed in the usual DLM, the design vector in this model is
determined by the product of two stochastic quantities, S(t) and L(t), which represent respectively the effects of seasonality and life cycle stage in the period t. These are described
in further detail below.
10
See (Yelland and Lee 2003) for a description of some of our early investigations.
Incidentally, West and Harrison (loc. cit.) indicate that the first-order polynomial model is widely
used for forecasting in inventory and operations management applications like that of the Forecasting System.
11
11
A Bayesian Model for Sales Forecasting at Sun Microsystems
Seasonality
Seasonality is described straightforwardly using dummy variables (see Ghysels, Osborn,
and Rodrigues (2006), for example). Parameters Q1t , Q2t and Q3t give the size of the
multiplicative changes in the respective quarters of the financial year, relative to sales in
the fourth quarter of the year:
S(t) = 1 + Q1t ϕ1 + Q2t ϕ2 + Q3t ϕ3 ,
where Q jt =


1
if t is the jth quarter of a year,

0
otherwise.
Life cycle
Mirroring the conceptual framework generally used by Company forecasters, the model
divides a product’s life cycle into three phases: (1) a ramp up phase, in which product sales
increase consistently period over period, (2) a mature phase, in which product sales are
roughly constant and (3) a ramp down phase, in which product sales dwindle again. In the
model, lengths of these phases are delimited by the three positive random variables τ1 , τ2
and τ3 , respectively. The value of the life cycle coefficient varies between 0 and 1, the latter
corresponding to sales in maturity. The ramp up and down phases are characterized by
geometric growth and decay, defined by the positive constants α and β. 12 For the sake
of identification, β is set (arbitrarily) to 0.05, so that sales dwindle to 5% of mature sales
during the ramp down.
In symbols, the life cycle coefficient is defined as follows:


(τ1 −t)/τ1


α
L(t) = 1



 (t−τ1 −τ2 )/τ3
β
if 0 ≤ t < τ1 ,
if τ1 ≤ t < τ1 + τ2 ,
if τ1 + τ2 < t.
Figure 3 illustrates the general shape of the life cycle curve in the model, and Table 1
summarizes its interpretation.
Formal modeling of product life cycles has a long history, dating back (at least) to the
work of Rogers (2003, first published in 1962) and Bass (1969); see (Mahajan, Muller, and
Wind 2000a) for a compendium of work in the area. The vast majority of models of this
12
It is generally the case that α and β are less than 1. Values of α and β in excess of 1 are permissible,
however, though in such instances the value of L(t) may exceed 1 in some periods, and the epithets
“ramp up” and “ramp down” are probably inappropriate.
12
A Bayesian Model for Sales Forecasting at Sun Microsystems
L(t)
1
α
Ramp
up
Ramp
down
Maturity
β
0
τ1
0
τ1 + τ2
τ1 + τ2 + τ3
t
Figure 3. Life cycle coefficient over time
Life cycle phase
Span
Behavior
Ramp up
0 ≤ t < τ1
Geometric growth from a value α when t = 0 to 1
as t → τ1 .
Mature
τ1 ≤ t < τ1 + τ2
Constant value 1.
Ramp down
τ1 + τ2 < t
Geometric decay from 1 when t = τ1 + τ2 to β as
t → τ1 + τ2 + τ3 .
Table 1. Interpretation of the life cycle coefficient
sort are structural in nature; product demand emerges endogenously as a result of consumer responses to the experience of other consumers and to the marketing actions of
the producing firm. By contrast, the life cycle representation used in the Sun Labs Forecasting System is a purely empirical account of what Mahajan, Muller, and Wind (2000b)
refer to as strategy-driven diffusion—the life cycles of the products modeled by the Forecasting System do not result from consumer actions, but from strategic and operational
13
A Bayesian Model for Sales Forecasting at Sun Microsystems
decisions taken by Sun, its suppliers and its distributors. In particular, the ramp up phase
in the model reflects product quality assurance processes and supply chain inertia, and
the ramp down phase is initiated not by market saturation, as in the Bass model and its
successors, but by a decision on the part of the Company to withdraw the product in the
face of technological development. We should note that of late, mainstream work in the
area of life cycle modeling has employed piecewise representations like that used in the
Forecasting System—see (Niu 2006, Huang and Tzeng 2008) and (Sood, James, and Tellis
2009).
Prior Estimation
Regarding prior estimation, assume that a judgment-based forecast has been produced at
the end of quarter t a product’s life cycle for sales of the product in quarters t + 1, . . . , t + h.
Let ϑ ∈ {α, β, τ1 , τ2 , τ3 , ϕ1 , ϕ2 , ϕ3 , µ0 } 13 denote a model parameter for which a prior is
required. The prior estimation procedure is as follows:
(1) Prepend actual sales (y1 , . . . , yt ) to the forecast, producing a combined series ŷ of
length t + h.
(2) Using a non-linear least-squares regression, fit to the series ŷ the model (listing explicitly all the parameters that enter into S(t) and L(t) ):
yt = S(t; ϕ1 , ϕ2 , ϕ3 )L(t; α, τ1 , τ2 , τ3 )µ0 + νt .
(3)
Let ϑˆ + be the estimated value of ϑ in the above regression.
The model in equation (3) is a restriction of the full model in (2), without the level
evolution specified in equation (2b). In practice, we found the restriction necessary
to ensure reliable fitting of judgment-based forecasts—especially early in a product’s
life, where few actual demand points are available to supplement the forecast demands. The quality of the resulting priors was not substantially compromised.
(3) Collect together a set ŷ1 , . . . , ŷ N of combined forecasts and actuals for a set of recentlyobsolete products similar to the current one, 14 where the forecasts were made in the
corresponding quarter t of the products’ lives. Since these products are obsolete, their
entire sales histories y1 , . . . , yN are also available.
(4) Fit the model in equation (3) (using non-linear least squares again) to the series collections ŷ1 , . . . , ŷ N and y1 , . . . , yN . For i ∈ 1, . . . , N, let ϑˆ i be the estimate of the parameter
of interest that results from fitting to ŷi , with ϑi the estimate from the corresponding
13
The prior for µ0 actually constitutes the values m0 and C0 in (2c).
Sun’s products are generally categorized by market, such as “enterprise storage”, “low-end
server”, etc., and this categorization is used to identify like products.
14
14
A Bayesian Model for Sales Forecasting at Sun Microsystems
series yi . Thus ϑi represents the parameter value that characterizes the actual trajectory of the ith product’s sales, whereas ϑˆ i is the value suggested by the forecast in the
tth quarter of its life cycle.
(5) Fit a linear regression ϑi = a + bϑˆ i + υi to the estimates from step 4, 15 yielding coefficient estimates â and b̂ and an estimate of the residual standard deviation ς̂.
(6) The prior for ϑ derived from the value ϑˆ + estimated for the new forecast in step 2 is
given by standard results for classical normal regression (e.g. Gelman and Hill 2006,
p. 48)—i.e., a normal distribution with mean a + bϑˆ i and standard deviation:
# 12
1
(ϑˆ + − mean ϑˆ i )2
ς̂ 1 +
.
+
N ∑iN=1 (ϑˆ i − mean ϑˆ i )2
(7) The set of priors required for the model is completed by priors for the observation and
evolution noise variances in equation (2a) and equation (2b), respectively. Following
a rule of thumb we have found to work well in practice, these are set to relatively
diffuse scaled inverse χ2 distributions (see Gelman et al. 2003, p. 480) Inv–χ2 (4, 0.2 ×
m0 ) and Inv–χ2 (4, 0.1 × m0 ), resp., where m0 is the prior mean for µ0 .
"
Bayesian Updating
The Bayesian updating step in figure 2 (which revises the forecast priors in light of observed sales to date) is carried out using a Gibbs sampling routine, schematic descriptions
of which now abound in the literature—see (Gilks, Richardson, and Spiegelhalter 1996,
chp. 1), for example. The individual steps of this particular sampler are described in the
Appendix. Many of the steps rely on standard results concerning conjugate updating in
Bayesian analysis, which may be found in reference texts such as (Gelman et al. 2003) or
(Bernardo and Smith 1994). Where such closed-form updates are not available, we resort
to Metropolis-Hastings sampling (also discussed by Gilks et al.); proposals are generated
using Geweke and Tanizaki’s (2003)’s Taylored chain procedure, details of which are also
provided in the Appendix. The Gibbs sampler is run for 10,000 iterations, with the first
3,000 samples discarded; convergence is verified using Geweke’s (1992) single chain diagnostic (in additional to occasional sample-path plots). In fact execution of the sampler for
10,000 iterations is quite conservative—we are almost invariably able to establish convergence by about 4,000 iterations.
15
For parameters restricted to the positive half-line, the linear regression is fit on a log scale to
ensure coherent predicted parameter values. The resulting estimates are actually used to derive
truncated (not log-) normal distributions on the original scale (by matching first and second moments), as we have found that truncated normal priors make for far better forecast performance
than log-normal ones.
15
A Bayesian Model for Sales Forecasting at Sun Microsystems
Forecast Performance
To demonstrate the efficacy of the Sun Labs Forecasting System, we compare its forecast
performance with that of alternative forecasting methods, using the demand histories of a
sample of Sun’s products.
Setup
The test is conducted using a collection of 32 products, representing a cross-section of
Sun’s recent product lines. Of these products, 27 are randomly selected for calibration,
and holdout forecasting is performed on the remaining 5. 16 Point forecasts at horizons
of 1, 2 and 3 quarters are prepared for each quarter of the holdout products’ demand
histories, yielding some 77 forecast values in total for each method at each horizon.
Four forecast methods are compared in the test:
Sys
This forecasting method relies on posterior predictive distributions calculated by the
Forecasting System in the Bayesian updating step of figure 2. 17 With calibration
data (i.e. demand histories and judgmental forecasts) available at time t collectively denoted Dt , the posterior predictive distribution for the forecast value yt+h
at horizon h is given by p(yt+h | Dt ), which is itself calculated by marginalizing
over the posterior density for the model parameters (here represented by ϑ):
p ( y t + h | Dt ) =
∝
Z
Z
p(yt+h |ϑ ) p(ϑ | Dt )dϑ
(4)
p(yt+h |ϑ ) p( Dt |ϑ ) p(ϑ )dϑ.
(5)
The integral expression in equation (5) is readily approximated using the Gibbs
sampler specified in the Appendix, as described by Albert (2008), for example.
For the point forecast ŷt+h|t (the forecast for yt+h made in period t), we use the
mean of the posterior predictive distribution of yt+h .
DLM
16
This method is a straightforward implementation of a univariate dynamic linear
model, as set out in (W&H, chp. 5–8). Following West and Harrison’s prescription, the model comprises a second-order polynomial- (“local level with trend”)
The relatively large number of products in the calibration collection is necessary to ensure reliable operation of forecast methods
,
and
, which require a representative set of
products from each of the company’s product groups in order to correct priors and forecasts.
17 This is the standard approach to forecasting with Bayesian models—see (Neelamegham and
Chintagunta 1999, 2004), for instance.
Sys DLM
CJudg
16
A Bayesian Model for Sales Forecasting at Sun Microsystems
and four-period seasonal component. As Gardner and McKenzie (1989) suggest,
to improve forecasts at over longer horizons, the trend in the polynomial component is damped, so that it decays over time; the design and evolution matrices
necessary to accomplish this are adapted from analogous structures described for
the closely-related single source of error (SSOE) structural time series models in
(Hyndman, Koehler, Ord, and Snyder 2008, p. 48).
Formulae for updating and forecasting with this DLM are standard: We use the
“unknown, constant variance” results summarized on (W&H, p. 111), which incorporate the estimation of a time-invariant observation noise variance in equation (1a). Multiple discount factors are used to specify the evolution noise component in equation (1b)—c.f. op. cit. pp. 196–198 for details. Discount and damping
factors are derived using a grid search based on forecast accuracy for a sample
of the calibration set, and initial conditions are set from the same corrected priors
used for the Sys method, using a procedure similar to that described by Hyndman
et al. (ibid., sec. 2.6.1). All components in the state are uncorrelated in the prior, so
that the matrix C0 in (1c) is diagonal.
Judg
CJudg
This method simply reproduces the company’s judgmental forecast—the forecasts
from which the priors for methods Sys and DLM are derived.
For this method, the company’s judgmental forecast is “corrected” using Theil’s
(1971) optimal linear correction. Recall from the above that this involves regressing actual sales on predicted sales for products in the calibration set, using the
estimated coefficients to compute revised prediction from forecasts in the holdout
set—see Theil (1971, p. 34), or more recently, Goodwin (2000), for a detailed discussion. Separate regressions are calculated for each forecast horizon and each of
the product categories identified for prior correction.
Test Results
Table 2 summarizes the performance of the candidate methods in the forecasting test. We
have used the mean absolute scaled error (MASE) of Hyndman and Koehler (2006) as a performance metric; the Appendix defines the MASE, and sets out the considerations that
lead to its adoption. MASEs are given for each method at each forecast horizon—a smaller
entry indicates better performance. The rank (smallest to largest) of each method at each
horizon is recorded in parentheses after the corresponding entry. As the table shows, the
Forecast System consistently exhibits superior overall performance, with the purely statistical DLM method generally turning in the worst performance, and the corrected judgmental forecast also consistently outperforming its uncorrected counterpart. The MASE is
17
A Bayesian Model for Sales Forecasting at Sun Microsystems
Horizon
Sys
DLM
Judg
CJudg
1
2
3
0.53 (1)
0.84 (1)
0.80 (1)
0.73 (3)
1.13 (4)
0.97 (4)
0.73 (4)
1.10 (3)
0.96 (3)
0.66 (2)
1.00 (2)
0.94 (2)
Table 2. Mean absolute scaled error by forecast horizon
defined such that a value less than 1.00 indicates a performance better (on average) than
that of the “naïve” random-walk forecast, which simply uses the most recent observation
to produce a forecast. Thus according to the table, only the Sys method consistently improves on the benchmark.
For a more detailed perspective, Figure 4 is a box-and-whisker plot (Tukey 1977) of the
distribution of absolute scaled errors of the forecast methods at each forecast horizon.
Though the distributions of the ASEs are clearly skewed, the plot broadly confirms the
superiority of the Sys method established by the MASEs.
Implementation
A depiction of the current implementation of the Forecast System is given in figure 5.
Remarkably, we found it possible to implement the System almost entirely using open
source and/or freely-available software. The core of the System is written in the statistical
programming language R (Venables and Smith 2002). It resides on a dedicated server and
carries out the following operations:
• Calculating priors from judgmental forecasts,
• Updating the priors given actual product sales, and
• Forming sales predictions from the updated priors.
This portion of the System is run once a week to incorporate the latest sales figures. Input
to the R code comes from the company’s data warehouses by way of Java interface modules, and output is stored in a dedicated MySQL database (Williams and Lane 2004) on
the same server.
18
19
0
1
2
3
Sys
●
DLM
●
●
Judg
●
●
Horizon: 1
0
1
2
Sys
●
●
DLM
Judg
CJudg
●
0
1
2
3
●
●
●
Sys
Figure 4. Distribution of ASEs by model and forecast horizon
CJudg
●
●
3
Horizon: 2
DLM
●
●
●
Judg
●
●
●
Horizon: 3
CJudg
●
●
●
●
A Bayesian Model for Sales Forecasting at Sun Microsystems
A Bayesian Model for Sales Forecasting at Sun Microsystems
Manufacturing
Actual sales
Sun Labs
Forecasting
System
Data warehouse
Sales,
Forecasts
Web interface
Revised
forecasts
Sales,
Forecasts
Revised
forecasts
Forecast
meeting
Supply chain
management
Forecasts
Plans
Figure 5. How the Forecast System supports Sun’s supply chain management process
Access to the forecast data is Web-based, relying on the Adobe Flash/JavaScript/PHP/Apache combination common in modern Web applications (Wiedermann 2006, Schafer
2005). Figure 6 provides a (mildly disguised) snapshot of the forecast display for a typical
product: On the left of the page, the set of forecast products is arrayed by category, while
at the top of the page is a selection of the “base week” in which the forecast was produced,
allowing the user to review historical records as desired. The animated chart (provided
by amCharts: http://www.amcharts.com/) provides an interactive pictorial representation
of forecasts and actual sales, which are also presented in tabular form at the bottom of
the page. The system also provides forecasts in the form of PDF reports, produced on the
server using LATEX (Lamport 1994).
20
A Bayesian Model for Sales Forecasting at Sun Microsystems
Figure 6. Snapshot of Forecast System user interface
Concluding Remarks
The Sun Labs Forecasting System has been in use at Sun for over a year at the time of
writing. The System provides an effective combination of judgmental and statistical forecasting information that consistently improves upon the forecast accuracy of both its constituents. The system operates almost entirely unattended, extracting the requisite information from the Company’s data warehouses and delivering forecasts in a regular and
expeditious manner through a simple, intuitive user interface. Furthermore, with Bayesian priors derived automatically from the output of the existing sales forecasting process,
the Forecasting System imposes no additional procedural burdens on its users. Taken to-
21
A Bayesian Model for Sales Forecasting at Sun Microsystems
gether, these factors no doubt account for the warm reception the System has received
from the Company’s supply chain managers.
Development of the Forecasting System continues. Most immediately, we are investigating means by which forecasters can inform the System of upcoming events such as very
large orders or promotions that are likely significantly to perturb the sales of one or more
products. The technical mechanisms by which such interventions may be achieved are
fairly straightforward to implement—see (West and Harrison 1997, chp. 11) for a discussion. More delicate are the organizational issues involved, for articles such as Sanders and
Ritzman (1991), Fildes et al. (2009) and Sanders (2009) testify that although management
intervention generally increases forecast accuracy when managers are possessed of specific and accurate information about special events, forecasts are usually degraded if managers are allowed to intervene willy-nilly. Our current intentions are to provide some form
of decision support system—along the lines of that described by Lee, Goodwin, Fildes,
Nikolopoulos, and Lawrence (2007)—to help forecasters make only considered interventions.
The Forecasting System makes no attempt to provide normative information—such as inventory levels or purchase quantities (Zipkin 2000)—based on the forecasts it produces.
This is quite deliberate, since the complexity of Sun’s supply chain and the sourcing arrangements upon which it rests makes the derivation of policy recommendations from demand forecasts formidably complex; inventory-, depreciation- and expediting costs, costs
of lost sales, contractual supply arrangements, sales of complementary products and many
other factors vary from product to product, between different configurations of the same
product, and across quarters. The Forecasting System system therefore passes off responsibility for operating decisions to the users of its forecasts. A somewhat vexatious upshot
of this is that it is very difficult to place a firm dollar amount on the benefits that accrue
from superior accuracy of the System’s forecasts. Certainly, the suggestions of Mentzer
(1999) and Kahn (2003) provide means of reaching plausible rough estimates, but a truly
definitive figure would require a thorough-going appraisal of the company’s forecasting
and supply arrangements along the lines proposed by Moon et al. (2003).
Policy prescriptions are easier to make at a less aggregate level of demand: Purchasing
and inventory management recommendations are much more tractably computed from
a demand forecast for a single manufacturing part. In regard of this, we have begun to
apply many of the techniques used in the Forecasting System (such as Bayesian updating
and a semi-parametric life cycle representation) to the forecasting of part demands. At this
level of disaggregation, priors derived from judgmental forecasts of product demand are
frequently unreliable, because a multitude product configurations and frequent substitu-
22
A Bayesian Model for Sales Forecasting at Sun Microsystems
tion of manufacturing parts due to changes in technology or sourcing arrangements makes
the correspondence between product and part demands too unstable. However, we have
found hierarchical priors (Gelman and Hill 2006)—which pool information across sets of
parts—to be effective replacements. On a more pragmatic note, we should note that in developing a parts forecasting solution, our dealings with Sun’s operating units have been
greatly enhanced by the credibility garnered by the Sun Labs Forecasting System.
23
A Bayesian Model for Sales Forecasting at Sun Microsystems
Appendix
Gibbs Sampler
The following details the Gibbs sampler used for Bayesian updating in the Forecasting
System. Each step in the sampler is introduced by the full conditional distribution from
which a sample is to be drawn. Variables of which the sampled quantity is conditionally
independent are omitted from the conditioning set.
µ0 , . . . , µ T yt , λ, ϕ, V, W, m0 , C0
Conditionally on the other parameters of the model, µ0 , . . . , µ T constitutes a sample path of the state vector of a first-order polynomial dynamic linear model. Procedures for sampling the state of a DLM (generally under the moniker forward filtering/backwards sampling algorithms) are described by Frühwirth-Schnatter (1994),
Carter and Kohn (1994), West and Harrison (1997) and Durbin and Koopman (2001),
amongst others.
W µ0 , . . . , µ T
Again, a standard step in DLM sampling—c.f. (W&H, p. 568), for example.
α τ1 , τ2 , τ3 , yt , µt , ϕ, V
The full conditional is proportional to the expression:
"
T
∏ N(yt |L(t)S(t)µt ,V )
#
× N[0,∞) (α|µα , σα2 ).
t =1
This is sampled using a Taylored chain proposal in a Metropolis-Hastings step—the
calculation of the Taylored chain proposal is described in detail in the following section.
ϕ yt , µt , λ, V
Draw from the posterior distribution of the coefficients ϕ1 , ϕ2 , ϕ3 in the linear regression:
yt − L(t)µt = [L(t)µt Q1t ] ϕ1 + [L(t)µt Q2t ] ϕ2 + [L(t)µt Q3t ] ϕ3 + νt ,
given independent priors—see e.g. (Gelman et al. 2003, chp. 8).
24
νt ∼ N(0, V ),
A Bayesian Model for Sales Forecasting at Sun Microsystems
V yt , λ, ϕ, µt
The conditional likelihood of V is ∏tT=1 N(yt |S(t)L(t)µt ,V ); with prior V ∼ Inv–χ2 (ν, σ),
the posterior is given by conjugacy—see e.g. (Gelman et al. 2003, chp. 2).
The Geweke -Tanizaki (2003) 18 Taylored Chain
With x the current state of the sampler, to produce a proposal x + for a target kernel p(z),
let q(z) = log p(z), with q0 (z) and q00 (z) the first and second derivatives thereof. Proceed
by cases:
Case 1: q00 ( x ) < −ε, where ε is a suitable small constant, such as 0.1 . 19
Rewrite the Taylor expansion of q(z) around x:
1
q(z) ≈ q( x ) + q0 ( x )(z − x ) + q00 ( x )(z − x )2
2
2
1
q0 ( x )
= q( x ) − (−q00 ( x )) z − x − 00
2
q (x)
|
{z
}
‡
Since q00 ( x )
< 0, the component (‡) of the latter expression constitutes the exponential
part of a normal distribution, which implies that the target kernel in the vicinity of
x may be approximated by a normal distribution with mean x − q0 ( x )/q00 ( x ) and
p
standard deviation 1/ −q00 ( x ); sample x + accordingly.
Case 2: q0 ( x ) ≥ −ε and q0 ( x ) < 0
Approximate q(z) by a line passing through x and x1∗ , the largest mode of q(z) smaller
than x:
q( x1∗ ) − q( x )
q(z) ≈ q( x1∗ ) +
(z − x1∗ )
x1∗ − x
|
{z
}
‡
18
Sampling techniques similar to the Taylored chain are also discussed by Qi and Minka (2002).
By ensuring that | q00 ( x ) | > 0, using ε rather than 0 reduces the occurrence of proposed values
that depart too markedly from the current state.
19
25
A Bayesian Model for Sales Forecasting at Sun Microsystems
In this case, the component (‡) indicates an exponential distribution, and the proposal
is: 20
x + = x̂1 + w,
where w ∼ Exp(λ1 ),
q( x1∗ ) − q( x ) ,
λ1 = x1∗ − x x̂1 = x1∗ − 1/λ1
Case 3: q0 ( x ) ≥ −ε and q0 ( x ) > 0
Approximate q(z) by a line passing through x and x2∗ , the smallest mode of q(z) larger
than x. The proposal is developed in a manner parallel to that in Case 2:
x + = x̂2 − w,
where w ∼ Exp(λ2 ),
q( x2∗ ) − q( x ) ,
λ2 = x2∗ − x x̂2 = x2∗ − 1/λ2
Case 4: q0 ( x ) ≥ −ε and q0 ( x ) = 0
In this instance, x + is sampled from a uniform distribution over a range [ x1 , x2 ], such
that x1 < x < x2 . End points x1 and x2 are set to suitable modes of q(·), if they can be
found, and to user-supplied values otherwise.
Measures of Forecast Accuracy
As operationalizations of the quadratic loss criterion that pervades statistical inquiry, the
mean square error and its related metric, the root mean square error—we refer to both metrics
jointly as the (R)MSE—have long been a staple of academic research in forecasting—see
(Granger and Newbold 1973), for example. Unfortunately, though the (root) mean square
error is analytically attractive, researchers such as Armstrong and Collopy (1992) point to
a number of practical problems with its use: (1) Performance with respect to the (R)MSE
may be significantly affected by outliers. Such problems may be ameliorated by eliminating outliers, though this might draw the objectivity of the procedure into question.
(2) More seriously, the (R)MSE is inherently scale dependent, in that its magnitude depends not only on forecast accuracy, but also on the level of the underlying series; ceteris
paribus, a forecast 10% in excess of an actual value of 1,000,000 will result in a substantially
greater (R)MSE than one 10% above an actual of 1,000. This largely invalidates summary
The origin of the proposal, x̂1 , is offset from the mode x1∗ in order to guarantee irreducibility of
the resulting Markov chain; see (Geweke and Tanizaki 2003) for details.
20
26
A Bayesian Model for Sales Forecasting at Sun Microsystems
measures based on the (R)MSE of performance across a heterogeneous collection of series. 21 Since the series in this test described in the paper vary in maximum between 200
and 4000, the (R)MSE is unsuited to this application.
The mean absolute percentage error (MAPE) favored by practitioners, which expresses error
as a fraction of the associated actual value, avoids the scale-dependency of the (R)MSE.
The MAPE has disadvantages of its own, however: (1) In contradistinction to the (R)MSE,
summary MAPE measures may be skewed by small actuals—indeed, the MAPE is infinite for an actual value of 0. Some researchers, such as Coleman and Swanson (2007) have
suggested taking logarithms as a way of mitigating the problem, though this makes the resulting metric more difficult to interpret. (2) The MAPE exhibits a rather counter-intuitive
asymmetry; a forecast of 5 units on an actual of 10 produces an absolute percentage error
of 50%, whereas a forecast of 10 units on an actual of 5 gives an APE of 100%. Attempts
to amend the MAPE in order to overcome this problem (Makridakis 1993) have met with
limited success (Koehler 2001).
In light of these problems, some authors—including Fildes (1992) and Armstrong and
Collopy (1992)—have proposed the use of metrics based on relative absolute errors (RAEs).
These are absolute forecast errors divided by the corresponding error from a benchmark
method (normally the “naïve” or “random walk” method, which simply repeats the last
observation). Hyndman and Koehler (2006) point out, however, that RAE-based metrics
suffer from some of the same problems from which the (R)MSE and MAPE suffer: They
are sensitive to outliers, and may be skewed by small benchmark forecast errors (again,
the RAE is infinite if the forecast error is 0); remedies involving outlier elimination and
log-transformation are subject to the same criticisms, too.
Hyndman and Koehler (ibid.) propose the mean absolute scaled error (MASE) as a robust,
scale-independent metric that largely avoids the problems set out above. Formally, for
series y1 , . . . , y T , and denoting by ŷt+h|t the forecast for yt+h made in period t, the absolute
scaled error ASEth is defined:
ASEth =
|ŷt+h|t − yt+h |
1
T −h
−h
∑tT0 =
1 | y t0 + h − y t0 |
Then the mean absolute scaled error, MASEh at horizon h for the entire series is simply the
mean T −1 ∑tT=1 ASEth , and a summary metric for a collection of series may be calculated by
taking the mean of the ASEs across all the series. In recommending the MASE, Hyndman
21
Chatfield (1988), for instance, points out that Zellner’s (1986) analysis was unduly affected by
the outsized contribution to summary MSE metrics of only 5 out of 1,001 test series.
27
A Bayesian Model for Sales Forecasting at Sun Microsystems
and Koehler note that normalization with respect to the benchmark forecast error confers
scale-independence, while use of the in-sample aggregate in the denominator of scaled
errors makes the MASE metric more stable than those based on relative errors.
References
D. A. Ahlburg. Forecast evaluation and improvement using Theil’s decomposition. Journal
of Forecasting, 3:345–351, 1984.
J. Albert. Bayesian Computation with R. Springer, 2008.
J. Armstrong and F. Collopy. Error measures for generalizing about forecasting methods—
empirical comparisons. International Journal of Forecasting, 8:69–80, 1992.
J. M. Armstrong. Judgmental bootstrapping: Inferring experts’ rules for forecasting. In
J. M. Armstrong, editor, Principles of Forecasting: A Handbook for Researchers and Practitioners, pages 171–192. Kluwer, 2001a.
J. M. Armstrong. Combining forecasts. In J. M. Armstrong, editor, Principles of Forecasting:
A Handbook for Researchers and Practitioners, pages 417–439. Kluwer, 2001b.
J. M. Armstrong and F. Collopy. Integration of statistical methods and judgment for time
series forecasting: Principles from empirical research. In G. Wright and P. Goodwin,
editors, Forecasting with Judgment, chapter 10, pages 269–293. John Wiley & Sons, New
York, 1998.
A. H. Ashton, R. H. Ashton, and M. N. Davis. White-collar robotics: Levering managerial
decision making. California Management Review, 37:83–109, 1994.
Carliss Y. Baldwin and Kim B. Clark. Sun wars: Competition within a modular cluster,
1985-1990. In David B. Yoffie, editor, Competing in the Age of Digital Convergence, pages
123–158. Harvard Business School Press, 1997.
F. Bass. A new product growth model for consumer durables. Management Science, 15:
215–227, 1969.
R. Batchelor and T. Y. Kwan. Judgemental bootstrapping of technical traders in the bond
market. International Journal of Forecasting, 23:427–445, 2007.
J. M. Bates and C. W. J. Granger. The combination of forecasts. Operational Research Quarterly, 20:451–468, 1969.
J. M. Bernardo and A. F. M. Smith. Bayesian Theory. Wiley, 1994.
R. C. Blattberg and S. J. Hoch. Database models and managerial intuition: 50% model +
50% manager. Management Science, 36(8):887–899, August 1990.
F. Bolger and N. Harvey. Heuristics and biases in judgmental forecasting. In G. Wright
and P. Goodwin, editors, Forecasting with Judgment, chapter 5, pages 113–137. John Wiley
& Sons, New York, 1998.
J. Boulden. Fitting the sales forecast to your firm. Business Horizons, 1:65–72, 1958.
28
A Bayesian Model for Sales Forecasting at Sun Microsystems
G. E. P. Box, G. M. Jenkins, and G. C. Reinsel. Time Series Analysis. Prentice-Hall, 3rd
edition, 1994.
D. Bunn and G. Wright. Interaction of judgemental and statistical forecasting methods:
Issues & analysis. Management Science, 37(5):501–518, May 1991.
C. Carter and R. Kohn. On Gibbs sampling for state space models. Biometrika, 81:541–543,
1994.
C. Chatfield. Apples, oranges and mean square error. Journal of Forecasting, 4:515–518,
1988.
R. T. Clemen. Combining forecasts: A review and annotated bibliography. International
Journal of Forecasting, 5:559–583, 1989.
C. D. Coleman and D. A. Swanson. On MAPE-R as a measure of cross-sectional esimation
and forecast accuracy. Journal of Economic and Social Measurement, 32:219–233, 2007.
D. Dalrymple. Sales forecasting practices: Results from a United States survey. International Journal of Forecasting, 3:379–391, 1987.
Jessica Davis. Dell takes wraps off channel partner program, 2007. eWeek.com, December 5, http://www.channelinsider.com/article/Dell+Takes+Wraps+O+Channel+Partner+
Program/220950\_1.aspx.
R. M. Dawes. A case study of graduate admissions: Application of three principles of
human decision making. American Psychologist, 26:180–188, 1971.
Ton de Kok, Fred Janssen, Jan van Doremalen, Erirk van Wachem, Methieu Clerkx, and
Winfried Peters. Philips electronics synchronizes its supply chain to end the bullwhip
effect. Interfaces, 35(1):37–48, January–February 2005.
Jason Dean. The forbidden city of Terry Gou, 2007. The Wall Street Journal, August 11,
http://online.wsj.com/public/article/SB118677584137994489.html.
F. X. Diebold. Forecast combination and encompassing: Reconciling two divergent literatures. International Journal of Forecasting, 5:589–592, 1989.
F. X. Diebold and P. Pauly. The use of prior information in forecast combination. International Journal of Forecasting, 6:503–508, 1990.
A. Dolgui and M. Pashkevich. On the performance of binomial and beta-binomial models of demand forecasting for multiple slow-moving inventory items. Computers and
Operations Research, 35:893–905, 2008a.
A. Dolgui and M. Pashkevich. Demand forecasting for multiple slow-moving items with
short requests history and unequal demand variance. International Journal of Production
Economics, 112:885–894, 2008b.
J. Durbin and S. J. Koopman. Time Series Analysis by State Space Methods. Oxford University
Press, Oxford, 2001.
B. Efron. Bootstrap methods: Another look at the jackknife. The Annals of Statistics, 7:1–26,
1979.
29
A Bayesian Model for Sales Forecasting at Sun Microsystems
P. T. Elgers, H. L. May, and D. Murray. Note on adjustments ot analysts’ earning forecasts
based upon systematic cross-sectional components of prior-period errors. Management
Science, 41:1392–1396, 1995.
R. Fildes. Efficient use of information in the formation of subjective industry forecasts.
Journal of Forecasting, 10:597–617, 1991.
R. Fildes. The evaluation of extrapolative forecasting methods. International Journal of
Forecasting, 8:81–98, 1992.
R. Fildes and P. Goodwin. Good and bad judgement: Lessons from four companies. Foresight: The International Journal of Applied Forecasting, 8:5–10, 2007.
R. Fildes, P. Goodwin, M. Lawrence, and K. Nikolopoulos. Effective forecasting and judgmental adjustments: An empirical evaluation and strategies for improvement in supplychain planning. International Journal of Forecasting, 25:3–23, 2009.
P. Franses. Merging models and experts. International Journal of Forecasting, 24:31–33, 2008.
P. H. Franses and R. Legerstee. Properties of expert adjustments on model-based SKUlevel forecasts. International Journal of Forecasting, 25:35–47, 2009.
S. Frühwirth-Schnatter. Data augmentation and dynamic linear models. Journal of Time
Series Analysis, 15:183–102, 1994.
E. S. Gardner and E. McKenzie. Seasonal exponential smoothing with damped trends.
Management Science, 35(3):372–376, March 1989.
A. Gelman and J. Hill. Data Analysis Using Regression and Multilevel/Hierarchical Models.
Cambridge University Press, 2006.
A. Gelman, J. Carlin, H. Stern, and D. Rubin. Bayesian Data Analysis. Chapman &
Hall/CRC Press, Boca Raton, 2nd edition, 2003.
J. Geweke. Evaluating the accuracy of sampling-based approaches to calculating posterior
moments. In J. M. Bernardo, J. O. Berger, A. P. Dawid, and A. F. M. Smith, editors,
Bayesian Statistics, volume 4, pages 169–193. Clarendon Press, 1992.
J. Geweke and H. Tanizaki. Note on the sampling distribution for the Metropolis-Hastings
algorithm. Communications in Statistics – Theory and Methods, 32(4):775–789, 2003.
E. Ghysels, D. R. Osborn, and P. M. M. Rodrigues. Forecasting seasonal time series. In
G. Elliott, C. Granger, and A. Timmermann, editors, Handbook of Economic Forecasting,
volume 1, pages 660–706. North-Holland, 2006.
W. R. Gilks, S. Richardson, and D. J. Spiegelhalter. Markov Chain Monte Carlo in Practice.
Chapman & Hall, London, 1996.
P. Goodwin. Statistical correction of judgmental point forecasts and decisions. Omega:
International Journal of Management Science, 24:551–559, 1996.
P. Goodwin. Adjusting judgemental extrapolations using Theil’s method and discounted
weighted regression. Journal of Forecasting, 16:37–46, 1997.
P. Goodwin. Correct or combine? Mechanically integrating judgmental forecasts with
statistical methods. International Journal of Forecasting, 16:261–275, 2000.
30
A Bayesian Model for Sales Forecasting at Sun Microsystems
C. W. J. Granger and P. Newbold. Some comments on the evaluation of economic forecasts.
Applied Economics, 5:35–47, 1973.
C. W. J. Granger and R. Ramanathan. Improved methods of combining forecasts. Journal
of Forecasting, 3:197–204, 1984.
K. R. Hammond, C. J. Hursch, and F. J. Todd. Analyzing the components of clinical inference. Psychological Review, 71:255–262, 1964.
A. C. Harvey. Forecasting, Structural Time Series Models and the Kalman Filter. Cambridge
Univ. Press, Cambridge, 1989.
Joseph Heller. Catch-22. Simon & Schuster, New York, 1961.
R. Hill. Applying Bayesian methodology with a uniform prior to the single period inventory model. European Journal of Operational Research, 98:555–562, 1997.
P. J. Hoffman. The paramorphic representation of clinical judgment. Psychological Bulletin,
57:116–131, 1960.
C. Huang and G. Tzeng. Multiple generation product life cycle predictions using a novel
two-stage fuzzy piecewise regression analysis method. Tehnological Forecasting and Social
Change, 75:12–31, 2008.
H. D. Hughes. An interesting seed corn experiment. The Iowa Agriculturist, 17:424–425,
428, 1917.
C. J. Hursch, K. R. Hammond, and J. L. Hursch. Some methodological considerations in
mutliple-probability studies. Psychological Review, 71:42–60, 1964.
R. J. Hyndman and A. B. Koehler. Another look at measures of forecast accuracy. International Journal of Forecasting, 22:679–688, 2006.
R. J. Hyndman, A. B. Koehler, J. K. Ord, and R. D. Snyder. Forecasting with Exponential
Smoothing: The State Space Approach. Springer, 2008.
K. Kahn. How to measure the impact of a forecast error on an enterprise? Journal of
Business Forecasting, pages 21–25, 2003. Spring.
R. Klassen and B. Flores. Forecasting practices of Canadian firms: Survey results and
comparisons. International Journal of Production Economics, 70:163–174, 2001.
A. B. Koehler. The asymmetry of the sAPE and other comments on the M3-competition.
International Journal of Forecasting, 17:570–574, 2001.
L. Lamport. LATEX: A Document Preparation System. Addison-Wesley Professional, 2nd
edition, 1994.
M. Lawrence and M. O’Connor. Judgement or models: The importance of task differences.
Omega: International Journal of Management Science, 24:245–254, 1996.
M. Lawrence, P. Goodwin, M. O’Connor, and D. Önkal. Judgmental forecasting: A review
of progress over the last 25 years. International Journal of Forecasting, 22:493–518, 2006.
M. J. Lawrence, M. O’Connor, and R. H. Edmundson. A field study of sales forecasting
accuracy and processes. European Journal of Operational Research, 122(1):151–160, April
2000.
31
A Bayesian Model for Sales Forecasting at Sun Microsystems
C.-Y. Lee, J.-D. Lee, and Y. Kim. Demand forecasting for new technology with a short
history in a competitive market: The case of the home networking market in South
Korea. Technological Forecasting and Social Change, 75:91–106, 2008.
W. Y. Lee, P. Goodwin, R. Fildes, K. Nikolopoulos, and M. Lawrence. Providing support
for the use of analogies in demand forecasting tasks. International Journal of Forecasting,
23:377–390, 2007.
P. Lenk and A. Rao. New models from old: Forecasting product adoption by hierarchical
Bayes procedures. Marketing Science, 9:42–57, 1990.
D. V. Lindley. Reconciliation of probability distributions. Operations Research, 31:866–880,
1983.
V. Mahajan, E. Muller, and Y. Wind, editors. New-Product Diffusion Models. Kluwer, 2000a.
V. Mahajan, E. Muller, and Y. Wind. New-product diffusion models: From theory to practice. In New-Product Diffusion Models Mahajan et al. (2000a), chapter 1.
S. Makridakis. Accuracy measures: Theoretical and practical concerns. International Journal
of Forecasting, 9:527–529, 1993.
W. H. McGlothlin. Stability of choices among uncertain alternatives. American Journal of
Psychology, 69:604–615, 1956.
P. E. Meehl. When shall we use our heads instead of the formula? Jounal of Counseling
Psychology, 4:268–273, 1957.
J. T. Mentzer. The impact of forecasting improvement on return on shareholder value.
Journal of Business Forecasting, pages 8–12, 1999. Fall.
J. T. Mentzer and C. C. Bienstock. Sales Forecasting Management: Understanding the Techniques, Systems and Management of the Sales Forecasting Process. Sage Publications, Thousand Oaks, CA, 1998.
W. W. Moe and P. S. Fader. Using advance purchase orders to forecast new product sales.
Marketing Science, 21(3):347–364, Summer 2002.
A. Montgomery. Creating micro-marketing pricing strategies using supermarket scanner
data. Marketing Science, 16:315–337, 1997.
M. A. Moon, J. T. Mentzer, and C. D. Smith. Conducting a sales forecasting audit. International Journal of Forecasting, 19(1):5–25, 2003. January–March.
M. Moriarty. Design features of forecasting systems involving management judgments.
Journal of Marketing Research, 22:353–364, 1985.
P. A. Morris. Decision analysis expert use. Management Science, 20:1233–1241, 1974.
P. A. Morris. An axiomatic approach to expert resolution. Management Science, 29:24–32,
1983.
R. Neelamegham and P. Chintagunta. A Bayesian model to forecast new product performance in domestic and international markets. Marketing Science, 18(2):115–136, 1999.
R. Neelamegham and P. Chintagunta. Modeling and forecasting the sales of technology
products. Quantitative Marketing and Economics, 2:195–232, 2004.
32
A Bayesian Model for Sales Forecasting at Sun Microsystems
S.-C. Niu. A piecewise-diffusion model of new-product demands. Operations Research, 54:
678–695, 2006.
M. O’Connor, W. Remus, and K. Lim. Improving judgmental forecasts with judgmental
bootstrapping and task feedback support. Journal of Behavioral Decision Making, 18:247–
260, 2005.
L. D. Pankoff and H. V. Roberts. Bayesian synthesis of clinical and statistical prediction.
Psychological Bulletin, 80:762–773, 1968.
A. Pole, M. West, and J. Harrison. Applied Bayesian Forecasting and Time Series Analysis.
Chapman Hall, 1994.
Y. Qi and T. P. Minka. Hessian-based Markov chain Monte Carlo algorithms. In First Cape
Cod Workshop on Monte Carlo Methods, 2002.
T. Åstebro and S. Elhedhli. The effectiveness of simple decision heuristics: Forecasting
commercial success for early-stage ventures. Management Science, 52:396–409, 2006.
E. M. Rogers. Diffusion of Innovations. Free Press, 5th edition, 2003.
N. R. Sanders. Comments on “Effective forecasting and judgmental adjustments: An empirical evaluation and strategies for improvement in supply-chain planning”. International Journal of Forecasting, 25:24–26, 2009.
N. R. Sanders and K. B. Manrodt. Forecasting practices in US corporations: Survey results.
Interfaces, 24:92–100, 1994.
N. R. Sanders and K. B. Manrodt. Forecasting software in practice: Use, satisfaction, and
performance. Interfaces, 33:90–93, 2003.
N. R. Sanders and L. P. Ritzman. On knowing when to switch from quantitative to judgmental forecasts. International Journal of Operations and Production Management, 11:28–37,
1991.
N. R. Sanders and L. P. Ritzman. Judgmental adjustment of statistical forecasts. In J. M.
Armstrong, editor, Principles of Forecasting: A Handbook for Researchers and Practitioners,
pages 405–416. Kluwer, 2001.
Steven M. Schafer. Web Standards Programmer’s Reference: HTML, CSS, JavaScript, Perl,
Python, and PHP. Wrox, Chicago, IL, 2005.
E. Silver. Bayesian determination of the reorder point of a slow moving item. Operations
Research, 13:989–997, 1965.
SMCI. Sun microsystems company info, 2009. http://www.sun.com/aboutsun/company/
index.jsp.
A. Sood, G. James, and G. Tellis. Functional regression: A new model for predicting market
penetration of new products. Marketing Sciences, 28:36–51, 2009.
H. Theil. Applied Economic Forecasting. North-Holland, 1971.
A. Timmermann. Forecast combinations. In G. Elliott, C. Granger, and A. Timmermann,
editors, Handbook of Economic Forecasting, chapter 4, pages 135–194. North-Holland,
2006.
33
A Bayesian Model for Sales Forecasting at Sun Microsystems
L. R. Tucker. A suggested alternative formulation of the developments by Hursch, Hammond and Hursch and by Hammond, Hursch and Todd. Psychological Review, 71:528–
530, 1964.
J.W. Tukey. Exploratory data analysis. Addison-Wesley, 1977.
A. Tversky and D. Kahneman. Judgment under uncertainty: Heuristics and biases. Science,
185:1124–1131, 1974.
T. T. Tyebjee. Behavioral biases in new product forecasting. International Journal of Forecasting, 3:393–404, 1987.
H. J. van Heerde, C. F. Mela, and P. Manchanda. The dynamic effect of innovation on
market structure. Journal of Marketing Research, 41:166–183, 2004.
W. N. Venables and D. M. Smith. An Introduction to R. Network Theory, 2002.
R. Webby and M. O’Connor. Judgmental and statistical time series forecasting: A review
of the literature. International Journal of Forecasting, 12:91–118, 1996.
M. West and P. J. Harrison. Bayesian Forecasting and Dynamic Models. Springer-Verlag, New
York, 2nd edition, 1997.
Rick Whiting. Sun takes build-to-order to the next level, 2006. InformationWeek, September 12, http://www.informationweek.com/story/showArticle.jhtml?articleID=192700331.
J. Wiedermann. Web Design Flash Sites. Tachen, 2006.
H. E. Williams and D. Lane. Web Database Applications with PHP and MySQL. O’Reilly
Media, 2nd edition, 2004.
J. Wisner and L. Stanley. Forecasting practices of JIT and non-JIT purchasers. European
Journal of Purchasing and Supply Management, 1:219–225, 1994.
B. Worthen. Future results not guaranteed. CIO, 16:1–4, 2003. URL http://www.cio.com.
au/article/168757/future_results_guaranteed.
G. Wright and P. Ayton. The psychology of forecasting. Futures, 18:420–439, 1986.
P. M. Yelland. A model of the product lifecycle for sales forecasting. Technical Report 127,
Sun Microsystems Laboratories, 2004. URL http://research.sun.com/techrep/2004/smli_
tr-2004-127.pdf .
P. M. Yelland and E. Lee. Forecasting product sales with dynamic linear mixture models.
Technical Report TR-2003-122, Sun Microsystems Laboratories, March 2003.
A. Zellner. A tale of forecasting 1001 series: The Bayesian knight strikes again. International
Journal of Forecasting, 2:491–494, 1986.
Paul H. Zipkin. Foundations of Inventory Management. McGraw-Hill/Irwin, New York,
2000.
34