Measuring price change for consumer
electronics using scanner data
Crown copyright ©
This work is licensed under the Creative Commons Attribution 3.0 New Zealand licence.
You are free to copy, distribute, and adapt the work, as long as you attribute the work to
Statistics NZ and abide by the other licence terms. Please note you may not use any
departmental or governmental emblem, logo, or coat of arms in any way that infringes any
provision of the Flags, Emblems, and Names Protection Act 1981. Use the wording
‘Statistics New Zealand’ in your attribution, not the Statistics NZ logo.
Liability
While all care and diligence has been used in processing, analysing, and extracting data
and information in this publication, Statistics New Zealand gives no warranty it is error free
and will not be liable for any loss or damage suffered by the use directly, or indirectly, of the
information in this publication.
Citation
Statistics New Zealand (2014). Measuring price change for consumer electronics using
scanner data. Available from www.stats.govt.nz.
ISBN 978-0-478-42940-4 (online)
Published in November 2014 by
Statistics New Zealand
Tatauranga Aotearoa
Wellington, New Zealand
Contact
Statistics New Zealand Information Centre: [email protected]
Phone toll-free 0508 525 525
Phone international +64 4 931 4600
www.stats.govt.nz
Contents
List of tables and figures ................................................................................................... 4
1 Using scanner data in the CPI ...................................................................................... 5
Consumer electronics categories that use scanner data ................................................ 5
Contribution to the CPI .................................................................................................... 5
2 Scanner data from market research company GfK ................................................... 6
3 Benefits of using scanner data for consumer electronics........................................ 8
Scanner data allows more accurate price measurement ................................................ 8
Scanner data allows us to re-use existing data............................................................... 8
Scanner data accurately reflects seasonality in quantities ............................................. 8
Scanner data reflects product substitution .................................................................... 10
4 New methods to create price indexes ....................................................................... 11
The need for new methods ............................................................................................ 11
Rolling year GEKS (RYGEKS) index ............................................................................ 11
Imputation Törnqvist rolling year GEKS (ITRYGEKS) index......................................... 12
5 Implementing the ITRYGEKS index in production................................................... 14
Deriving the quarterly index from quarterly average prices and expenditure shares ... 14
Using two months of the quarter for production ............................................................ 14
Integration with other changes arising from the 2014 CPI review ................................ 15
References ........................................................................................................................ 16
3
List of tables and figures
List of tables
1. Number of characteristics in each category (August 2014) ........................................ 6
2. Number of distinct products in each category (August 2014) ..................................... 7
List of figures
1. Seasonality of quantity, expenditure, and average price – digital cameras................ 9
2. Quality-adjustment of the price index – digital cameras ........................................... 10
3. Data incorporated into each quarterly index at time of production ........................... 15
4
1 Using scanner data in the CPI
Measuring price change for consumer electronics using scanner data describes the
methodology we introduced for creating price indexes from consumer electronics scanner
data.
From the September 2014 quarter, we incorporated retail transaction data, or ‘scanner
data’, into the consumers price index (CPI), to measure price change for consumer
electronics categories.
Consumer electronics categories that use scanner data
We use scanner data for 12 consumer electronics categories in the CPI:
 heat pumps
 desktop computers
 laptop computers
 tablet computers
 multi-function devices
 cellphone handsets
 digital cameras
 digital camera memory cards
 television sets
 set-top boxes for television sets
 DVD, Blu-ray players and player/recorders
 home theatre and stereo systems.
Contribution to the CPI
In terms of expenditure weighting:
 heat pumps contribute a fifth of the ‘major household appliances’ class, which
contributes 0.71 percent to the all groups CPI expenditure weight for the June 2014
quarter
 cellphone handsets contribute over 90 percent of the ‘telecommunication
equipment’ class, which contributes 0.29 percent to the all groups CPI
 the remaining 10 categories contribute four-fifths of the ‘audio-visual and
computing equipment’ subgroup, which contributes 1.16 percent to the all groups
CPI.
5
2 Scanner data from market research company GfK
The market research company GfK supplies us with the scanner data. GfK adds
information about product features, or characteristics, to the data collected from retailers.
Total monthly sales values and quantities sold are available, across retailers, for each
product, along with extensive information about the characteristics of each product. The
scanner data represents hundreds of thousands of transactions each month.
The number of characteristics in the data range from 10 for digital camera memory cards
to 77 for digital cameras, as shown in Table 1 for August 2014, the latest month data is
available.
The coverage of the data is between 80 and 95 percent of transactions for most of the
categories, with lower coverage for some categories, such as heat pumps.
Table 1
Number of characteristics in each category (August 2014)
1. Number of characteristics in each category (August 2014)
Category
Heat pumps
Characteristics
27
Desktop computers
56
Laptop computers
71
Tablet computers
73
Multi-function devices
53
Cellphone handsets
59
Digital cameras
77
Digital camera memory cards
10
Television sets
62
Set-top boxes for television sets
49
DVD, Blu-ray players, and player/recorders
50
Home theatre and stereo systems
62
6
Measuring price change for consumer electronics using scanner data
The number of distinct products represented in the data for each category, in August
2014, is shown in Table 2.
Table 2
Number of distinct products in each category (August 2014)
2. Number of distinct products in each category (August 2014)
Category
Heat pumps
Products
72
Desktop computers
107
Laptop computers
445
Tablet computers
148
Multi-function devices
102
Cellphone handsets
392
Digital cameras
228
Digital camera memory cards
254
Television sets
325
Set-top boxes for television sets
24
DVD, Blu-ray players, and player/recorders
129
Home theatre and stereo systems
224
7
3 Benefits of using scanner data for consumer
electronics
We have moved to using scanner data to measure price change for consumer electronics
because it:




allows more accurate price measurement
allows us to re-use existing data
accurately reflects seasonality in quantities
reflects product substitution.
Scanner data allows more accurate price
measurement
Until now, we have relied on sampling consumer electronics prices across several
dimensions – categories, products, outlets, and time. For each consumer electronics
category, a product was priced at each of about 60 appliance retailers and department
stores. When a product was no longer available, we replaced it with a similar product,
based on discussion with retailers about market share and features.
We based quantities, or expenditure shares, on information we acquired during the
Household Economic Survey reference period, and updated every three years.
In contrast, scanner data has the potential to give a more complete picture of both prices
and quantities sold at any point in time. With information on the characteristics of each
product, we are also able to use statistical methods to explicitly quality-adjust the price
indexes.
Information on region of sale is not available in the data, so we will use national
movements in each of the regions.
Scanner data allows us to re-use existing data
Market research companies already collate consumer electronics scanner data for
businesses, so it is good practice to re-use the data to generate official statistics. This
reduces fieldwork, and the respondent load associated with collection, which involves
observing products and prices in stores, and discussing product changes with store staff.
Scanner data accurately reflects seasonality in
quantities
Scanner data for consumer electronics has monthly information on both prices and
quantities. Quantities can be seasonal, as shown in Figure 1 for digital cameras.
Correspondingly, total expenditure on digital cameras is also highly seasonal. Note that
average price, unadjusted for quality change, also shows seasonality, corresponding to
cheaper cameras being bought around the Christmas period.
8
Measuring price change for consumer electronics using scanner data
Figure 1
1. Seasonality of quantity, expenditure, and average price – digital cameras
Seasonality of quantity, expenditure and average price
- digital cameras
July 2008 to June 2011
Base: July 2008 month (=1)
Index
3.5
3.0
2.5
2.0
1.5
1.0
0.5
0.0
J A S O N D J F M A M J J A S O N D J F M A M J J A S O N D J F M A M J
08
09
10
11
Total expenditure
Average price
Quantity sold
Note: This graph is based on earlier research data (Statistics New Zealand, 2012). Note that each of the three
series is expressed as an index and based to 1 in July 2008, for confidentiality reasons.
Source: Statistics New Zealand
The current fixed-basket approach to price measurement, when applied to seasonal
prices and quantities, has the potential to over- or under-state actual price movements
when we combine seasonal prices with quantities that were fixed at an average annual
level.
Figure 2 shows that when we appropriately incorporate the seasonal prices and
quantities of digital cameras using the imputation Törnqvist rolling year GEKS
(ITRYGEKS) index (discussed in Chapter 4), the resulting quality-adjusted price
movement is no longer seasonal.
9
Measuring price change for consumer electronics using scanner data
Figure 2
2. Quality-adjustment of the price index – digital cameras
Quality adjustment of the price index - digital cameras
July 2008 to June 2011
Base: July 2008 month (=1)
1.2
Index
1.0
0.8
0.6
0.4
0.2
0.0
J A S O N D J F M A M J J A S O N D J F M A M J J A S O N D J F M A M J
08
09
10
11
ITRYGEKS index
Average price
Note: It is possible, and valid, for quality-adjusted price indexes for some goods (eg fresh fruit and
vegetables) to still display seasonality even after appropriate incorporation of seasonal quantities.
Source: Statistics New Zealand
Scanner data reflects product substitution
Because scanner data provides prices and quantities for the most detailed level of
product specification (ie the barcode level), we can use it to incorporate new products in
the index at the time they are introduced, and to reflect their relative importance based on
actual quantities sold.
We can also use scanner data to empirically test substitution effects across different
product categories, to infer the appropriate level to fix expenditure shares between CPI
basket and weight reviews.
10
4 New methods to create price indexes
To produce price indexes from scanner data, we are using the imputation Törnqvist
rolling year GEKS (ITRYGEKS) index. The ITRYGEKS is an extension of the rolling year
GEKS (RYGEKS) index.
This chapter provides more detail about the:



need for new methods
rolling year GEKS (RYGEKS) index
imputation Törnqvist rolling year GEKS (ITRYGEKS) index.
The need for new methods
Consumer electronics is a rapidly changing product class, so it is particularly important
that we use methods that will appropriately adjust for the change in quality of the products
purchased.
In addition, because consumer electronics products can have short life cycles, it is
important to introduce new products into the index in such a way that the implicit price
movement associated with their introduction is appropriately reflected in the index. That
is, if a new product has a low introductory price, relative to its set of features, then the
price index should reflect this price decrease.
In addition to this requirement for appropriate quality adjustment, the CPI is nonrevisable. This places another constraint on methods for creating price indexes from
scanner data using statistical models, where the models are generally based on a past
window of data – usually a year – ending with the most recent quarter.
The two key reasons why traditional index number methods do not work well in the case
of scanner data are:
 the high level of ‘churn’ – products appearing and disappearing from the market
 volatile prices and quantities due to discounting, which leads to a bias called ‘chain
drift’ when chained superlative indexes such as a chained Törnqvist are used to
continually update the basket in the presence of high churn.
Over the past five years, we have collaborated on research to determine an index method
that is appropriate for producing non-revisable, quality-adjusted price indexes from
scanner data.
Rolling year GEKS (RYGEKS) index
Ivancic, Diewert, and Fox (2011) proposed a method for producing price indexes from
scanner data that uses all the prices and quantities in the data, and is free of chain drift.
Called the rolling year GEKS (RYGEKS) index, it is based on the Gini, Eltetö and Köves,
and Szulc (GEKS) index used for multilateral spatial price indexes such as purchasing
price parities – which compare prices in different countries at a point in time.
Within a window of time (usually just over one year – ie five quarters for a quarterly index,
or 13 months for a monthly index), the RYGEKS index between periods t1 and t2 is the
geometric mean of all the superlative bilateral indexes (such as the Törnqvist index or, as
used in Ivancic et al (2011), the Fisher index) between:
 t1 and all the other periods in the window, and
 t2 and all the other periods in the window.
The monthly RYGEKS, based on a 13-month rolling estimation window, is as follows:
11
Measuring price change for consumer electronics using scanner data
For the first window (ie T=0 to 12), the RYGEKS index is equal to the GEKS index:
0T
RYGEKS
P
P
0T
GEKS

12
 P P
0t
tT

1
13
(equation 1)
t 0
Where P ij is any superlative index (eg a Törnqvist index) between periods i and j.
From t=13 onwards, RYGEKS links on the most recent movement from the GEKS
calculated on the next window (ie from t=1 to 13, then from t= 2 to 14, and so on) as
follows:

13
0,13
0,12
PRYGEKS
 PGEKS
 P12,t  Pt ,13

1
13
t 1

14
0,14
0,13
PRYGEKS
 PRYGEKS
 P13,t  P t ,14

12
  P 0t  Pt ,12
t 0

1
13
  P
13
1
13
12, t
 Pt ,13

1
13
t 1
(equation 2)
t 2
and so on.
However, a limitation of the RYGEKS method is that it does not reflect the implicit price
movements of new or disappearing products entering or leaving the market.
For example, if the initial price of the latest model of a cellphone is high relative to its set
of features, then this implicit price increase is not reflected in the RYGEKS index.
Imputation Törnqvist rolling year GEKS (ITRYGEKS)
index
Jan de Haan, of Statistics Netherlands, proposed an extension of the RYGEKS index,
called the imputation Törnqvist rolling year GEKS (ITRYGEKS). A paper on the method
by de Haan and Statistics NZ senior researcher Frances Krsinich was recently published
in the Journal of Business and Economic Statistics (de Haan and Krsinich, 2014).
The ITRYGEKS method is an extension of the GEKS method. Like the GEKS, the
ITRYGEKS can utilise all the information in the data, while remaining free of chain-drift. In
addition, it reflects the implicit price movements of new and disappearing products by
imputing price movements based on statistical modelling of the relationship between
price and product features.
Unlike the RYGEKS method described above, which is based on superlative indexes
ij
such as the Törnqvist or Fisher (ie the P in equations (1) and (2), the ITRYGEKS index
is based on ‘bilateral time-dummy hedonic’ indexes. The ‘Törnqvist’ in its name refers to
the fact that it is algebraically equivalent to a Törnqvist index based on real and predicted
prices, as shown in equation (4).
A bilateral time-dummy hedonic index between any periods 0 and t is derived from a
statistical regression model based on the data from periods 0 and t.1
The estimating equation for the bilateral time-dummy hedonic regression model is:
K
ln pit     t Dit    k z ik   it
(equation 3)
k 1
1
This can be generalised to any two periods i and j.
12
Measuring price change for consumer electronics using scanner data
Where:
ln pit is the log of the average monthly price for product i in period t.
 is the intercept term.
Dit has the value 1 if the observation relates to period t (t ≠ 0) and the value 0 if the
observation relates to period 0.
z it is the quantity of the kth characteristic, or feature, for product i.2 By definition, this will
be the same in both periods.3
 it is an error term with an expected value of 0.
Since the model described by equation (3) controls for changes in the product
characteristics, exp  is a measure of quality-adjusted price change between periods 0
and t (de Haan and Krsinich, 2014).
t
In addition, when the average4 expenditure shares are used as weights for the matched
products, and half of the expenditure shares for the unmatched products (in the periods
they are available), the ITRYGEKS index from period 0 to t can be expressed as follows:
0t
PITRYGEKS

p 


iU 0 t 

  p
t
i
0
i
si0  sit
2
 pˆ 


iU D0 t 

  p
t
i
0
i
si0
2
p 


iU N0 t 

  pˆ
t
i
0
i
sit
2
(equation 4)
Where:
U 0t is the set of matched products i with respect to periods 0 and t.
U D0t is the set of 'disappearing' products i with respect to periods 0 and t – that is,
products that exist in period 0 but not in period t.
U N0t is the set of 'new' products i with respect to periods 0 and t.
s it
is the expenditure share of product i in period t.
Equation (4) demonstrates that the ITRYGEKS index is equivalent to a Törnqvist index
for the matched products, and for new and disappearing products, the ITRYGEKS index
applies a Törnqvist formula to prices predicted from time-dummy hedonic models for the
period in which there is no price available. The derivation of (4) is given in de Haan and
Krsinich (2014).
2
For categorical characteristics, such as those on the consumer electronics scanner data, each
characteristic k will be represented by a set of dummy variables corresponding to all possible values of
characteristic k – that is, variables which are set to 1 (or 0) in the presence (or absence) of that value of
the characteristic.
3
A product corresponds to a distinct set of characteristics, or features. Therefore, any change in
characteristic would result in a different product.
4
Across both time periods.
13
5 Implementing the ITRYGEKS index in production
This chapter summarises key decisions we made before using the ITRYGEKS index to
produce price indexes for consumer electronics categories in the CPI:



deriving the quarterly index from quarterly average prices and expenditure shares
using two months of the quarter for production
integration with other changes arising from the 2014 CPI review
Deriving the quarterly index from quarterly average
prices and expenditure shares
We receive monthly scanner data from GfK, but the New Zealand CPI is a quarterly
index. We can either derive the quarterly index from a monthly index, or pre-aggregate
the data to a quarterly level before deriving a quarterly index.
While it is useful to produce a monthly index as part of the monitoring and analysis
process, it is conceptually more appropriate to derive the quarterly index from quarterly
average prices and expenditure shares. This ensures the prices for products sold in each
month of the quarter are appropriately weighted for price deflation, to produce quarterly
volume indexes in the National Accounts.
Using two months of the quarter for production
Consumer electronics data for just the first two months of the quarter are available in time
to incorporate into the CPI.
There are four options for how to deal with this limitation in production. Using back-data,
we assessed each of these options against the benchmark of the index we could
calculate if all three months of the quarter were available.
The four options were to:
1. base the published index for the most recent quarter on the first two months of the
quarter, with complete back-data feeding into the estimation (ie the third month of
the quarter will be incorporated into the five-quarter estimation window for the
following quarter’s index calculation)
2. base the published index on three months of data, lagged by one month
3. base the published index on only the middle month of each quarter
4. base the published index on only the first two months of each quarter.
Option 1 performed best of all four options, and sits very close to the benchmark index.
Therefore, we derived the ITRYGEKS index for the latest quarter from the first two
months of that latest quarter, with the third month of the quarter then being incorporated
into the five-quarter estimation window used to calculate the following quarter's CPI, as
shown in Figure 3.
14
Measuring price change for consumer electronics using scanner data
Figure 3
Data incorporated into each quarterly index at time of production
3. Data incorporated into each quarterly index at time of production
2013Q3
2013Q4
2014Q1
2014Q2
2014Q3
2014Q4
2015Q1
2014Q3
2014Q4
2015Q1
Figure 3 shows that, for example, to calculate the quarterly price movement for the third
quarter of 2014 (ie the top row), we use full quarterly data for each of the quarters from
quarter three of 2013 through to quarter two of 2014, and the first two months of data for
the third quarter of 2014.
Integration with other changes arising from the 2014
CPI review
We implemented the ITRYGEKS index in production along with the CPI basket and
weight changes arising from the 2014 CPI Review. The basket has been realigned to
reflect the consumer electronics categories that we are measuring using scanner data.
The price movement used in the CPI for the June to September 2014 quarters is a likefor-like movement based on the scanner data, linked in on the new price reference period
– the June 2014 quarter.
Initially, we fixed the expenditure shares at the item level of the CPI basket for consumer
electronics categories. However, we will review this decision in the future to determine
whether we will use information in the scanner data to reflect substitution across
consumer electronics categories, between the three-yearly CPI basket and weight
reviews.
15
References
de Haan, J, & Krsinich, F (2014). Scanner data and the treatment of quality change in
nonrevisable price indexes. Journal of Business and Economic Statistics, 32(3). Available
from www.tandfonline.com
Ivancic, L, Diewert, WE, & Fox, KJ (2011). Scanner data, time aggregation and the
construction of price indexes. Journal of Econometrics, 161(1). Available from
www.sciencedirect.com
Statistics New Zealand (2012). A fresh look at patterns in gadget sales. Economic News,
April 2012. Available from www.stats.govt.nz
16