Regional benchmarks for the
Household Economic Survey
Crown copyright ©
This work is licensed under the Creative Commons Attribution 3.0 New Zealand licence.
You are free to copy, distribute, and adapt the work, as long as you attribute the work to
Statistics NZ and abide by the other licence terms. Please note you may not use any
departmental or governmental emblem, logo, or coat of arms in any way that infringes any
provision of the Flags, Emblems, and Names Protection Act 1981. Use the wording
'Statistics New Zealand' in your attribution, not the Statistics NZ logo.
Liability
While all care and diligence has been used in processing, analysing, and extracting data
and information in this publication, Statistics New Zealand gives no warranty it is error free
and will not be liable for any loss or damage suffered by the use directly, or indirectly, of the
information in this publication.
Citation
Statistics New Zealand (2012). Regional benchmarks for the Household Economic Survey.
Available from www.stats.govt.nz.
ISBN 978-0-478-40801-0 (online)
Published in November 2012 by
Statistics New Zealand
Tatauranga Aotearoa
Wellington, New Zealand
Contact
Statistics New Zealand Information Centre: [email protected]
Phone toll-free 0508 525 525
Phone international +64 4 931 4610
www.stats.govt.nz
Contents
List of table
es and figu
ures..................................................................................................... 4
Purpose an
nd summarry ........................................................................................................ 5
About the Household
d Economicc Survey ........................................................................... 5
Clarifying the
t problem
m ......................................................................................................... 5
Benchmark
king’s role in the HES ......................................................................................... 7
Food exp
penditure movements u
using old and
d new bench
hmarks....................................... 7
Constructin
ng the new benchmark
ks ................................................................................... 8
Example
e of construc
cting a benc hmark ............................................................................. 8
Applying
g the new be
enchmarks .......................................................................................... 8
Comparing the G-fac
ctors for old and new be
enchmarks ................................................... 8
Ensuring th
he new ben
nchmarks a re fit for the
e intended purpose ................................. 10
Results of
o analysing the benchm
marks ............................................................................. 11
Summary............................................................................................................................. 12
3
List of tables and figures
List of tables
1 Household counts in HES at 30 June 2007 and 2010, Wellington and nationally ...... 6
2 Household counts in HES 2006/07 and 2009/10, for selected regions ...................... 6
3 Average weekly expenditure on food, by region, for old and new benchmarks .......... 7
List of figures
1 G-factor plots for HES benchmarks 2010/11, new and old benchmarks .................... 9
2 Original and new household weights for 2009/10 HES ............................................ 10
3 Original and new household weights for 2010/11 HES ............................................ 10
4 Average weekly mortgage for Wellington region, 2006/07–2010/11 ........................ 11
5 Percentage change in average weekly mortgage in Wellington region, 2006/07–
2010/11 ....................................................................................................................... 11
4
Purpose and summary
This report explains how previous benchmarks used in the Household Economic Survey
(HES) led to inconsistent reported estimates, and how new benchmarks have corrected
these estimates.
For HES 2009/10, household counts for Wellington region were not as expected. While
we report food estimates at a national level in our information releases, the unexpected
household counts led to unusual movements for the food estimates when we analysed
them by region. Our new benchmarks control the number of households at the regional
level and correct the problem with the estimates.
In this report we describe the problem then explain how the new benchmarks are
constructed. Next we compare the original, or ‘old’, benchmarks with the new
benchmarks, and follow this with analysis that shows the new benchmarks are fit for the
intended purpose.
About the Household Economic Survey
HES is an annual survey, run by Statistics New Zealand, that is used to provide a picture
of household income and expenditure. We collect different information in different years.
In the years ending June 2007 (HES 2006/07) and 2010 (HES 2009/10), we collected
information on all household expenditure costs and household incomes. In the years
between, HES (Income) collected information on income and minimal information on
expenditure on essential housing costs.
Clarifying the problem
In HES 2009/10 the weighted household counts for Wellington region decreased 4.5
percent (see table 1). The national total increased 3.5 percent, and we expected a similar
increase in Wellington.
The actual household counts in table 1 are derived from population estimates and are the
best estimates for regions. For Wellington, the actual count increased 3.3 percent. The
difference between the actual count and the weighted count indicated there was
something amiss with the benchmarks for HES.
The weighted household counts for the other four HES regions showed overestimates for
the Auckland and Canterbury household counts, compared with actual changes (see
table 2). The change in the weighted estimate for Auckland (9.6 percent) was nearly twice
the actual household count movement (5.3 percent). The weighted count for Canterbury
was also exaggerated – 8.4 percent compared with an actual 5.8 percent increase in
households
At a national level the weighted household count was the same as the actual count.
Therefore the overestimated increases in the Auckland and Canterbury regions seemed
to be the cause of the decrease in the Wellington weighted household count. They also
caused the underestimated change in the household counts for the Rest of North Island
and Rest of South Island regions.
The weighted household counts are produced using the original benchmarks from the
same years. The actual counts are equivalent to the household counts produced by the
new benchmarks, because households are benchmarked at the regional level for the new
benchmarks.
5
Table 1
1 Household counts in HES at 30 June 2007 and 2010, Wellington and nationally
Household counts in HES at 30 June 2007 and 2010
Wellington and nationally
2007
2010
Movement
estimate (%)
Weighted count
184,450
176,100
- 4.5
Actual count
179,170
185,100
3.3
Weighted count
1,569,215
1,623,355
3.5
Actual count
1,650,800
1,710,400
3.6
Wellington
National
Table 2
2 Household counts in HES 2006/07 and 2009/10, for selected regions
Household counts in HES 2006/07 and 2009/10
For selected regions
2007
2010
Movement
estimate (%)
Weighted count
468,200
512,970
9.6
Actual count
475,900
501,199
5.3
Weighted count
525,580
526,440
2.0
Actual count
511,915
527,355
3.0
2,083,110
2,257,720
8.4
210,280
222,470
5.8
Weighted count
182,670
182,120
-0.3
Actual count
181,870
187,230
2.9
Auckland
Rest of North
Island
Canterbury
Weighted count
Actual count
Rest of South
Island
6
Benchmarking’s role in the HES
Benchmarking is the final stage of the weighting process. For calibration we adjust the
final weights to meet a separate set of population estimates called ‘benchmarks’. More
specifically HES uses integrated weighting, which is a unique type of benchmarking. For
integrated weighting all people in the same household have the same weight, ensuring
we can compare household and person estimates. Benchmarking improves the quality of
survey estimates by reducing the effect of sampling and non-sampling errors.
The original household benchmark gives the number of two-person and other-person
households at a national level. However, as the regional-level household counts were not
original benchmarks, the weights did not necessarily meet expectations in all areas – due
to the sample’s composition. This affected household counts for Wellington in 2009/10.
Food expenditure movements using old and new
benchmarks
Our work demonstrated the impact of the regional discrepancies by looking at food
expenditure movements from 2006/07 to 2009/10 (see table 3). The estimate for the
Wellington region, based on the original benchmarks, increased 16.5 percent between
the survey periods. This was higher than expected. We didn’t see similar movements in
the other HES regions and, nationally, the average movement in food expenditure was
9.2 percent.
When we kept the number of households for Wellington region in 2009/10 at the 2006/07
level, the food expenditure estimate increased 11.3 percent. Using the new benchmark
the estimate increases 10.8 percent, which brings it more in line with the other regions.
Table 3
3 Average weekly expenditure on food, by region, for old and new benchmarks
Average weekly expenditure on food
By region
For old and new benchmarks
Mean expenditure – old
benchmarks
$
Region
Mean expenditure – new
benchmarks
%change
$
%change
2007
2010
2007 to
2010
2007
2010
2007 to
2010
Auckland
184.2
194.5
5.6
182.6
195.5
7.1
Wellington
178.8
208.3
16.5
181.2
200.8
10.8
Rest of North
Island
140.8
157.0
11.5
143.9
157.6
9.5
Canterbury
160.0
169.2
5.7
157
171.6
9.3
Rest of South
Island
158.5
171
7.9
161.2
168.6
4.6
Total New
Zealand
162.8
177.7
9.2
163.8
177.4
8.3
7
We have used the new benchmarks to recalibrate the regional estimates for all years
from 2006/07 onwards, and will use them for future information releases.
Constructing the new benchmarks
We created the new benchmarks in three steps.
1. We used 2006 Census ratios of adults per household to convert current ‘adult by
region’ counts into ‘household by region’ counts.
2. We used 2006 Census ratios of the number of two-adult households and otheradult households to convert the ‘household by region’ counts into two-adult and
other-adult household counts by region.
3. We did minor scaling to ensure the two-adult and other-adult household counts
met the national totals.
The 2006 Census ratios were stable when compared with those from 2001. This ensured
we could use the 2006 ratios for producing these benchmarks.
Example of constructing a benchmark
The following example traces the steps for benchmarking Auckland region (numbers
rounded for confidentiality).
1. Auckland had 1,132,000 adults in the HES 2011 year. There were 2.30 adults per
household, leading to 493,000 households.
2. Of these households, 47 percent were two-adult households – making 231,700
two-adult households and 261,300 other-adult households.
3. To meet the national totals these counts needed to be scaled by 3.4 percent and
2.3 percent, respectively. This gives the final benchmarks of 239,465 two-adult
households and 267,285 other-adult households for Auckland region.
Applying the new benchmarks
When the new benchmarks are applied to HES sample weights, it is important that they
do not change the weights more than the original benchmarks did. We measure this
change by the ‘G-factor’, using the following equation:
G-factor = Calibrated final weight
Input weight
Note: The G-factor is a measure of stress put on the household weights when they are
calibrated from their non-response weight to fit the benchmarks. All households in the
HES are given an initial sample weight, which is then adjusted for non-response. The
initial sample weight of each household is further adjusted when the sample is
benchmarked to fit the benchmarks.
Comparing the G-factors for old and new benchmarks
Ideally, the G-factor should not change the weight by more than a 100 percent increase,
or a 50 percent decrease. However, the change for some subgroups does exceed these
limits – in order to meet population totals. In these instances the outlier G-factors should
be minimal.
8
For this analysis, we compared the G-factors between the old and new benchmarks for:
the overall sample, for people by region, and for the number of households nationally.
The purpose of comparing the G-factor plots between the different benchmarks is to
determine the effect the new benchmarks are having on the stress of the weights during
benchmarking. For the comparison, 2009/10 and 2010/11 were analysed, which means
both the full HES and HES (Income) are covered.
Figure 1 shows the overall G-factor boxplots for the old benchmarks, and the new
benchmarks, for HES 2010/11.
Figure 1
1 G-factor plots for HES 2010/11, new and old benchmarks
G-factor plots for HES 2010/11
New and old benchmarks
For both boxplots in figure 1, the mean G-factor values are close to 1 and have similar
dispersion. This is typical of the other benchmarks examined. The new benchmarks
tended to have slightly bigger mean G-factor values, although this is not evident in figure
1. Comparing all G-factor plots for the new and old benchmarks, for both 2009/10 and
2010/11, we see that changing to the new benchmarks does not significantly add stress
on the weights during the benchmarking process.
Scatterplots (see figures 2 and 3) of the household weights derived from the old and new
benchmarks show the movement of the weights with the change in the benchmarks.
Points on the imaginary 45 degree line represent no movement in the household weights
from introducing the new benchmarks. By comparing the scatterplots for the new and old
benchmarks, for 2009/10 and 2010/11, we see that changing the old benchmarks to the
new benchmarks has not significantly changed most household weights. Most points are
on the imaginary 45 degree line.
9
Figure 2
2 Original and new household weights for 2009/10 HES
Figure 3
3 Original and new household weights for 2010/11 HES
Ensuring the new benchmarks are fit for the intended
purpose
Before we implemented the new benchmarks we analysed them to ensure they fitted the
intended purpose. This analysis is important as any significant changes in the estimates
caused by the change in benchmarks (and therefore the weights) could cause significant
changes in the time series, and therefore a break in the series.
We needed to know if changing the benchmarks caused significant movements for the
key estimates. We assessed rent and mortgage estimates for different geographical
regions in HES, for 2006/07 to 2010/11, as these are key outputs reported each year.
10
Results of analysing the benchmarks
Figure 4 shows the average weekly mortgage in the Wellington region, and figure 5
shows the percentage change in the average weekly mortgage in the region. Note that
figure 4 shows standard errors, and figure 5 shows relative standard errors. We analysed
national figures and all five HES regions: Auckland, Wellington, Canterbury, Rest of
South Island, Rest of North Island. These are the regions used in the information
releases.
The standard error measure, expressed as a number, indicates the extent to which a
survey estimate is likely to deviate from the true population. The relative standard error,
expressed as a fraction, is the standard error of the estimate. If the new estimates are
within the error bars of the old estimates then the new estimates are not significantly
different from the original ones. Therefore, changing the benchmarks would not
significantly affect the estimates.
Figure 4
4 Average weekly mortgage for Wellington region, 2006/07–2010/11
Figure 5
5 Percentage change in average weekly mortgage in Wellington region, 2006/07–2010/11
11
In both figures 4 and 5 we see the estimates from the new weights fit within the error bars
of the estimates generated by the old weights. The same general result appeared for all
the regions we examined (for details, email [email protected]). All regional changes in
the means and medians that resulted from changing the benchmark were well within the
sample error.
This shows that the new ‘household count by region’ benchmarks do not affect the time
series and are able to be used. The main benefit from the new benchmarks is to enable
better regional analysis for other estimates.
Summary
Investigating the significance of the effect of the new benchmarks on key estimates
shows that all changes in the mean and median for the five regional estimates were well
within the sample error. The new benchmarks did not significantly change the key
estimates. However, they did bring into line other estimates that were affected by
incorrect household counts, such as the estimates for expenditure on food for Wellington.
We have applied the new benchmarks to all HES survey data since 2006/07. This means
the entire series of HES data from 2006/07 has been revised and the new household
benchmark figures are now used for all analysis
12