For Official Use
STD/NA(2001)19
Organisation de Coopération et de Développement Economiques
Organisation for Economic Co-operation and Development
21-Sep-2001
___________________________________________________________________________________________
_____________
English - Or. English
STATISTICS DIRECTORATE
STD/NA(2001)19
For Official Use
National Accounts
QUALITY MEASURES FOR AUSTRALIAN SYSTEMS OF ECONOMIC ACCOUNTS
Agenda item 7
Australian Bureau of Statistics - Australia
OECD MEETING OF NATIONAL ACCOUNTS EXPERTS
Château de la Muette, Paris
9-12 October 2001
Beginning at 9:30 a.m. on the first day
English - Or. English
JT00113078
Document complet disponible sur OLIS dans son format d'origine
Complete document available on OLIS in its original format
STD/NA(2001)19
QUALITY MEASURES FOR AUSTRALIAN SYSTEMS OF ECONOMIC ACCOUNTS
Introduction
1. The Australian Bureau of Statistics (ABS) is an active participant in discussions regarding
international standards for assessing the quality of economic statistics, such as the IMF Data Quality
Assessment Framework (DQAF). DQAF and related standards are applicable to quality assessments
of statistics compiled by a wide variety of national statistical agencies; and they provide some support
to cross-country quality comparisons of macroeconomic statistics1. The ABS feels the need, however,
for a more detailed array of measures tailored to assessing the ways in which the quality of economic
accounts in a statistically advanced country evolves over time. During the past year or so, the ABS has
been experimenting with such measures, under the project title Quality Measures for Systems of
Economic Accounts (QMSEA). A broad draft framework has been laid out; the ABS is now trialling it
on some subsystems of Australian economic data.
Statistical Quality - Current and Possible Future Measures
The ABS already shows a variety of quality indicators. They range from:
1.
•
the descriptions of scope, coverage, collection methods and quality indicators (such as relative
standard errors on levels and movements) found in all ABS publications that present the results of
sample surveys to
•
manuals such as Australian National Accounts: Concepts, Sources and Methods(ABS Cat. no.
5216.0) which includes descriptions of the concepts, data sources and methods used, assumptions
made, and qualitative assessments of major national accounts aggregates.
But the developers of DQAF advise caution regarding cross-country comparisons: " In providing
feedback on the usefulness of the IMF's Reports on the Observance of Standards and Codes, some
users, particularly those in the financial markets, have called for an assessment system that would
permit a country ranking or scoring system for data quality. However, the data quality assessment
framework does not lend itself to such an approach. The element of subjectivity inherent in the
frameworks, the detail embedded in the dataset specific frameworks, and the great diversity of country
circumstances largely preclude using them to make meaningful country rankings." Carsons, C. S.
(IMF paper, Dec. 20, 2000), "Towards a Framework for Assessing Data Quality", para. 37 p. 15.
2
STD/NA(2001)19
In addition, quantitative measures of accuracy (viz. comparisons between the published chain
Laspeyres volume measures and comparable chain Fisher measures) and revisability of national
accounts statistics are published from time to time. In part, QMSEA is intended as a portal into those
documents; it is also a platform for further selective extension of the available quality measures. Key
features of QMSEA include the following:
•
It is designed to display relevant information about aspects of quality so that the user can assess the
degree to which systems of economic accounts are likely to be "fit for purpose".
•
It is designed primarily for use within the ABS and by a relatively small, sophisticated group of
external users (such as the key economic policy agencies and academic researchers).
•
It concords with and expands upon the measures in the DQAF.
Dimensions of the QMSEA Framework
2. Twelve "facets" of quality. QMSEA identifies twelve aspects of quality for systems of economic
accounts:
•
•
•
•
•
•
•
•
•
•
•
•
accessibility
accuracy
compliance
consistency
continuity
frequency
longevity
lucidity
relevance
revisability
sense
timeliness.
Attachment 1 contains descriptions of these twelve facets.
3. Three phases of compilation. These facets of quality are appropriate, in varying degrees, to assessing
statistical quality during three phases of compiling economic accounts:
•
Gathering of the input data.
•
Transformations of the input data, such as seasonal adjustment, construction of price and volume
indexes, modelling of multifactor productivity, construction of supply-use tables and so on.
•
Assembly of the key national accounting aggregates.
QMSEA aims to support quality assessments for input data, intermediate transformed data and national
accounting aggregates.
4. Six streams of input data. QMSEA looks beyond sample survey data (for which a variety of quality
measures are already available) to the full range of input data (such as purposive surveys and
administrative by-product datasets) used to compile the accounts. The input data streams include:
•
•
•
•
probability-based sample surveys;
non-probability-based surveys;
censuses or complete enumerations;
administrative or transactional by-product collections;
3
STD/NA(2001)19
•
•
marketplace and media intelligence; and
the synthesis or "modelling" of data from the other streams.
5. Data sources other than the ABS may raise particular challenges for assessment of quality. This arises
in large part because quality indicators for these sets of input data are rarely available from their
sources. ABS experience has shown that with administration by-product collections, for instance,
quality issues regularly arose that needed to be managed carefully.
7. Beyond sampling errors. In developing a comprehensive suite of quality measures for a system of
economic accounts, it is recognised that "sample survey standard error" is associated with only one of
the input data streams to systems of economic accounts, namely the probability sample survey based
results. Non-probability based or purposive surveys such as used for the Consumer Price Indexes (CPI)
or Producer Price Indexes (PPI) do not have this measure of quality. Nor do the other input data
streams. Moreover, the role of sampling standard error is a relatively minor one compared with that of
non-sample errors.
Future Activities
8. The ABS is undertaking two activities to pursue the QMSEA strategy.
9. Agreeing on quality criteria. The proposed suite of indicators is being discussed with key users inside
and outside the ABS and with colleagues in other statistical agencies. The aim is to agree upon a suite
of quality indicators that are comprehensible and comprehensive for the task of assessing quality in the
many diverse uses to which systems of economic accounts are applied.
10. Testing QMSEA on subsystems of the economic accounts. Before attempting to implement QMSEA on
the gamut of economic accounts, the ABS is undertaking some small-to-medium scale "proof of
concept" studies. Possible testbeds for the concepts include a satellite account (such as the recently
released tourism account), another topic-specific slice of the core accounts (such as the subsystem used
to estimate capital services or multifactor productivity, or an analytical array (such as the array of
household wealth estimates, dissected by household type).
QMSEA - A Real or Virtual Array of Quality Measures?
11. In concept, QMSEA consists of a huge multidimensional array of quality measures (all twelve facets of
quality x all economic accounting variables, at all levels of disaggregation x all time periods). In
practice, it would be very difficult and prohibitively expensive to build a fully populated array. So
QMSEA is best regarded as a virtual array of quality measures, or a portal into the quality information
stored in the ABS databanks. Important features of this virtual array include the following:
•
Some metadata concerning facets or dimensions of quality will be common across all or most of a
system of accounts. This quality metadata is referred to as global. The definitions, descriptions
and relationships of facets of quality are examples of global metadata. Similarly, some of the
quality metadata will be common to data items within a particular account, but may differ between
accounts. This type of quality metadata will be referred to as account-specific — for example,
metadata regarding quality aspects such as accessibility and timeliness. On the other hand some
quality metadata will be data-item-specific.
4
STD/NA(2001)19
•
Some quality indicators will be numerical (say, the average revision to such-and-such an aggregate
during such-and-such a time span); others will be qualitative (say, a judgment-based grading of the
quality of administrative by-product data) and other textual (say, descriptions of the modelling
procedures or of quality assurance procedures).
12. The ABS is considering several means of delivering the quality information assembled under the
QMSEA framework. Within the ABS's Lotus Notes environment, it will be possible to provide users
with a roadmap of (and links to) the quality statements already residing in text databases and other
repositories; this is likely to be our first trial implementation, as it should be relatively inexpensive to
establish. In the longer run, we may consider developing a browser-style interface that permits ABS
and external users to enter queries regarding the quality of the system of accounts and particular
aggregates, then to navigate to the relevant global, account-specific and data-item-specific information.
Before the ABS undertook such an investment, however, we must assess what range and depth of
quality information would be valuable to (and digestible by) different classes of users.
For More Information
13. More details of this work can be obtained from [email protected] or [email protected]
Issues for Discussion
1) Do other countries think it valuable to develop a more detailed array of measures tailored to
assessing the way the quality of economic accounts in statistically advanced countries evolves over
time? Have other countries developed something of this kind?
2) Are the proposed twelve facets of quality proposed appropriate for this purpose?
3) How do other countries store their arrays of quality measures, and how are they made available to
internal and external users?
5
STD/NA(2001)19
Attachment 1
Twelve Dimensions of Quality Proposed in Quality Measures for Systems of Economic Accounts
Quality Dimension
Working Definition
Accessibility
Expresses the degree to which the statistical product is available
to users. Embraces such aspects as - data presentation; what
cells may have been suppressed (or collapsed) to preserve
confidentiality; availability of metadata relating to concepts,
sources and methods; distribution medium; etc.
Accuracy
Expresses the proximity of an estimate to the "true" value.
Embraces such aspects as - errors in the source data; errors in
timing; errors in valuation; methodological errors.
Compliance
Expresses the degree to which the statistical product conforms
with the relevant conceptual and accounting standards.
Consistency
Expresses the degree to which the framework, concepts and
classification boundaries are complete and internally consistent.
Continuity
Expresses the degree to which the statistical product supports
meaningful interpretation from time period to time period.
Frequency
Expresses the time resolution of the statistical product (eg,
monthly, quarterly, annual, longer-than-annual).
Longevity
Expresses the time span of the statistical product.
Lucidity
Expresses the clarity of the documentation accompanying the
statistical product (both the concepts, sources and methods
documentation and the interpretations of main features).
Relevance
Expresses the ability of the statistical product to meet the
contemporary (and likely future) needs of policy makers,
business, and other decision makers and analysts.
Revisability
Expresses the degree to which the estimates in a statistical
product are revised from one vintage to another.
Sense
Expresses the degree to which the (social or economic) "story"
told by the statistical product is consistent with the stories told
by other measures (both ABS and nonABS).
Timeliness
Expresses the time lag between the reference period of the
statistical product and its release to the user community.
6