Data Quality Assurance: Quality Gates Framework for
Statistical Risk Management
Narrisa Gilbert
Australian Bureau of Statistics, 45 Benjamin Way, Belconnen, ACT, Australia 2615
Abstract
Statistical collections are often exposed to the risk that the components of the process fail to meet
the quality standard expected, such that the quality of the statistical outputs are affected. We refer
to this kind of risk as "statistical risk". Statistical risk arises for various reasons, some of which
may include inadequate inputs, processes not being well defined, changes to existing processes, or
human error.
This paper introduces an approach to managing the quality of statistical processes called quality
gates. Quality gates are a risk mitigation tool designed to improve the early detection of errors or
flaws in statistical processes. They improve a user's ability to manage statistical risk by providing
explicit evidence relating to the statistical process, improving knowledge management and
information sharing.
This paper will provide an explanation of the concepts underpinning quality gates so that they can
be applied for statistical risk management. A discussion of some of the challenges that the
Australian Bureau of Statistics has experienced in implementing quality gates will also be
provided.
Key Words: quality gates, statistical risk, statistical quality management, quality management,
quality assurance
1. Introduction
As the world of data continues to expand at an increasingly rapid pace, so too does the need of a
National Statistics Office to provide relevant, timely, accurate, coherent, easy to understand and
accessible data. National Statistics Offices are faced with the challenges of responding to
increasing requests for data, reduced resources, the need for improved efficiency and higher staff
turnover more than ever before. Those challenges coupled with increased scrutiny of data post
global financial crisis and increasing organisational fiscal restraints, mean that new improved
methods need to be created to produce statistics at a lower cost whilst maintaining or improving
quality. All of these stress factors increase the chance of things going wrong in the statistical
process which can impact on an organisation's credibility.
The Australian Bureau of Statistics is Australia's official statistical organisation. To be trusted to
assist with informed decision making in the community, the Australian Bureau of Statistics uses
quality assurance as a fundamental principle around statistical processes. Yet often the approach to
quality assurance has been left to the discretion of each statistical area of the organisation to
undertake. This has led to variations in how the statistical outputs are quality assured across
collections.
Due to the increasing scrutiny and expectations by external clients and a need to provide
leadership to other providers of statistics in Australia, the Australian Bureau of Statistics has been
promoting the use of a standard and systematic framework for quality assuring data. This
framework helps to maintain the credibility and relevance of statistical outputs. The method
primarily focusses on reducing known statistical risk, where statistical risk refers to the risk that
the components of the statistical process fail to meet the quality standards expected, such that the
quality of the statistical outputs are affected.
The standard procedure that the Australian Bureau of Statistics promotes for use by its statistical
subject matter areas is an initiative called quality gates. This initiative is similar to process control
which is used in the manufacturing industry. The Australian Bureau of Statistics has adapted this
concept and applied it to a statistical process cycle. As a result, quality gates are defined by six
components: Placement; Quality Measures; Roles, Tolerance; Actions and Evaluation.
A summary of the components of quality gates is below. More detailed information on the
components, application and an example of quality gates can be found in the information paper
"Quality Management of Statistical Risk Using Quality Gates, Dec 2010, (cat no. 1540.0)".
2. Six Components of Quality Gates
2.1 Placement
The key driver behind implementing quality gates into a statistical process cycle is to enable errors
to be detected earlier in the statistical process. This allows sufficient time to correct for the issues
that the problem presents. To do this quality gates need to be 'placed' at key strategic points
throughout the statistical process in order to be most effective at helping to identify errors sooner
rather than later. To assist with placement of quality gates it is useful to consider: what can go
wrong?; when can it occur?; and what impact can it have? Examining the risk profile of the
collection is a good place to start when considering the placement of quality gates throughout a
statistical process.
Alternatively, placement could be considered as the occurrence of natural beginnings and endings
throughout the statistical process. For example, before data is collected a sample needs to be
selected. Hence, there is a natural quality gate occurring between these two events to ensure that
the statistical process up to that point (before collection), is fit for purpose not only at that point
but for the entire statistical process outcomes.
Drawing a simple business process map of the statistical process cycle is recommended as a useful
way to help identify the placement of quality gates. It is important however to not create too many
quality gates as this will detract from their effectiveness.
2.2 Quality Measures
The effectiveness of quality gates is highly dependent on the use of appropriate quality measures,
which help to identify if there are issues with the statistical process or data. Quality measures can
be a summary of a group of similar quality assurance tasks (e.g. Validation). Or they may be
distinct detailed items considered to be important for determining the quality of the statistical
process (e.g. movements in data items of interest).
Quality measures occur throughout the process leading up to quality gates. Issues identified by the
quality measure are dealt with at the time that they are discovered. Quality gates provide a
consolidation of the quality measures that have occurred up to that point to allow a big picture
assessment of the quality of the process and data to be made.
2.3 Roles
With any process it is important to ensure that roles are clearly defined. Quality gates include roles
relating to sign-off, compilation, and stakeholders (providing information, receiving information).
Quality gates may involve people from different areas of the organisation, hence it is important to
make sure that the role and responsibility of each party to the quality gate is well defined.
2.4 Tolerance
Having an expectation about an acceptable level of quality at a given point in time is an important
component of a quality gate. Putting in place a pre-determined tolerance assists with making a
decision as to whether the statistical process and data are fit for purpose. A tolerance can be
qualitative (yes or no) or quantitative (97%).
2.5 Action
If the expectations (tolerance) set for a quality gate at a given point in time are not met, having
actions clearly stated as to what needs to be done to fix the issue is an important way to ensure
effective quality management of the statistical process. The idea of a traffic light is often used to
describe this component where green is 'go'; amber is to 'proceed with caution whilst
investigating'; and red is 'stop and remedy the situation before continuing with the process'.
2.6 Evaluation
Systematically reviewing quality gates in a statistical process cycle to identify improvements is an
important aspect of quality gates. It ensures that the specifications of the quality gate are fit for
purpose because there is an implicit assessment of each quality gate against the seven dimensions
of the Australian Bureau of Statistics Data Quality Framework. For example during the evaluation
stage the assessment may reveal that the tolerance was too strict and needed to be relaxed for a
particular quality measure; additional quality measures are required for a particular quality gate;
and some quality measures need to move into different quality gates, etcetera.
2. History of Development and Implementation
The quality gates initiative was developed by the Australian Bureau of Statistics in the second half
of 2005. The motivation for the development of quality gates was to minimise the risk of the
reoccurrence of a significant statistical quality incident. It impacted on the interpretation of the
status of the economic and the social conditions in Australia and on the organisation's reputation.
This led to an independent review of the processes involved in the quality assurance of the data
that was released. The review identified ways in which improvements could be made to that
process and applied more generally to all statistical processes across the Australian Bureau of
Statistics.
There were a number of recommendations for improvements to quality assurance and processes
from that independent review of the statistical process in question. These included that "an
organisation's strategy throughout development and operation should be to minimise the chance of
an error occurring and maximise the chance of its detection" and "Such an (quality risk) analysis
should identify areas of greatest vulnerability or greatest impact from an error, and checks decided
and implemented to quality assure processes from development, through to implementation and
then ongoing operation..."(Cornish 2005).
The introduction of quality gates to the Australian Bureau of Statistics began with an internal
methodological paper that was endorsed by senior managers to be promoted across the statistical
subject matter areas of the organisation. The paper also included templates to guide areas in their
implementation of quality gates into their statistical processes.
The methodology division supplied assistance to those statistical subject matter areas that were
keen to adopt and utilise the quality gates method. However, early adoption and implementation of
quality gates by statistical subject matter areas was slow in the first few years after their launch to
the organisation. As a result of the limited uptake of quality gates the methodology division
developed a set of three half day training courses to help promote data quality assurance which
were heavily promoted, especially to new staff in the organisation. Of the three, one focussed on
the concept of quality gates and was implemented in the middle of 2009. The quality gates training
course provided specific information to help people better understand the concept of quality gates
and facilitated the implementation of them within statistical subject matter areas.
A few statistical subject matter areas who were successful early implementers of quality gates in
2008-09 were enlisted to help build the profile and promote the benefit of quality gates in their
work processes to other statistical subject matter areas. This was achieved through presentations to
other similar statistical subject matter areas (either economic or social statistics) in forums
designed for information sharing across statistical subject matter areas. These presentations
allowed individual statistical subject matter area experiences in the implementation of quality
gates to be shared in a way that allowed robust discussions to take place in order to assist other
statistical subject matter areas with their understanding and perception of quality gates.
In December 2010, an information paper was published on the Australian Bureau of Statistics
website called "Quality Management of Statistical Processes Using Quality Gates, Dec 2010, (cat
no. 1540.0)". This paper was designed to provide step by step guidance on how to implement
quality gates in a statistical process. It was primarily aimed at organisations involved in the
collection, process, analysis or dissemination of statistics. However, many statistical subject
matter areas of the Australian Bureau of Statistics found the paper to be useful in assisting them
with their work. This information paper helped to consolidate the various tasks that had been
undertaken to promote the use of quality gates across the organisation.
After the reissue of the September quarter 2009 National Accounts estimates, quality gate
workshops for economic areas were instigated at the Australian Bureau of Statistics. This error
raised concerns from key users of economic statistics as to the quality and coherence of the
economic statistical data produce by the Australian Bureau of Statistics. As a result of these
concerns the economic areas of the Australian Bureau of Statistics have placed more importance
on implementing quality gates to reduce the risk of errors and consequential lack of coherence
occurring in future releases of economic data. A co-ordination area within the economic group
became the Project Owner for co-ordinating, managing and monitoring the introduction and
development of quality gates in each economic area, this gained momentum in 2011.
The project owner area launched a campaign to promote quality gates and make economic
statistical subject matter areas aware that there would be a requirement for all of them to
participate. Campaigning included a seminar, articles in an internal electronic economic area
specific newsletter ("The Hub"), promotion of the concept of quality gates and the reporting
requirements by economic statistical subject matter areas, as well as a contact for more
information. Posters were also displayed across the largest Australian Bureau of Statistics office
(where the majority of economic areas reside), to lift the profile of quality gates. These posters
used comical situations to promote the use of quality gates. They were displayed near the lifts and
stairwells throughout the office to capture all employees' attention as they went about their daily
work.
An example of a poster used to promote awareness and buy-in from economic statistical subject
matter areas (and others) across the Australian Bureau of Statistics is below:
The project owner area assisted the co-ordination between the methodology division and the
economic statistical subject matter areas to find suitable times to conduct workshops. These
workshops focussed on the statistical business process of a particular statistical subject matter area
and discussed the potential placement of quality gates and some of the components that would fit
within the quality gates.
The onus was still on the statistical subject matter areas to continue to develop and finalise quality
gates outside of the workshops in order to encourage ownership of the development of quality
gates for their statistical processes. The project owner area specified up front in the workshops the
reporting requirements required from the economic statistical subject matter areas. The deadlines
of six and twelve months for quality gate development and completion were set from the date of
the workshop for each economic statistical subject matter area.
The project owner area was responsible for reporting back to senior management on the progress
of each area in their implementation of quality gates post workshop. Leniency and flexibility
around the deliverable of quality gates to the project owner area was provided to some statistical
subject matter areas which were already extremely busy implementing new systems,
methodologies or processes. This flexibility has helped in the implementation of quality gates
across economic statistical subject matter areas, as the workshops and expectations from the
workshops took into account their individual needs.
3. Lessons Learnt and Challenges
Quality gates are a useful quality assurance tool that can provide a lot of benefit in the long term.
However, there are barriers to implementation despite the expectation that the benefits will
outweigh the costs. Some of the challenges that the Australian Bureau of Statistics has faced in the
implementation of quality gates include, but are not limited to: time pressure; misunderstanding of
concepts; system issues; and competing priorities.
This section discusses some of these challenges and how they have been addressed to try and
overcome them.
3.1 Time Pressures
Over many years the limited time that a statistical subject matter area has to process information
for release has been a barrier to the uptake of quality gates. This includes the notion of areas
already trying to adapt their processes to become more efficient as part of an ongoing
organisational need to find efficiencies in all aspects of work. This combination of stresses in itself
is a large risk to the organisation and its production of statistics that are fit for purpose. Whilst
statistical subject matter areas acknowledge the benefit of quality gates and their ability to help
manage the quality of the processes during this environmental shift in the organisation, few were
able to commit to implementing quality gates into their practices.
Senior management outside the methodology division of the Australian Bureau of Statistics started
to actively engage in the promotion and use of quality gates across collections. This eventuated
from the few high profile errors mentioned earlier, making it into the public domain and near
misses occurring more than is liked in the pre-release of data. Time was set aside for quality gate
workshops (a minimum of two hours), for the statistical subject matter areas to discuss the
statistical risks and quality assurance of collections. This included discussion around how quality
gates and quality measures could be implemented to help monitor the process more effectively.
Time was also set aside for members of the statistical subject matter area to further develop the
quality gates outside of the initial workshop.
3.2 Common Misunderstandings
Quality gates, at a broad level, are an easy to understand concept. However, when applied in
practice, issues in peoples’ understanding of the concepts may become apparent. This was the case
with the implementation of quality gates.
Experience across the Australian Bureau of Statistics in the implementation of quality gates
highlighted gaps in the explanation of some of the components underlying quality gates. This
included confusion with the components of Placement, Quality Measures and Roles.
3.2.2 Placement vs. Quality Measures
Extra effort explaining the difference between Placement and Quality Measures for a quality gate
occurred. Statistical subject matter areas had difficulty in distinguishing between the two and as a
result had a quality gate placed wherever a major quality measure needed to be examined. This led
to an excessive number of quality gates being created.
To help clarify this issue, further effort was put into explaining the difference between the two
components and basic examples provided in training and workshops to emphasise this difference.
Work was done to highlight that a quality gate consists of multiple quality measures and that
quality measures occur throughout the process (and problems fixed if issues identified) before a
quality gate is reached. The placement component refers to the point at which an overall
assessment of the quality measures (collectively) is made.
3.2.3 Roles
A quality gate can have many stakeholders involved in creation and sign-off. Not all of these
stakeholders are from the same statistical subject matter area in the Australian Bureau of Statistics.
This caused confusion for some areas in the administration of quality gates, whereby duplicate
quality gates were created for different areas to populate rather than consolidating the reporting to
one gate.
The methodology division provided advice to help statistical subject matter areas better understand
the physical monitoring and use of quality gates. Advice on contacting a stakeholder in another
statistical subject matter area to obtain specific information pertaining to quality measures within a
quality gate was empahsised to owner areas.
3.2.4 Use of Existing Quality Assurance Procedures
Another issue with misunderstanding the concept of quality gates has been that some statistical
subject matter areas have regarded quality gates as an entirely new way to undertake quality
assurance for their processes.
Quality gates are a systematic way of managing statistical risk. The majority of the content of a
quality gate is often already in use by the statistical subject matter area as part of their existing
quality assurance processes.
Misunderstanding has occurred where areas have thought they needed to develop all new quality
measures to populate quality gates. This has led to the protest about not having enough time to
implement quality gates as they are perceived to be entirely new work.
This is not the case for quite a large proportion of the content. Often what has been found during
workshops is that there is a distinct lack of documentation around processes and the workshop has
highlighted this as one of the main deficiencies in current practices.
3.3 System Issues
There have been several issues raised over the past few years with the implementation of quality
gates in regards to system issues. This has varied from an assumption that quality gates needed to
have a specific system built in order to use them; to existing systems not being flexible enough to
assist in the implementation of quality gates as much as statistical subject matter areas would like.
Statistical subject matter areas initially assumed quality gates were an information technology fix
to quality assurance. This is not the case. Whilst quality gates can be automated to a degree, for
instance quality measures built into systems so that the results are automatically generated for
monitoring purposes, one of the myths that the methodology division has tried to debunk is that
quality gates require a new system to be built. This myth debunking helped overcome some initial
hesitancy in the uptake of quality gates.
Another challenge that presents itself in the current climate of system change at the Australian
Bureau of Statistics is that a few statistical subject matter areas are waiting to undergo system
redevelopment. As a result they are understandably not enthusiastic about putting a lot of effort
into developing quality gates into their current processes when their systems may change.
However, one thing that quality gates highlight which is immaterial of system issues, is the need
for better documentation of all quality assurance procedures for a statistical collection.
3.4 Competing Priorities
The implementation of quality gates at the Australian Bureau of Statistics varies across statistical
collections due to competing priorities of everyday business requirements. Some statistical subject
matter areas have implemented quality gates, other areas have not due to business as usual
demands and general collection updates (e.g. new classification implementation). These
constraints have meant that for the economic statistics implementation of quality gates, reporting
of progress and setting of target deadlines for implementation is tailored to each statistical subject
matter area need.
4. Conclusion
The success of the implementation of quality gates across the organisation will be measured in
several ways. One is to compare performance indicators relating to the number of quality incidents
that are identified before and after the dissemination of statistical data. This will be used to
monitor the effectiveness of implemented quality gates, both within and across statistical subject
matter areas, to address issues highlighted.
Success will also be measured by the progress with which statistical subject matter areas develop
and implement quality gates into their existing quality assurance processes. In the economic
statistics division a dedicated project owner will be reporting back to senior management on the
progress of each economic statistical subject matter area in implementing quality gates into their
work.
Future work in the implementation of quality gates at the Australian Bureau of Statistics will see
them being embedded into a new information management transformation system that is currently
being developed.
This work will require that all statistical subject matter areas have well defined quality measures
and quality gates in order to monitor their statistical processes effectively under the new system.
Due to this organisational change it is imperative that statistical subject matter areas across the
Australian Bureau of Statistics identify and implement quality gates and more importantly identify
the corresponding quality measures required for their statistical processes before they are required
to move to the new system. This will enable statistical subject matter areas to take advantage of
the re-engineering opportunity that this new information management transformation system will
afford. It also has the effect of improving knowledge management of statistical processes by
promoting an understanding of the data and the processes in order to specify the quality measures
required.
Quality gates require effort to implement successfully into an organisation because of the
challenges involved. However, given all the challenges that the Australian Bureau of Statistics has
noted with their implementation, it is not impossible to overcome these with persistence. This
persistence is aided by the collective desire of an organisation to succeed and improve on the
current situation.
Acknowledgements
The quality gates framework described in this paper is based on an internal ABS paper prepared
by Paul Schubert (2007). Many staff of the ABS, both past and present, contributed to the
implementation of quality gates. Special thanks to Jen Dunn, Mike Booth, Bill Allen, Paul
Schubert, Emma Farrell and the subject matter areas of the ABS for their work in this area.
References
ABS (Australian Bureau of Statistics) 2009a, Quality Gates in the Integrated Collection
Branch, April 2009, Internal ABS paper, ABS, Canberra.
ABS 2009b, Managing Statistical Risk, July 2009, Internal ABS presentation, ABS,
Canberra.
ABS 2010, Quality Management of Statistical Processes Using Quality Gates, Dec 2010,
cat
no.
1540.0,
ABS,
Canberra.
Website
address:
http://www.abs.gov.au/ausstats/abs@nsf/mf/1540.0
ABS 2012, Quality Gates Implementation Strategy, February 2012, Internal ABS
presentation, ABS, Canberra.
Conrish, J 2005, Review of retail Trade Error, July 2005, Internal ABS paper, ABS,
Canberra.
Schubert, P 2007, Quality gates SISC paper, March 2007, Internal ABS paper, ABS,
Canberra.
UNECE (United Nations Economic Commission for Europe) Secretariat 2009, Generic
Statistical Business Process Model, Version 4, April 2009, ENICE, Geneva.