GIS Data Quality Evaluator
Version 5.0
DataLOGIC, Inc.
DataLOGIC Corporation
72 Dartmouth Avenue
Avondale Estates, GA 30002
404-289-4050
www.datalogic-systems.com
Overview
•
•
•
•
Concepts
Types of QA Checks Performed
Evaluation Process
Future Development
DataQE – Data Quality Evaluator
• 1
What is DataQE?
• GIS Data Quality Evaluation Tool
• Evaluations based on user-specified rules
• Focuses on evaluations of attribute fields
and values
• Includes:
–
–
–
–
Layer Checks
Data Structure Checks
Field Checks
Query Checks
DataQE – Data Quality Evaluator
• 2
What Can You Evaluate
with DataQE?
Compliance
• Is data compliant with established standards, such as a
Data Dictionary?
Completeness
• Has required data been appropriately populated?
Suitability
• Is a data source appropriate for a given purpose?
• Does data make sense from a real-world perspective?
DataQE – Data Quality Evaluator
• 3
Who Can Use DataQE?
• Anyone who needs to evaluate the quality or
suitability of a GIS data layer or tabular data
–
–
–
–
–
Data Entry Staff
Data Migration Specialists
GIS Managers
Quality Assurance Staff
Analysts
DataQE – Data Quality Evaluator
• 4
Requirements
• ESRI ArcMap 9.3 or higher
• Windows 7, Vista or XP
• Operates in Desktop (Single User) or
Enterprise (Multi-User) environments,
including Citrix
• Supports the following Data Formats:
– Geodatabases (SDE, File, Personal)
– Shapefiles/Coverages/CAD Files
– Tables (SQL, Oracle, Access, Excel, DBF, CSV)
DataQE – Data Quality Evaluator
• 5
The Data Evaluation Process
• Create Rule Sets containing the rules your data
should follow
• Assign a Rule Set to the data source being
evaluated
• Launch an Evaluation of the data source using the
selected Rule Set
• View Evaluation Results and identify data errors
• Correct any errors as necessary using ArcMap or
other editing tools
• Re-evaluate as necessary
DataQE – Data Quality Evaluator
• 6
Layer Checks
Verify commonly used properties within a spatial data source.
• Data Source/File Name Check
– Verifies that the name of the evaluated data source name matches
a list of valid names
• Layer Type Check
– Verifies that the layer type of the evaluated data source matches
specified requirements (GDB vs. Shapefile vs. Coverage, etc.)
• Projection Check
– Verifies that a layer’s current projection matches established
projection requirements
DataQE – Data Quality Evaluator
• 7
Data Structure Checks
Verify that a data source’s field structure matches
appropriate requirements.
• Field Exists Check
– Verifies that required fields exist in the data source being
evaluated
• Field Type Check
– Verifies that each field is the correct type (Text, Date, etc.)
• Field Width Check
– Verifies that each field is the correct width
DataQE – Data Quality Evaluator
• 8
Field Checks
Verify that attribute values within a field match established
standards.
• Required Value Check
– Verifies that each record in a field contains a value
• Unique Value Check
– Verifies that each record in a field contains a unique value
• List of Values (LOV) Check
– Verifies that each record in a field contains a value that matches a
specified list of values (domain)
• Valid Range Check
– Verifies that each record in a numeric field contains a value that
falls within a specified range
DataQE – Data Quality Evaluator
• 9
Query Checks
User-defined rules based on customizable queries.
• Used to perform in-depth evaluations of a data source
• Query Rules can be created using the DataQE Query
Builder, or copied from existing ArcMap or database
queries
• Allows queries to be stored permanently in Rule Sets
• Can be used to expand basic compliance evaluations to
determine whether data meets established Business Rules
• Can be used by analysts to evaluate whether data is
appropriate for a particular purpose
• Can be used to validate whether data “makes sense”, as
opposed to simply meeting data dictionary standards
DataQE – Data Quality Evaluator
• 10
Data Quality Evaluator Window
• Used to perform evaluations of selected
data sources
• View Rule Set
Assignments
and properties
• Review
Evaluation
Results
DataQE – Data Quality Evaluator
• 11
Data Quality Evaluator
Data Sources
Window
Rule Set
Explorer
Results
Window
Rule Set Editor
Used for managing and editing Rule Sets
• Create Rule Sets
• Add/Edit rules
within a Rule Set
• Rule Sets can be
published and
shared between
multiple users
DataQE – Data Quality Evaluator
• 13
Rule Set Editor – Properties Window
Rule Set
Explorer
Properties
Window
Rule Set Editor – Layer Rule Properties
Rule Set
Explorer
Layer
Rules
Rule Set Editor – Field Rule Properties
Rule Set
Explorer
Field Rule
Properties
Rule Set Editor – Field Rules Grid
Rule Set
Explorer
Field Rules
Grid
Query Rule Properties
• Allow users to
build custom
Query Rules.
• Query Rule
Properties tab
helps users build
a query
statement that
can be used to
evaluate a field
or combination
of fields.
DataQE – Data Quality Evaluator
• 18
Rule Set Editor – Query Rule Properties
Rule Set
Explorer
Query Rule
Properties
List of Values
Used for managing and editing Lists of
Values (LOV’s) for Domain Checks
• Create new LOV’s
• Add/Edit Values
within an LOV
• Assign LOV’s to
Rule Sets on the
Field Rule
Properties and the
Field Rules Grid
tabs.
DataQE – Data Quality Evaluator
• 20
Rule Set Editor – List of Values
Rule Set
Explorer
Valid Values
Create LOV’s from Existing Data
Use data from the current ArcMap session
or browse for spatial or tabular data.
• Create new LOV’s
based on unique
field values.
• Specify a field for
the LOV value
and another field
for the LOV
description.
DataQE – Data Quality Evaluator
• 22
Create LOV’s from Existing Data
Evaluation Manager
Used to manage and edit predefined Evaluation
Definitions to support batch processing
• Allows creation of Evaluations
with pre-assigned properties
• Easy selection of Data
Sources and Rule Sets
• Evaluation Definitions can be
stored & shared
• Allows definitions to be saved
and performed at any time,
individually or in batches
DataQE – Data Quality Evaluator
• 24
Evaluation Manager
Evaluation
Definitions
Window
Properties
Window
Future Development
Coming in future versions of DataQE:
• Associated Tables/Orphan Records Check
• Metadata Check
• User Definable Field Rules
– Business Logic Check
– Spatial Overlay Check
DataQE – Data Quality Evaluator
• 26