Test Metrics
Metrics Framework
Metrics are critical for gathering information that will lead to good decision-making.
Metrics should be limited to a “vital few” metrics.
Mission
Establish a standard framework and implement appropriate tools to gather, track and report
standardized performance metrics and measurements, enabling an organization to
effectively manage test planning and test execution efforts, to accurately evaluate the
quality of delivered solutions, and to consistently assess the overall test effectiveness of
applications and systems.
“Lord Kelvin, a renowned British physicist, is reputed to have said:
‘When you can measure what you are speaking about, and express
it in numbers, you know something about it … [otherwise] your
knowledge is of a meager and unsatisfactory kind; it may be the
beginning of knowledge, but you have scarcely in thought advanced
to the stage of science.””
- quotation from ‘Measuring Business Performance’, A. Neely, The
Economist Books,1998
Metrics - Definitions
Basic Definitions
 Metric: Quantitative performance measure of the degree to which a system, component, or
process possesses a given attribute. A calculated or composite indicator based upon a measure
value. Example: Total Test Effort
 Measure: To ascertain or appraise by comparing to a standard. A standard or unit of
measurement; the extent, dimension, capacity, etc. of anything, especially as determined by a
standard; a result of a measurement. Example: # of total actual hours billed to the project
 Measurement: The act or process of recording or assessing a value of something, such as a
figure, extent or amount. Example: 368 actual hours billed to project A
 Core Metric: Required metric that is essential to support solution delivery test management on
systems development projects. Example: Percentage of requirements met
 Non-Core Metric: Optional metric that can help to create a more balanced picture of the quality
and effectiveness of test efforts. Example: Total number of defects by test phase
 Base Measure: Direct performance measurement of a system, component, or process.
Example: Total number of open defects
 Derived Measure: Calculated performance measurement using base measures as input to
obtain a more sophisticated view of a system, component, or process. Example: Return on
Quality Investment
Cost
Performance
Productivity
Metric Category
Best Practice Metrics
Quality of System Number of Defects per Test Pass/Cycle
Number of Defects Found and Fixed
Number of Business Requirements met During
Testing
Test Progress
Test Progress Measured by Test Scenario
Completion
Test Progress Measured by Test Pass/Cycle
Completion
Defects Raised and Closed
How Quickly Defects are Being Raised and
Fixed (by Severity and/or Priority)
Defect Fix Effort by Severity and/or Priority
Defect Repair Effort Quality
Number of Defects by Group or Area
Number of User Execution or Validation Errors
Test Management Testing Effort
Overtime
Number of Defects Waiting for Retest
Quality
Sample testing metrics
v
v
v
v
v
v
v
v
v
v
v
v
v
v
v
v
Sample Metrics – Test Planning
Test Activity
Approach
Approach
Plan
Plan
Plan
Plan
Plan
Category
Productivity
Productivity
Productivity
Productivity
Productivity
Productivity
Productivity
Primary Indicator
Test Plan
Test Plan
Conditions
Conditions
Conditions
Conditions
Conditions
Prep
Prep
Prep
Prep
Prep
Prep
Prep
Productivity
Productivity
Productivity
Productivity
Productivity
Productivity
Productivity
Scripts
Scripts
Scripts
Scripts
Scripts
Scripts
Scripts
Environment
Environment
Productivity
Productivity
Environments
Environments
Metric
Calculation
Total numbers of hours spent establishing test approach
Total hours dedicated to test planning, by stage
Total number of signoffs on test approach
Total number of signoffs on test approach
Total numbers of hours spent on planning test (test Conditions) Total hours dedicated to test planning, by stage
Total number of test conditions documented and signed off
Total number of test conditions that have been signed-off
Test Conditions /Function
Test Conditions /Function
Test Conditions /Release or Package
Test Conditions /Release or Package
Test Planning Rate
Total number of hours spent planning the test stage/ Total number of
conditions documented and signed off for the stage
Total # of test scripts created
Total # of test scripts created
Total % of test scripts created
(Total # of test scripts created/Total # of planned scripts)*100
Total number of hours spent preparing scripts
Total hours dedicated to test planning, by stage
Total number of hours spent preparing scripts by stage
Total hours dedicated to creating test scripts, by stage
Number of scripts entered into repository
Number of scripts entered into repository
% of scripts entered into repository
% of scripts entered into repository
Test Preparation Rate
Total number of hours spent scripting the test stage/ Total number of
cycles which have been scripted and signed off this test stage
Test environments established on time
Test environments established on time
Test environments validated on time
Test environments validated on time
Sample Metrics – Test Execution
Below are sample metrics for productivity measures of test execution using test scripts…
Test Activity
Execution
Execution
Execution
Execution
Execution
Category
Productivity
Productivity
Productivity
Productivity
Productivity
Primary Indicator
Scripts
Scripts
Scripts
Scripts
Scripts
Execution
Execution
Execution
Execution
Execution
Execution
Execution
Execution
Execution
Execution
Execution
Execution
Productivity
Productivity
Productivity
Productivity
Productivity
Productivity
Productivity
Productivity
Productivity
Productivity
Productivity
Productivity
Scripts
Scripts
Scripts
Scripts
Scripts
Scripts
Scripts
Scripts
Scripts
Scripts
Scripts
Scripts
Metric
Total hours spent executing test scripts
Total hours spent executing each test stage
Test Hrs/Function
Test Scripts/Hour
Test Execution Rate
Total # of test scripts executed
Total # of test scripts In Progress
Total # of test scripts NOT Started
Total # of test scripts Scheduled
Total % of test scripts executed
Total % of test scripts In Progress
Total % of test scripts NOT Started
Total % of test scripts Scheduled
Total # of test scripts added
Total # of test scripts removed
Total # of test scripts (results) signed off
Total % of test scripts (results) signed off
Calculation
Total hours of testing
Total hours of testing per stage
Test Hrs/Function
Test Scripts/Hour
Total number of hours spent executing the test stage/ Total number of
test scripts executed in the stage
Total # of test scripts executed
Total # of test scripts In Progress
Total # of test scripts NOT Started
Total # of test scripts Scheduled
Total # of test scripts executed
Total # of test scripts In Progress/ Total number of scripts
Total # of test scripts NOT Started/ Total number of scripts
Total # of test scripts Scheduled/ Total number of scripts
Total # of test scripts added
Total # of test scripts removed
Total # of test scripts (results) signed off
Total % of test scripts (results) signed off
Sample Metrics - Defects
Test Activity
Execution
Execution
Execution
Execution
Category
Productivity
Productivity
Productivity
Productivity
Primary Indicator
Defects
Defects
Defects
Defects
Metric
Average time to fix a defect for a given stage
Number of hours spent fixing defects at a given stage
Number of defects fixed correctly for a given stage
(Stage Containment) Repair Effort Percentage
Execution
Productivity
Defects
Repair Effectiveness Percentage
Execution
Productivity
Defects
Average Turnaround Days of Defects by priority
Execution
Execution
Execution
Execution
Productivity
Productivity
Productivity
Productivity
Defects
Defects
Defects
Defects
Best Defect Turnaround by priority
Worst Defect Turnaround by priority
Defect Fix Rate
Defect Closure Rate by priority
Execution
Productivity
Defects
Defect Emergence Rate by priority
Execution
Productivity
Defects
Testing Effectiveness Percentage
Calculation
Total # of defects/Total time to fix defects
Total hours to fix defects by stage
# of defects fixed per stage
(# of hours spent repairing defects from a given stage/original number
of hours to build the stage)*100
(# defects fixed correctly the first time for a given stage/total #
attempted fix for the stage)*100
Sum of all Turnaround Days of Defects of Priority P resolved during the
week/Number of Defects of Priority <P> resolved during the week
Minimum among the Turnaround Days of All Defects of Priority <P>
Maximum among the Turnaround Days of All Defects of Priority <P>
number of defects fixed / hours spent fixing defects
Defects of Priority <P> whose Date Closed falls within the start and end
of reporting week
Defects of Priority <P> whose Date Open falls within the start and end
of reporting week
(Number of defects found during testing for a given stage/ Hours of
testing for a given stage) * 100
Sample Metrics – General Productivity
Test Activity
Execution
Execution
Execution
Execution
Execution
Execution
Execution
Execution
Execution
Execution
Execution
Execution
Execution
Execution
Execution
Execution
Execution
All/General
All/General
All/General
All/General
Category
Quality
Quality
Quality
Quality
Quality
Quality
Quality
Quality
Quality
Quality
Quality
Quality
Quality
Quality
Quality
Quality
Quality
Productivity
Productivity
Productivity
Productivity
Primary Indicator
Defects
Defects
Defects
Defects
Defects
Defects
Defects
Defects
Defects
Defects
Defects
Defects
Defects
Scripts
Scripts
Scripts
Scripts
time/schedule
work/effort
work/effort
work/effort
Metric
Total # of defects found
Total # of defects fixed
Total # of outstanding defects
Number of defects by module
Total defects by stage
Total defect by severity
Total defect by severity by stage
# of defects by type (application, data, environment, script)
# of defects by component (online, batch, report, etc)
# of defects by component complexity
Defect rate
Defect ratio
Defect density
Total # of test scripts Passed
Total % of test scripts Passed
Total # of test scripts Failed
Total % of test scripts Failed
Duration Variance Percentage
Effort Variance Percentage
Number of Personnel entering time into time management tool
% of Personnel entering time into time management tool
Calculation
Total # of defects found
Total # of defects fixed
Total # of outstanding defects
# of defects per module
# of defects by stage
# of total defects by severity
# of total defects by severity by stage
# of defects created for each category
# of defects created for each component
# of defects for each component, by component complexity
number of defects / days or weeks of execution
# of defects /software size
# Defects/KSLOC (thousand lines of code)
Total # of test scripts Passed
Total # of test scripts Passed/ Total number of scripts
Total # of test scripts Failed
Total # of test scripts Failed/ Total number of scripts
[(actual duration/planned duration) *100]-100
[(actual effort/planned effort) *100]-100
Number of Personnel entering time into time management tool
% of Personnel entering time into time management tool
Test Metric Start Up
Hours Accounting
MS Project
Project
Workplans
- Test Process Task Mapping
- Compliance Reports
Total Effort
Total QA Effort
Total Rework Effort
Total Test Effort
Total RW Effort
Cost Variance
Schedule Variance
- Accuracy and Compliance Reports
- Cost / Schedule Reports
Defect Tracking
Development Defects
- Test Management Tool
- Project Database (Interim)
- Project Tools (Temporary)
Production
Problems
- Project Reports
- Test Management Tool Reports
Accuracy & Completeness, Integrity Verification
The consolidation of metric data sources, integrated verification of data quality, and a
centralized repository for capturing measurement results enables an organization to
successfully establish a test metric framework and its associated processes.
Exec
Dashboard
Test Manager
Dashboard
Dashboard
Reporting
Engine
& Archive
(Central Repository)
Other Sources
Risk-Based Testing
Training Statistics
Customer Satisfaction