Software Test Metrics
When you can measure what you are speaking about and
express it in numbers, you know something about it; but
when you cannot measure, when you cannot express it in
numbers, your knowledge is of a meager and
unsatisfactory kind: it may be the beginning of knowledge,
but you have scarcely, in your thoughts, advanced to the
stage of science.
(reference: unknown)
• Measure - quantitative indication of extent, amount,
dimension, capacity, or size of some attribute of a
product or process.
• Metric - quantitative measure of degree to which a
system, component or process possesses a given
attribute. “A handle or guess about a given
attribute.”
Why We need Metrics?
“You cannot improve what you cannot measure.”
“You cannot control what you cannot measure”
AND TEST METRICS HELPS IN
–
–
–
–
–
–
Take decision for next phase of activities
Evidence of the claim or prediction
Understand the type of improvement required
Take decision on process or technology change
Improve software quality
Anticipate and reduce future maintenance needs
Metric Classification
Metric Classification
• Products
– Explicit results of software development activities
– Deliverables, documentation, by products
• Processes
– Activities related to production of software
• Resources
– Inputs into the software development activities
– hardware, knowledge, people
Product vs. Process
• Process Metrics
– Insights of process paradigm, software engineering tasks,
work product, or milestones
– Lead to long term process improvement
• Product Metrics
–
–
–
–
–
Assesses the state of the project
Track potential risks
Uncover problem areas
Adjust workflow or tasks
Evaluate teams ability to control quality
Types of Measures
• Direct Measures (internal attributes)
– Cost, effort, LOC, speed, memory
• Indirect Measures (external attributes)
– Functionality, quality, complexity, efficiency,
reliability, maintainability
Size-Oriented Metrics
•
•
•
•
•
Size of the software produced
LOC - Lines Of Code
KLOC - 1000 Lines Of Code
SLOC – Statement Lines of Code (ignore whitespace)
Typical Measures:
– Errors or Defects/KLOC, Cost/LOC, Documentation
Pages/KLOC
McCabe’s Complexity Measures
• McCabe’s metrics are based on a control flow
representation of the program.
• A program graph is used to depict control
flow.
• Nodes represent processing tasks (one or
more code statements)
• Edges represent control flow between nodes
Steps for CC:
1: Draw a control flow graph
2: Calculate Cyclomatic complexity (CC)
3: Choose a “basis set” of paths
– A basis path is a unique path through the software where no iterations are
allowed - all possible paths through the system are linear combinations of
them.
4: Generate test cases to exercise each path
• Control Flow Graph
– Any procedural design can be translated into a
control flow graph
– Lines (or arrows) called edges represent flow of
control
– Circles called nodes represent one or more actions
– Areas bounded by edges and nodes called regions
– A predicate node is a node containing a condition
Control Flow Graph Structures
Basic control flow graph structures:
Cyclomatic Complexity
• Set of independent paths through the graph
(basis set)
• V(G) = E – N + 2
– E is the number of flow graph edges
– N is the number of nodes
• V(G) = P + 1
– P is the number of predicate nodes
Example One
int example1 (int value, boolean cond1, boolean cond2)
{
1 if ( cond1 )
2 value ++;
3 if ( cond2 )
4 value --;
5 return value;
}
V(G) = E – N + 2; 6 – 5+2
Complexity = 3
Basis Paths
Test Data
135
false false
1235
true false
12345
true true
• Cyclomatic Complexity - Exercise • Calculate CC for the following Control flow
graphs
Solution
• Cyclomatic complexity for the two control flow
graphs;
• V(G) = edges - nodes + 2
– 12 edges
– 9 nodes
– 12-9+2=5
– 13 edges
– 10 nodes
– 13-10+2=5
• Predicate Nodes Formula
• One possible way of calculating V(G) is to use
the predicate nodes formula:
• V(G) = Number of Predicate Nodes + 1
• Thus for this particular graph: V(G)=1+1=2
• A limitation on the predicate nodes formula is
that it assumes that there are only two
outgoing flows for each of such nodes.
• Basis Path Testing During Integration
• Basis path testing can also be applied to
integration testing when units/components
are integrated together
• McCabe’s Design Predicate approach:
• Draw a “structure chart”
• Calculate integration complexity
• Select tests to exercise every “type” of
interaction, not every combination
Types of Interactions
Types of unit/component interactions:
• Unconditional
– Unit/component A always calls unit/component B. A calling
tree always has an integration complexity of at least one. The
integration complexity is NOT incremented for each occurrence
of an unconditional interaction.
• Conditional
– Unit/component A calls unit/component B only if certain
conditions are met. The integration complexity is incremented
by one for each occurrence of a conditional interaction in the
structure chart.
• Mutually Exclusive both)
– Unit/component A calls either unit/component B or
unit/component C (but not based upon certain conditions
being met.) The integration complexity is incremented by one
for each occurrence of a mutually exclusive conditional
interaction in the structure chart.
• Iterative
– Unit/component A calls unit/component B one or more times
based upon certain conditions being met. The integration
complexity is incremented by one for each occurrence of an
iterative interaction in the structure chart.
• Integration Basis Path Test Cases
• Basis path testing can also be applied to
integration testing when units/components are
integrated together
• Complexity: 5
• Basis set of paths:
–
–
–
–
–
1. A,B,C,E & then A calls I
2. A,B,C,F & then A calls I
3. A,B,D,G (only once) & then A calls I
4. A,B,D,G (more than once) & then A calls I
5. A,B,D,G,H & then A calls I
• Integration Basis Path Testing
• Interaction types:
–
–
–
–
2 conditional
1 mutually exclusive conditional
1 iteration
1 simply because we have a graph
• Basis set of paths:
1. A,C,F (only once)
2. A,C,F (more than once)
3. A,B,D,G & then B calls E &
then A calls C
4. A,B,D,H & then B calls E & then A calls C,F
5. A,B,D,H & then B calls E, I & then A calls C,F
Test Case defect density
Total number of errors found in test scripts v/s developed and
executed.
• (Defective Test Scripts /Total Test Scripts) * 100
Example: Total test script developed 1360, total test script
executed 1280, total test script passed 1065, total test script
failed 215
So, test case defect density is
215 X 100
---------------------------- = 16.8%
1280
This 16.8% value can also be called as test case efficiency %,
which depends upon total number of test cases which
uncovered defects
***(1280-1065=215)
Review Efficiency
The Review Efficiency is a metric that offers insight on the review
quality and testing
Some organization also use this term as “Static Testing” efficiency and
they are aiming to get min of 30% defects in static testing
Review efficiency=100*Total number of defects found by reviews/Total
number of project defects
Example: A project found total 269 defects in different reviews, which
were fixed and test team got 476 defects which were reported and
valid
So, Review efficiency is [269/(269+476)] X 100 = 36.1%
• Defect Removal Effectiveness
DRE= Defects removed during development phase x100%
Defects latent in the product
Defects latent in the product = Defects removed during
development phase+ defects found later by user
Example:
•
•
•
•
•
The Defect Removal Effectiveness for each of the phases would be as follows:
Requirements DRE = 10 / (10+3+0+2+1) x 100% = 63%
Design DRE = (3+18) / (3+0+2+1+18+4+5+2) x 100% = 60%
Coding DRE = (0+4+26) / (0+2+1+4+5+2+26+8+7) x 100% = 55%
Testing DRE = (2+5+8) / (2+1+5+2+8+7) x 100% = 60%
•
Development DRE = (Pre-release Defect) / (Total Defects) x 100%
(10+3+2+18+4+5+26+8) / (10+3+2+1+18+4+5+2+26+8+7) x 100 = 88%
• According to Capers Jones, world class organizations have Development
DRE greater than 95%.
• The longer a defect exists in a product before it is detected, the more
expensive it is to fix.
• Knowing the DRE for each phase can help an organization target its
process improvement efforts to improve defect detection methods where
they can be most effective.