Testing Metrics
Software Reliability
CSSE 376, Software Quality Assurance
Rose-Hulman Institute of Technology
April 5, 2007
1
Outline
Testing Metrics
 An Important Metric: Reliability

2
Common Metrics

Product

KLOC




thousands of lines
of code
need to remove
comment lines?

Process



Staff hours
Tests planned
Tests passed
Function Points
#Bugs
3
Bug Density
Measure #Bugs/KLOC
 Expect different densities at different
stages of a project
 May categorize bugs by severity

4
Example Bug Density
30
Bugs/KLOC
25
20
Catastrophic
15
Major
Minor
10
5
0
Reqts
HLD
LLD
Code UnitTest SysTest
5
Cartoon of the Day (1/3)
6
Using Bug Metrics
Count bugs discovered during each
phase of a project
 Compare to previous projects

provides estimates of expected values at
each phase -- could use to set milestone
 deviation of more than 20% from expected
indicates need for investigation

7
Analysis of Bug Data

Root cause analysis
Search for explanations
 Might look at other process data (effort,
experience of team, etc.)


Trend analysis

Make predictions from current data
8
Reliability
9
Failures vs. Faults

Fault:
developer-oriented
 6 faults/1000 source lines


Failures:
customer-oriented
 3 failures/1000 CPU hours

10
Calculating Reliability

Reliability
probability of failure-free operation for a
specified time interval
 0.82 for 8 CPU hours


Failure Intensity
number of observed failures within a
specified time interval
 3 failures/1000 CPU hours

11
Factors Influencing Reliability

Fault removal


by error correction (debugging)
Fault introduction
by error correction (unintended)
 by new feature development


Operational profile
12
Operational Profile
0.05
0.045
0.04
0.035
Probability
of Use
0.03
0.025
0.02
0.015
0.01
0.005
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
Functions
13
Example
Function
Usage
Distribution
Probability Interval
Change
32%
0-31
Delete
14%
32-45
Insert
46%
46-91
Print
8%
92-99
14
Test Generation
Test
1
2
3
4
Random
Numbers
29, 11, 47,
52, 26, 94
62, 98, 39,
78, 82, 65
83, 32, 58,
41, 36, 17
36, 49, 96,
82, 20, 77
Test Cases
C, C, I,
I, C, P
I, P, D,
I, I, I
I, D, I,
D, D, C
D, I, P,
I, C, I
15
Test Compression

Real use of a product involves repetitive
operations
different users invoke the same operations
 same user invokes the same operations on
different days

Redundant tests waste computer and
personnel time
 Compression: when generating random
tests, do not duplicate previous tests

16
Cartoon of the Day (2/3)
17
Cartoon of the Day (3/3)
18
Curve Fitting
Reliability models focus on failure
removal
 Use a random process to model the
failure removal process

19
Execution Time Model
Failure
Intensity
Goal
Execution Time
20
Resource Constraints

Early phase of a project


Middle phase


constrained by availability of developers
(debuggers)
constrained by availability of testers
Late phase
constrained by availability of machines
 may run tests in parallel to increase number
of tests per CPU hour

21
Adjusting for Calendar Time
Estimate resource usage during each
phase of the project
 Model calculates failure intensity in terms
of execution time
 Model adjusts fault removal rate
according to resource constraints

22
Calendar Time Component
Failure
Intensity
Constrained by debuggers
Constrained by testers
Constrained by machines
Goal
Execution Time
23
Calculating CalendarTime/ExecutionTime Ratio
10 staff-hours to fix each failure
2 failures/CPU-hr
That means it will take 10 * 2 = 20 staff-hrs
per CPU-hr
Suppose you have 5 developers
Then you have 20 / 5 = 4 hrs per CPU-hr
Each CPU-hr will take 4 calendar hours
24
Estimating Completion
1.
2.
3.
4.
Establish a failure intensity objective
Record execution times of failures
Run model to estimate reliability
Model reports estimated completion
date
Values are not absolute---within
confidence bounds
25
Estimating Completion
Failure
Intensity
Goal
Calendar Time
Ship
Date
26
Acceptance Charts
Bugs
Reject
Accept
Time
27