Metrics:TheForce Awakens
ByJosephOurs
NOTICE: Proprietaryand Confidential
Thismaterial isproprietaryto CentricConsulting,LLC. It containstrade secretsand informationwhich issolely the prope rty of CentricConsulting,LLC. Thismaterial is
or in part,without the expressconsent of Centric
Consulting,LLC.
© 2013CentricConsulting,LLC.All rightsreserved
If measureit you
cannot,
improve it you
cannot.
[IJ
se
rl t y S u n 1 m a r y : D e c 1 3 , 2 0 0 7 t hr o ug h M a r 1 2 , 2009
Vu
,..btlify T , . . d CAVC)
20
oJ..,J._,....._,..._. 500
0 ....
•;
ICAlt" , , . . . . . , (AVC.)
Chris's Dashboard
•.o
, ,..
10
"
..,
..,
,,
II
0
.111
Live
....,.... .
..
1r d Rt
- ·""
0 . t . c t COU: I
..
Examplesof SoftwareTestingMetrics
Test Cases
•
•
•
•
•
•
•
•
Test Case Counts by Execution Status
Test Case Percentages by Execution Status
Test Case Execution Status Trend
Test Case Status Counts vs Test Case Status Counts
Test Case Coverage
Test Case Status vs Coverage
Test Case First Run Failure Counts
Test Case Re-Run Counts
Automation extras
• Automation Index (Percent Automatable)
• Automation Progress
• Automation Test Coverage
Examplesof SoftwareTestingMetrics
Defects
•
•
•
•
•
•
•
Defect Counts by Status
Defect Counts by Priority
Defect Status Trend
Defect Density
Defect Remove Efficiency
Defect Leakage
Average Defect Response Time
Other
• Requirements Volatility Index
• Testing Process Efficiency
The productof
measuring,
quantitiesare
Test Cases Executed
45
40
35
30
25
20
15
10
5
0
Each Day of Testing
Test Cases Executed
350
300
250
200
150
100
50
0
Each Day of Testing
Test Cases Executed
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
Each Day of Testing
Test Cases Executed
350
300
250
200
150
100
50
0
Each Day of Testing
On Everything
Labels,Weput
Important,Not
EverythingIs
Defects By Severity
4-Low
1-Critical
1-Critical
2-High
3-Medium
3-Medium
4-Low
2-High
Defects By Severity
3-Medium
2-High
1-Critical
4-Low
3-Medium
4-Low
1-Critical
2-High
Defects By Severity
40
2-High
3-Medium
35
30
25
20
1-Critical
1-Critical
2-High
3-Medium
15
4-Low
10
5
0
1-Critical
2-High
3-Medium
4-Low
4-Low
Truncating
introduces
distortion.
Yeesssssss.
Defect Open and Close Rate
120
100
80
Open
60
Closed
40
20
0
Each Day of Testing
Defect Open and Close Rate
25
20
15
Open
Closed
10
5
0
Each Day of Testing
Defect Open and Close Rate
20
19
18
17
16
Open
15
Closed
14
13
12
11
10
Each Day of Testing
Win therace, the
turtlewill,but the
we'dratherwatch
the harerun. Yes,
hmmm.
Test Cases Executed vs Planned
350
300
250
200
Planned
Executed
150
100
50
0
Each Day of Testing
Test Cases Executed vs Planned
1000
100
Planned
Executed
10
1
Each Day of Testing
Seldom weighed in
thesame scales,
defectswe find
andtestswe runare.
Test Cases Executed vs Defects
350
300
250
200
Defects Opened
Executed Tests
150
100
50
0
Each Day of Testing
Test Cases Executed vs Defects
350
120
300
100
250
80
200
60
150
40
100
50
20
0
0
Each Day of Testing
Executed Tests
Defects Opened
An issueofmind
over matter,
defect ageis. If
mind younot, it
mattersnot.
Hmmmmmm.
Defects Opened vs Average Defect Age
120
25
100
20
80
15
Defects Opened
60
Defect Age
10
40
5
20
0
0
Each Day of Testing
Defects Opened vs Average Defect Age
120
0
100
5
80
10
Defects Opened
60
Defect Age
15
40
20
20
0
25
Each Day of Testing
CommonWaysto Manipulate Graphs
Plot Cumulative vs
Actual
Distort Perspective
with 3D
Truncate Axis
Plot Secondary
Axis
Logarithmic Axis
Invert Axis
((CENTR IC))
((CENTR IC))
DOGBERT CONSULTS
YOU NEED A D A S H BOARD APPLICATION
TO TRACK YOUR
KEY METRICS.
\
u
E
0
q
0
@)
,,
s::
0
0
Cl)
.E
THAT WAY YOU'LL HAVE
MORE DATA TO GNORE
WHEN YOU MAKE YOUR
DEC S ONS BASED ON
COMPA N POL T CS.
E
0
CJ
t::
.8
t::::,
>,
.0
u
-
0
0
.E
,;
E
l,J
W I L L THE
DATA BE
ACCURATE?
OKAY
LET'S
PRETEND
THAT
MATTERS.
'O
c(
ti
/i,
s
0
C'II
@
...
"'
I D O N ' T T H I N K OUR
NETW ORK CAN
HANDLE THE E X T R A
NEn..JORK
TRAFFIC .
DOUG . COl"\E
HERE FOR A
M NUTE .
HOW A B O U T L O G I C ?
OUR NETW ORK S
ALREADY TOO SLOW
A N D W E PLAN TO
QUADRUPLE USAGE.
THE DATA PROVES T H A T
OUR NETW ORK HAS
IN F IN ITE C A P A C IT Y .
SO YOUR BUDGET
SHOULD BE D I V E R TED
TO M Y PR..O J"ECT .
T H A T ' S N O T S CI E NCE I
T H A T ' S A WEASEL
l"\ISINTERPRETING
DATA TO GET EXTRA
FUNDING !
B A H i W ATCH HOW
S CI E NCE W ORKS
A N D M A Y B E YOU'LL
L E A RN S O M E T H I N G .
e
J
i
THERE ARE DAYS W HEN
YOU REALLY HATE TO
HEAR.. THE PHRASE
" P O-TA Y-T O , PO-TAH-TO. "
((CENTR IC))
Purpose ofMetrics
Measure of
Performance
Conformance to a
Stated Practice
Deviation from Goal
Changing theWorld
How to LeverageMetrics
Explicitly link metrics to goals
Use trends over absolute numbers
Track progress against total
Use shorter tracking periods
Leverage visualizations appropriately
Change metrics when they stop driving change
Q&A
Joseph Ours
Email:
[email protected]
Company Website:
https://centricconsulting.com/technol
ogy-solutions/software-qualityassurance-and-testing/
Twitter:
@justjoehere
LinkedIN:
www.linkedin.com/josephours
Personal Blog:
http://josephours.blogspot.com