sqaMethods Approach to Measuring
Software Quality
By
Leopoldo A. Gonzalez
[email protected]
sqaMethods Approach to Measuring Software Quality
Measuring Quality
Measuring software quality is one of the areas in software
development where if you can implement it properly, you can be the hero
in your company. Very few companies can tell you with a real degree of
certainty how good their software is. However, the old question remains
“How do I measure quality in my software?”
Strange as it may be, I have realized that the definition of software
quality varies from company to company; however, there are some
common principles that apply, these are:
1. The software will not report any fatal errors.
2. The software will work as advertised.
3. The software will report accurate results.
4. The software is pleasing to the eye.
There are many factors that make up quality, for example; stability,
speed, accuracy, ease of use, etc. Some companies like to track the
customer response and overall impression of how the software is
behaving. While others with more sophisticated schemes, attempt to
measure the lines of code and mean time between failures.
Let’s not forget also the number of trouble issues that are
generated when the software is ultimately deployed to customers.
There are several reasons why quality is not measured, among
some of the most common reasons are:
1. Managers don’t know what to measure.
Most managers find that, while measuring quality is something
they should be doing, it is an activity that does not yield
immediate results. Therefore it is not important enough to
spend time researching on how to implement it.
© 2008 sqaMethods
Page 2 of 11
sqaMethods Approach to Measuring Software Quality
2. It is too hard to create the infrastructure necessary to measure
software quality.
Most managers believe that they cannot afford to have
resources spend any time on creating the processes and reports
on software quality measurement. The ever present need to
deliver the next release in time takes priority over any other
activity in the QA department.
3. Afraid of the result.
Let’s face it; do we really want to know how bad the software
we produce is? What kind of impact will it have in my job? Why
should I cut my own throat?
4. Upper management doesn’t really care.
So you spend time and effort in creating a software quality
measurement and you proudly present it to your management,
but they seem uninterested, even worse, they reprimand you
for doing something other than managing your department.
Whatever the reason may be, our industry suffers from an effective way
of telling us the quality of our software. In the next few paragraphs, I will
present ways and methods to utilize your existing infrastructure to
measure the quality of your software.
A single quality measurement by itself does not reflect the quality
of the software as a whole. If you were to take the number of issues
generated during testing as the basis for measuring quality, the only thing
you would be discovering is how diligent your developers are. It wouldn’t
tell you whether the requirements intended by the designer were
implemented, or whether the customer likes the software.
© 2008 sqaMethods
Page 3 of 11
sqaMethods Approach to Measuring Software Quality
Every company will have its own quality checklist so in creating an
accurate way of measuring software in your shop; you must take in
consideration what is important to your company and what is not.
The best way to ensure that all of these areas are considered is
through the creation of a Software Quality Index that ties them all
together. You will measure every one of your company’s important areas
by applying measuring methods to each of those areas. Then the
combinations of all of the measurable areas create an Overall Quality
Index.
These indexes should be measured by their distance to an
acceptable point. In our method, we measure the percentage value of
acceptability; or the percentage of goodness in the software. The
following section will explain three different quality measurement types;
Test Cases, Bug Severity and Evaluation Sheet.
Quality Index Multiplier
The Quality Index Multiplier is a fractional decimal value that starts
at a given point you define and ends at 1.00. How far below 0 (zero) it
goes will depend on your company’s definition of unacceptable quality. If
your company feels that Sev 1 issues, or 1 Star rating, or lines of code in
a function above a given threshold merits a -2, then you can make that
your starting point. Just remember that the farther away you get from 0
(zero) in a negative direction, the more you will pull down the overall
quality index.
So in defining an index multiplier you will create a scale of numbers
that start, for example, at -1 and end at 1; which is the highest number
representing a perfect score. The scale of numbers in the middle will be
the rankings you will give to each of the measurable areas. For example,
© 2008 sqaMethods
Page 4 of 11
sqaMethods Approach to Measuring Software Quality
Sev 1 issues will be given a negative one (-1) , Sev 2 issues a negative .5
(-.50), Sev 3 issues a negative .25 (-.25) and so on.
In this paper I demonstrate three quality areas that are different in
the way quality may be measured; Test Cases, Bug Severity and Customer
Evaluation Survey. When analyzing your own quality areas, you may find
that they may fit in one of those three categories.
Test Cases
This category is the easiest to measure since the data is simple and
Boolean in nature. The test case either passes or fails. For this category
there is no quality multiplier and the effect on the quality is determined by
calculating the percentage of passed test cases. Hence if there are 200
test cases and 150 pass, then the quality index is .75 (or 75% but we
wont represent the number as percentage since this can get confusing).
This demonstrates a deviation of 25% from an “all good” state. This
same measurement type can be used for any category where the answer
is “yes/no”, “pass/fail” or “complete/not complete”.
© 2008 sqaMethods
Page 5 of 11
sqaMethods Approach to Measuring Software Quality
Bug Severity
The Bug Severity category measures the distance from the 0 (zero)
line since no matter how small the bug is, it affects quality. You cannot
have a bug that enhances quality. The distance between the severity
levels will depend on your definition of quality. In the example below I
have spaced out the severity scores evenly starting with -1.
The following graph demonstrates how the severity
multiplier plots against the severity types.
Severity Multiplier
0
Severity 1
Severity 2
Severity 3
-0.2
Severity 4
-0.25
-0.4
-0.5
-0.6
-0.75
-0.8
-1
-1
-1.2
The Quality Index Multiplier for Severity falls below the 0 (zero) line
since by definition, issues affect the quality of the product so they are
marked with a negative value. Issues classified with a severity level 1 are
more critical than issues with severity of 4. The absence of issues would
result in a 100% quality index because it would mean that testing did not
find any issues. While their existence represent a deviation from 100%,
weighted according to their severity type.
© 2008 sqaMethods
Page 6 of 11
sqaMethods Approach to Measuring Software Quality
When computing the Index Quality for any severity, I take the
percentage of the given severity and multiply it by the Quality Index
severity multiplier. I add all of the results and come up with the Quality
Index for this category.
The following table will help illustrate this concept.
Bug Severity Result
Description
Severity 1
Severity 2
Severity 3
Severity 4
Total Cnt
32
65
73
30
200
Total %
16%
33%
37%
15%
100%
Multiplier
-1
-0.75
-0.5
-0.25
The table above shows 200 issues discovered during testing, most
of them fall in severity 3 which is 37% of the total number of issues. The
multiplier for Severity 3 is -0.5, this gets multiplied by the percentage
yielding a result of -.18. Finally the indexes are added together and the
result becomes the total quality index for this category.
The point becomes clearer when you make all of the issues the
same severity. If the table above would have shown 100% of the issues
belonging to severity 1, then the index would have been -1.00. This
would indicate that there is nothing good in the software. All of the bugs
entered were of the worst kind.
If all of the issues were severity 2, then the software index would
be -0.75 bad. The deviation measured is the distance from an “all good”
state.
© 2008 sqaMethods
Page 7 of 11
Index
-.16
-.24
-.18
-.04
-.62
sqaMethods Approach to Measuring Software Quality
In the example above, all of the individual indexes were combined
to produce an overall value of -0.62, this represents the quality index
value for bug severity.
Evaluation Sheet
An evaluation sheet is a document containing questions that are
aimed at collecting feedback from the field about the perceived quality of
the application. The questions in the evaluation sheet are generated by
the QA manager and the Marketing department. These questions could
be “Were the features easily accessible?”, “Was the layout of the screen
acceptable?”, “Did the program help you complete your tasks?” etc. This
questionnaire can be given to the users to fill out as they perform their
everyday tasks. They will then assign a score to each of the questions on
the list.
The following table measures quality from an evaluation sheet. In
this case, the sheet contains items that improve or degrade quality. Any
question can be given 5 stars, which represent a great score, or 1 star
which represents a bad score.
Multiplier
1.20
1.00
1.00
Quality Value
0.80
0.76
0.60
0.52
0.40
0.28
0.20
0.00
-0.20
1-Star
2-Stars
3-Stars
-0.20
-0.40
Star Value
© 2008 sqaMethods
Page 8 of 11
4-Stars
5-Stars
sqaMethods Approach to Measuring Software Quality
In the chart above, notice how the Star Index multipliers crosses over the
0 (zero) line, this demonstrates that items that fall within the 1 Star
category degrade the quality. As the number of stars increase, quality
improves. I take the percentage value given of each Star category,
multiply it by its multiplier and come up with an index number.
If all of the questions receive a favorable response, they are given
5 Stars. The index value would then be 1.00, indicating that for this
category the quality index is perfect.
On the other hand, if the questions are given only 1 Star, then it
has a negative impact on the quality of the software. In this example, the
star multiplier for 1 Star is -.91. This number can be modified by each
company to reflect the company’s philosophy regarding 1 Star result.
At the end, all of these values are added and I generate a collective
Quality Index for this category.
The following table will help illustrate this concept.
Evaluation Sheet Results
Star Count
1-Star
2-Stars
3-Stars
4-Stars
5-Stars
Count
5
3
4
8
2
22
%
23%
14%
18%
36%
9%
100%
Mult
-0.20
0.28
0.52
0.76
1.00
Index
-0.05
0.04
0.09
0.28
0.09
0.45
In the table above, I compile the results from only one of the
Evaluation Sheets. This evaluation sheet contains 22 questions ranging
from ease of use, speed, aesthetics, etc. BETA testers complete these
evaluation sheets during their testing sessions. After the numbers are
combined, you will get the quality index measurement for this category.
The results of the table indicate that for this category, the quality index is
.45.
© 2008 sqaMethods
Page 9 of 11
sqaMethods Approach to Measuring Software Quality
Quality Index Summary
After you have compiled all the quality indexes, you then average
out the result and represent it as a floating point number. The following
table will help illustrate this point.
Software Quality Index
Category Index
Index
Test Cases
.75
Bug Severity
-.62
Customer Evaluation Sheet
.45
Index
0.48
The table above shows the areas where you need to focus your
attention. Notice that both the Test Cases and the Customer Evaluation
Sheet have positive values while the Bug Severity has a negative value.
This is because the existence of bugs impacts negatively the quality of the
software and therefore you will never attain a perfect index score of 1 for
an Overall Quality Index. However, you have successfully generated an
index of the overall quality of the software. This number can now be used
for Executive Summaries reports or to track your progress as you create
processes and procedures aimed at improving the overall score.
As you can see, most of these measurements can be gathered from
activities you currently perform in your SDLC. With a little bit of effort and
some discipline, you can finally tackle the Software Quality Assurance
measurement monster.
For more information on how to implement this type of quality
measurement in your company or to purchase the sqaMethods Quality
Index spreadsheet, visit us at www.sqamethods.com
© 2008 sqaMethods
Page 10 of 11
sqaMethods Approach to Measuring Software Quality
About Leopoldo Gonzalez
I started my career in the Software Development industry in 1984.
During this time I have performed duties as a Developer, Tester, User
Group Test Coordinator and Testing Automation Architect.
I have compiled my experiences, best practices and practical tools in a
workshop I call “What Every Tester Needs To Know About
Software QA”. This workshop is aimed at individuals who are new
to the world of Software Testing and QA, but also has valuable
information for seasoned Testers such as the subject discussed in this
paper.
For more information on my workshop, visit us at
www.sqaMethods.com
© 2008 sqaMethods
Page 11 of 11