Do Software Defect Backlogs really matter
if the Software is actively being used?
March 2014
Scope of this Report
The question driving this report assumes that a software defect backlog already exists. Hence, some but
perhaps not all defects have been identified and logged. It assumes that the defects described in the
software defect backlog are not sufficient to prevent the software from functioning. Hence, this report
ignores much of the literature about software defect detection and tracking (which is adequately
covered in the Sources listed for interested parties) to focus on the narrower perspective of what, if
anything, should be done about those unaddressed defects in the software that have been reported by
developers, testers, end-users or code analysis software.
The report concludes that there is no way to know if a particular Software Defect Backlog for a particular
application or product matters without further analysis. The report recommends an approach for this
further analysis.
Definitions
Defect
Latent Defect
Patent Defect
Defects/function point
Nonconformance of a characteristic with specified requirements, or a
deficiency in something necessary for an item’s intended, proper use.
A defect that is unknown at time of release
A defect that is known, reported and logged but not fixed at time of release
The average number of defects found per function point
Defect Backlog (ideal): The difference between defects created and fixed (may be one or more
severities) which is ideally zero at the start and end of each project
©2014 David Consulting Group
Page 1 of 6
Defect Backlog (realistic): New enhancement projects tend to inherit defects in exisitng code and to
close without fixing all of the defect sthat they created (as shown in the first chart below). This results
in an increase in the defect backlog over the ourse of the project (as shown in the secong chart below).
Defect Severity Classification
The actual terminologies, and their meaning, can vary depending on people, projects, organizations, or
defect tracking tools, but the following is a normally accepted classification:
Name label
(Typical)
Critical:
Major:
Definition
(Typical)
The defect affects critical
functionality or critical data. It
does not have a workaround.
The defect affects major
functionality or major data. It
has a workaround but is not
obvious and is difficult.
Minor:
The defect affects minor
functionality or non-critical
data. It has an easy
workaround.
Trivial:
The defect does not affect
functionality or data. It does
not even need a workaround.
It does not impact
productivity or efficiency. It is
merely an inconvenience.
©2014 David Consulting Group
Example
Unsuccessful installation,
complete failure of a
feature.
A feature is not functional
from one module but the
task is doable if 10
complicated indirect steps
are followed in another
module/s.
A minor feature that is not
functional in one module
but the same task is easily
doable from another
module.
Petty layout discrepancies,
spelling/grammatical errors.
Page 2 of 6
Severity label
(Typical)
1
(or “Sev1”)
2
3
4
Defect Management Practices
To quote Steven Thomas, “In fact defects is an area where software delivery, including agile software
development, can get messy.” Thomas goes on to list nine common strategies for managing a growing
fault list:
1. Do nothing - Who cares that the product backlog is large? Product Owner? Developers? Project
Manager? If nobody cares then it isn’t a problem.
2. Filter out the noise – Why do they care? Can some be labeled as “low impact”?
3. Stop logging low impact defects – If there is a well understood definition of “low impact” then
don’t even log them to avoid wasting time discussing them in future.
4. Close off low impact defects – get rid of the “low impacts” already on the list
5. Prune the backlog – every so often remove defects from the backlog that have “aged out” and
are no longer relevant.
6. Batch the defects - batch related defects together. They might be high or low impact individually
but in combination their overall value will increase. For example you might group together a set
of “email” defects and get the product owner to prioritize these groupings rather than the
individual items.
7. Blitz the defects - A common tactic is to blitz the defects. That means allocating the whole team
to fixing defects for a sprint. This often happens immediately before a major launch. Incidentally,
we have seen organizations set up special defect blitz events (maybe out of hours) where they
pay developers for the number of defects fixed. This does NOT generate the desired behavior!
8. Fix some every sprint - It is also common to allocate a certain proportion of effort to fixing faults
each sprint. Thomas advocates giving the technical team time each sprint to fix technical debt
and fixing faults can be rolled into that.
9. Automate the tests – Frankly, this is such a fundamental requirement in today’s software
development that the fact that Thomas left it to last says more about the overall attitude of
developers to testing.
Thomas’s list is more useful and instructive. Useful in that the tips provided are worthwhile, common
strategies and instructive in that we would argue that the first five approaches fall into the “don’t bother
me with defects, I’m a developer” category. We will dig deeper into strategy #6 when we consider best
practices.
How many defects are too many?
For the answer to this question, we recommend reference to an internal or external benchmark for
appropriate levels of defects based on the functional size of the application, the technology and process
used and the team capability. For example, a 1000 function point application in Java might be expected
to have a range from 32 to 61 defects based on industry data benchmarks. DCG is one source of such
benchmark data.
If for some reason, you cannot get access to benchmark data, Ian Watts offers the following approach:
1. When asked what the threshold for number of defects each sprint team should carry, the first
thought is zero. However this is impractical, so I started to think about how we should figure
out the size that was appropriate.
©2014 David Consulting Group
Page 3 of 6
2. If you think about agile you will realize the goal is to be able to stop what you are doing at the
end of a sprint and ship the product. So given this goal the defect backlog must always remain
low enough that your team can fix and verify the defects within a very short period of time.
3. In my case I desired 1 week (5d) was all I would allow them to have.
4. With this data point I now figured out the correct "fix rate" of a typical developer. On our team
that is about 1.5 defects per day.
5. The next data point is how long it takes to verify the defects. In one day, 2 testers can likely
verify all the defects the developers have fixed. Add to this that all the code the developers fix
has to get into a nightly build to be verified and we take the 5day - 1 day to equal the number of
days the development team has to fix defects.
6. So with these data points we can now figure out the "Max" backlog our team should have:
Max Backlog = (4d) * 1.5(defects per day) * (number of developers)
7. Typical scrum team has 7 people, with 4 developers, so...
Max Backlog = 4*1.5*4= 24 defects
Defect Management Best Practices
In these days of “big data”, it’s important to remember that, “there’s gold in them thar hills!” More
prosaically, there could be important information buried in the Software Defect Backlog that could not
only reduce its size but lead to defect prevention strategies that could easily pay back the time taken for
some causal analysis.
Defect Tracking Details for Causal Analysis
Effective causal analysis depends on having sufficient information in the defect backlog to be able to
identify batches of similar defects, trends or patterns. Our “Defect Tracking Best Practices” source (see
below) recommends the following Defect Report Content:
 Originator / Origination Date
 Defect Description
 Sequence of Events
 Supporting Data
 Defect Lifecycle History
 Subsystem or Domain
Kan recommends recording the following key attributes:
 Activity – The specific activity that exposed the defect. For example, during system test, a defect
occurs when one clicks a button to select a printer. The phase is system test but the activity is
function test because the defect surfaced by performing a function-test type activity.
 Trigger – The environment or condition that had to exist for the defect to surface.
 Impact – This refers to the effect the defect had on the customer in the field (or would have had
if it had escaped to the field).
Causal Analysis
1. Create a “Possible Cause” List or Chart. Gilb and Grady contain examples but the Fishbone
diagram form Grady reproduced here is a good illustration:
©2014 David Consulting Group
Page 4 of 6
Guidelines Not Followed
Lack of Feedback
Int. customers
decide based on
functionality
No time
Don’t read them
Prototype
not enough
Lack of Resources
Some product parts
don’t resemble others
Some panels not
used as much
No good model
Tests don’t model Users
environment adequately
Too many combinations of features
Corner cases, particular problems
Too busy to do them
Users not focused
On new product
No process
To provide
Early feedback
No Central Location
Lack of Guidelines
USER INTERFACE
Too many details
Decided based on
Incomplete
Customer base
Result of Changes
In a hurry
Resource limits
Different perspectives
OOPS (Forgotten)
Source: Grady
2. Select the Defects to be Analyzed. As Gilb and Graham note, “The defect rates during software
development are generally such that most development organizations cannot analyze every
defect, nor is it usually worthwhile analyzing them all.” They suggest the following selection
strategies (the order is our recommendation):



Group similar or related defects together into subsets. Pick a typical defect from each subset
to do causal analysis.
Analyze every severity 1 defect and a balanced subset of other defects. Do not analyze
severity 4 defects.
Select every fifth Major defect. Eliminate any obvious duplicates but make a note that that
particular type of defect occurred multiple times. Do not analyze Minor defects.
3. Convene a root cause analysis meeting. Two or more heads are always better than one on this.
Involving representative developers facilitates learning for all parties.
4. Capture findings in code inspection and development checklist updates helps to ensure that
today’s mistakes are not repeated tomorrow.
Static Analysis Tools
We agree with Todd Landry’s assertion that a static analysis of your code is a great way to help
categorize and prioritize defects in the code. When you run a static analysis product on your code base
for the very first time there is a good chance the first analysis is going to list a large number of defects,
some that are without question real and on your defect backlog, some that are real and not on your
defect backlog and some that perhaps are not real at all. Do not freak out! This is the first time that
analysis engine has ‘laid eyes’ upon your code and it is going to flex its muscles and show you any
weaknesses it believes exist. So how does one deal with this? Landry suggests a few strategies to help:
©2014 David Consulting Group
Page 5 of 6
1. Don’t boil the ocean. Before you even run that first analysis, don’t have a “wouldn’t it be cool”
moment, where you decide to turn on every single rule the analysis engine has. There is a reason
why static analysis tools haven’t turned on everything. They are showing the most accurate and
critical issues first. So unless you have unlimited time and resources, your best bet is to start
with a core set of rules and run the analysis based on that set.
2. Baseline your defects. Consider that first analysis your baseline and choose to ‘park’ them for the
time being. Chances are the product that the analysis was run on is one that has already been
released to the public, and in good working order. Zero out these defects for now, and start to
triage them, which leads into strategy #3.
3. This is going to sound pretty obvious, but when it comes to managing your issue backlog start
looking at the most critical issues first. These are the ones that are most likely to cause a failure
of some sort, so determine if these issues are real, and if so, fix them immediately. Once you’re
done with the most critical issues, move to the next level of severity, and continue on that way.
4. Finally, tune your analysis. Any good vendor will allow you to tune your analysis. The benefits of
tuning are twofold; 1) you can find code issues that would otherwise go undetected and, 2)
reduce the number of issues that the engine reports incorrectly in the context of your source
code. You should think of ways to give the tool more “context” about your code base to increase
accuracy.
Conclusion
There is no way to know if a particular Software Defect Backlog for a particular application or product
matters without further analysis of the detail of the defect backlog. Ideally, this analysis should be
regular and frequent. There is value in both identifying patterns of development or areas of code that
generate the most defects and in removing the spurious reporting of low impact defects that waste
analysis effort and deter organizations from tackling the really “valuable” and “useful” defects in the
backlog.
Sources
1.
2.
“Successful Software Process Improvement”, Robert B. Grady, Prentice Hall, 1997
3.
4.
5.
6.
7.
“Software Inspection”, Tom Gilb & Dorothy Graham, Addison Wesley, 1993
“IT Measurement”, International Function Point Users Group (Various Authors), Addison Wesley, 2002
“Project Management Terms” by J. Leroy Ward, ESI International, 2000.
“Software Testing Fundamentals” Blog post http://softwaretestingfundamentals.com/defect-severity/
“It’s a Delivery Thing” Blog post by Steven Thomas http://itsadeliverything.com/nine-strategies-formanaging-a-growing-defect-backlog
8.
“Kloctalk” blog post by Todd Landry http://www.klocwork.com/blog/nasty-bugs/dealing-with-adifferent-type-of-backlog-your-bug-backlog/
“Metrics and Models in Software Quality Engineering”, Stephen H. Kan, Addison-Wesley
Professional, 2003
9.
“developerWorks” blog post by Ian Watts
https://www.ibm.com/developerworks/community/blogs/TestAutomationForRest/entry/defect_backlogs
_how_to_know_when_you_have_to_many14?lang=en
10. “Defect Tracking Best Practices”, Author unknown,
http://www.stickyminds.com/sites/default/files/article/file/2013/XDD14942filelistfilename1_0.pdf
©2014 David Consulting Group
Page 6 of 6