defect
Ross Smith
Director of Engineering
Skype Customer Feedback
Everyone is responsible for
Quality
Focused on creating a quality experience
Don’t sacrifice quality for the sake of schedule
Ensure a robust feedback loop for users to tell us what’s missing
Defining Quality?
• The perception of the degree to which the product or service meets customer expectations
•
•
•
•
•
•
•
ISO 9000: Degree to which a set of inherent characteristics fulfills requirements.
Six Sigma: Number of defects per million opportunities
Philip B. Crosby: Conformance to requirements
Joseph M. Juran: Fitness for use by the customer
Gerald M. Weinberg: Value to some person
Robert Pirsig: The result of care
American Society for Quality: A subjective term for which each person has his or her own
definition. In technical usage, quality can have two meanings:
• a. The characteristics of a product or service that bear on its ability to satisfy stated or implied needs;
• b. A product or service free of deficiencies."
testing and
quality
Compatibility
Performance
Privacy
Readiness
Reliability
Security
Support
Usability
Value
Software Development
diminishing
returns
10%
10%
10%
10%
10%
10%
10%
10%
10%
10%
portfolio
selection
theory
efficient
frontier
james
Person vs System
The person approach focuses on the errors of
individuals, and often results in blaming them for
forgetfulness, inattention, or moral weakness
The system approach concentrates on the conditions
under which individuals work and tries to build
defences to avert errors or mitigate their effects
Swiss
Cheese
Model
defect
prevention
Root Cause Analysis
FMEA – Failure Modeling
Orthogonal Defect Classification
Defect Taxonomy
Prevention Knowledge Collection
Learn from our mistakes
Our products are highly
technical and having a specialist
analyze can often lead to
findings otherwise unattainable.
Ultimate Goal:
Specific Recommendations
Why
Root
Cause
Analysis
A careful look at “undesirable events”
Helps identify what, how, and why
something happened
Causes are underlying and reasonably
identifiable at varying levels
A cause is just an effect from a
different perspective
What is
Root
Cause
Analysis
Data Collection
Analysis
Classification
Causal Charting
Recommendations
Implementation
How
Bug Database
Source Control
Crash data
Telemetry
Targeted Surveys
Collaborative Analysis
Experts Group(s)
Interviews
Data
Collection
5 Why’s
Causal Charting
Classification
Defect Taxonomy
Pareto
Data
Analysis
Once the bug has been fixed and the
cause understood,
Developer – “how did I write this?”
tester “Why did I miss this?”
didn’t know it was possible
Unfamiliar with customer environment
Lack of clear spec
Didn’t do XYZ testing here
Didn’t I know it was possible?
Assumed certain conditions
Why was I unfamiliar with customer
environment
Didn’t research
Lack of clear spec to detail possible
configs
Five
Whys
Need to better understand the
customer environment
But that is so high level that it’s not
really actionable.
Recommendations are S.M.A.R.T.
Specific, Measurable,
Actionable/Achievable, Relevant,
Time-bound
Need check in tests on different
environments
Recommendations
Scope of Recommendations
•Training
•Process change (includes addition or deletion)
•Process automation – If a manual process is error-prone,
automate it
•Detection tools – Detect the problem upstream
•Prevention tools – Prevent the problem from occurring at all
•Focus change – Are we not doing something we should, are we
doing something incorrectly, is something we do too timeconsuming?
•Focused information on problem– avoid information overload
(too much in too many places).
Most difficult part of any RCA Study
Metrics to show improvement
Implementation
RCA is great, but expensive
Orthogonal Defect Classification are
more efficient, but still require heavy
analysis
FMEA requires an up-front
investment, which we often aren’t
willing to make
Defect Prevention can be a
lightweight answer to get proof-ofconcept required to justify more
investment.
Defect
Prevention
Work
Collect data at the point of
experience – when the bug is fixed
If a bug is preventable, the dev will
likely know the prevention technique.
Many bugs are preventable with the
exact same techniques.
Human error is consistent
.
Collecting
Prevention
Knowledge
Collecting Prevention information at the peak
knowledge – when being fixed
Better Design Review
Better Coding Practices
Better Code Review
Static Analysis tools
More Testing
Better Testing
.
Suggestion
Affinity
at some level, every bug is a
human error
thank you!
defectprevention.org
42projects.org
[email protected]
/42projects
@42projects
rosss42
References
www.defectprevention.org
www.42projects.org
Human Error - http://bmj.bmjjournals.com/cgi/reprint/320/7237/768.pdf
James Reason Human Error http://www.amazon.com/HumanError-James-Reason/dp/0521314194
FMEA
http://www.sematech.org/docubase/document/0963beng.pdf
ODC http://www.issre2009.org/papers/issre2009_201.pdf
Portfolio Theory
https://en.wikipedia.org/wiki/Modern_portfolio_theory