SQA – SWE 333
Software Quality Assurance
Prof. Dr. Mohamed BATOUHE
Dept. of Software Engineering
CCIS – King Saud University
1
What is CMM ??
2
Why CMM matters …
 It is the most widespread and detailed software
development model
 It is a standard for much DoD work, which is a lot of
software projects
 It is being used by many non-DoD businesses
 It is widely criticized, and has inspired several anti-
CMM models
3
History …
 Begun in 1986 by DoD to help improve government software contractors.
 Work started at Mitre, then at Software Engineering Institute (SEI) at Carnegie
Mellon Univ.
 Watts Humphrey was initial author, then Mark Paulk, Bill Curtis, and others.
 Borrows heavily from general Total Quality Management (TQM) and work of Philip
Crosby.
 Under active development for 15+ years, ongoing…
 Has gained significant interest among non-DoD software vendors.
 All documents are public, and many are free.
 Known as SW-CMM or CMM.
4
SW Development and Football
 What happens when a ball is hit to a Little League
team?
 Everyone runs around at random.
 They might do the right thing, or they might not.
 The next time the ball is hit in the same place, they may do
something different.
 What happens when a ball is hit to a professional
team?
 Everyone moves in a coordinated fashion, based on practicing
that play many times.
 Sometimes they fail to make the right play, but they almost
always try to do the right thing.
5
SW Development and Football
 A professional football team is more "mature" than a
Little League team (not referring to age).
 A professional team has self-perpetuating quality.
They
 Make good plays
 Develop new players like themselves
 Find ways to make better plays
6
SW Development and Football
 What happens when the team loses a star player?
 Little League team gets much worse.
 Professional team often has someone waiting to fill in.
 Self-improvement after a bad play…
 Little League players don’t know what went wrong, or
they blame each other.
 Professional teams discuss their play and look for ways
to improve.
7
… So what is CMM?
 In the same way, high-quality SW organizations are
different from low-quality orgs.
 CMM tries to capture and describe these differences.
 CMM strives to create software development organizations
that are “mature”, or more mature than before applying
CMM.
8
… So what is CMM?
 While awarding a contract, the customer needs to gain confidence that an organization is
capable of delivering the desired product.
 The US Department of Defense wanted to evaluate their contractors.
 They needed a framework to evaluate the maturity of software processes.
 In 1986, the Software Engineering Institute (SEI) initiated the development of a framework to be
called the CMM.
 In the CMM model, the maturity level of an organization tells us to what extent an
organization can produce low cost, high quality software.
 Having known the current maturity level, an organization can work to reach the next higher
level.
 There are five levels of SW process maturity in the CMM model.
 Includes lots of detail about each level.
9
CMM in details – Five levels of maturity
10
Five levels of maturity …
11
Summary of levels
 Level 1 – Initial. Anything at all. Ad-hoc and
chaotic. Will have some successes, but will also
have failures and badly missed deadlines.
 Level 2 – Repeatable. SW processes are defined,
documented, practiced, and people are trained in
them. Groups across an organization may use
different processes.
12
Summary of levels
 Level 3 – Defined. SW processes are consistent and
known across the whole organization.
 Level 4 – Managed. SW processes and results are
measured quantitatively, and processes are
evaluated with this data.
 Level 5 – Optimizing. Continuous process
improvement. Experimenting with new methods
and technologies. Change processes when find
something that works better.
13
Level 1 – Initial
 Team tackles projects in different ways each time
 Can have strong successes, but may not repeat
 Some time/cost estimates are accurate, many far off
 Success comes from smart people doing the right things
 Hard to recover from good people leaving
 Frequent crises and "firefighting.” (Many believe this is standard
for SW development. CMM says NO.)
 Most SW development organizations are Level 1.
14
Level 2 – Repeatable
 Key areas
 Requirements management
 Software project planning
 Project tracking and oversight
 Subcontracts management
 Quality assurance
 Configuration management
15
Level 3 – Defined
 Key areas. Level 2, plus…
 Organization-wide process focus
 Organization-wide process definition
 Training program in above
 Integrated software management (above applied per
project)
 Software product engineering (coding, etc.)
 Inter-group coordination
 Peer reviews
16
Level 4 – Managed
 Key areas. Level 3, plus…
 Quantitative process management (data gathering)
 Quality management (data-driven quality
improvement)
17
Level 5 – Optimizing
 Key areas. Level 4, plus…
 Defect prevention
 Technology change management (bring in new
methods)
 Process change management (improve processes)
18
Structure of the Levels
 Main idea: Saying you will do something is not enough.
19
Time required to progress to next level
Capability level
transition
Level 1 to level 2
Mean time
(months)
24
No. of
organizations
125
Level 2 to level 3
21.5
124
Level 3 to level 4
33
18
Level 4 to level 5
18
19
Source: Based on Gartner Inc. (2001)
20
Project Resources distribution
Percentage of project resources
CMM capability
level
Original work
Reworking
Testing and quality
assurance
1
34
41
25
2
55
18
27
3
67
11
22
4
76
7
17
21
Observation:
 CMM contains a lot of verbiage.
 Can seem like just a word game, which it might be to
some extent.
 If you get beyond the verbiage however, CMM does
describe some important methods for running
software projects.
22
Problems with CMM
 It is a goal, not a method
 Being used just as stamp of approval
 Doesn’t say anything about software!
 Doesn’t help in a crisis
 Only for repetitive tasks
23
CMM is a goal, not a method
 Organizations often look to CMM as a method or formula
for improvement
 Tempting to want easy answers.
 CMM is actually a management framework, with many
details left out
 Example: “You must have peer reviews.” But how should the reviews
be run?
 If you say, "we do CMM just like the book", you aren't doing
CMM
 To use CMM, you have to think
 You must be flexible, be creative, and integrate the goals of CMM
with the realities of your business.
24
CMM is a goal, not a method
 Related to this point, some organizations are rigid
about mandating CMM to their employees
 This has given CMM a bad reputation as an onerous,
inflexible method
 The problem here (usually) is misunderstanding
CMM, not its rigidity
25
Becoming stamp of approval
 Some organizations want CMM only as a stamp of
approval, without a high-level commitment to process
improvement or quality
 Want to find easiest way to get certified as Level 2 (or
3) without really changing
 A CMM assessor was tired and disillusioned because:
 She wanted companies to say, “Let’s work together to improve
our software processes.”
 Instead, they say, “Just tell us what to do to get Level 2, so we
can get back to work."
26
Not about software!
 CMM is, by design, a model for managing software
projects
 They claim most software failures are due to management
problems rather than technical barriers (I agree)
 But CMM goes too far in this direction
 An organization could write lousy code (consistently)
and be rated highly by CMM
 This is counter-intuitive, since good code is the goal of
software development
27
Does not help in a crisis
 CMM does not help projects that are in crisis right
now
 Trying to apply it then would make things worse
 Unfortunately, this is often when companies look for
help with their software processes
 Analogy: For long-term cardiac health…
 Eat lots of fruits and veggies, quit smoking, get exercise,
reduce stress
 But suppose you are having a heart attack now. Will
these actions help?
 It is too late for preventive measures. You need some other
kind of intervention.
28
Only for repetitive tasks
 CMM is based on re-using past results for future
software projects
 In management activities, quality measurements,
development processes
 But, this only makes sense for relatively repetitive
projects
 Example: MS Word team creating V7 after V1-6
 To create brand-new software of unknown size,
with unknown hurdles, using human creativity,
CMM is clearly not the right model.
29
“Maturity suite” (OPT)
 Personal Software Process (PSP)
 Team Software Process (TSP)
 People CMM (P-CMM)
 Software Acquisition CMM (SA-CMM)
 System Engineering CMM (SE-CMM)
 Integrated Product Development CMM (IPD-CMM)
 CMM Integration (CMMI): SW + SE + IPD
30
Versions of CMMI
(capability maturity model integration)
 CMMI-SE/SW
 System Engineering CMM (SE-CMM)
 Software engineering CMM (SW-CMM)
 CMMI-SE/SW/IPPD/SS




System Engineering CMM (SE-CMM)
Software engineering CMM (SW-CMM)
Integrated Product/Process Development (IPPD-CMM)
Supplier Sourcing
 CMMI-SE/SW/IPPD
 System Engineering CMM (SE-CMM)
 Software engineering CMM (SW-CMM)
 Integrated Product/Process Development (IPPD-CMM)
31
ISO 9000 vs. Capability Maturity Molel
32
ISO 9000/9001 and CMM
Definition
ISO 9000/9001
CMM
A set of documents
dealing with quality
systems that can be used
for external quality
assurance purposes.
The Capability Maturity
Model describes the
principles and practices
underlying software
process maturity and
intended to help
software organizations
improve the maturity of
their software process.
ISO 9000/9001 and CMM
ISO 9000/9001
CMM
Development
By The International
Standards Organization
By The Software
Engineering Institute
Written for
Wide range of industry
Software industry
Documents
Abstract
Detailed
ISO 9000/9001 and CMM
ISO 9000/9001
CMM
Pages long
Only 5 pages
Over 500 pages
Concept
To follow a set of
standards to make
success repeatable
To emphasize on
achieving and improving
its process continuously
Documents
On the
customer/supplier
relationship to reduce a
customer’s risk in
choosing a supplier.
The supplier to improve
the internal software
process.
ISO 9000/9001 and CMM
Management
Responsibility
ISO 9000/9001
CMM
Quality policy is
defined, documented,
understood,
implemented, and
maintained; that
responsibilities and
authorities for all
personnel specifying,
achieving, and
monitoring quality be
defined.
Quality policy is
primarily addressed in
Software Quality
Assurance.
ISO 9000/9001 and CMM
ISO 9000/9001
Quality System
CMM
Procedures are addresses
and assured in Software
Requires documented
Quality Assurance.
quality system, included
Software product
procedures and
Engineering tasks should
instructions, be
be defined, integrated,
established.
and consistently
performed.
ISO 9000/9001 and CMM
ISO 9000/9001
Contract Review
CMM
Contracts are reviewed
to determine whether the
Documented and
requirements are
reviewed and the
adequately defined,
agree with the bid, and missing requirements are
clarified.
can be implemented.
ISO 9000/9001 and CMM
ISO 9000/9001
Design Control
CMM
Requires that procedures
to control and verify the
The life cycle activities
design be established.
on requirements
This includes planning
analysis, design, code,
design activities,
and test are described in
identifying inputs and
Software Product
outputs, verifying the
Engineering.
design, and controlling
design changes.
ISO 9000/9001 and CMM
ISO 9000/9001
Document
Control
CMM
The configuration
Requires that the
management practices
distribution and
characterizing document
modification of
control are described in
documents be controlled. Software Configuration
Management.
ISO 9000/9001 and CMM
ISO 9000/9001
Purchasing
Requires that purchased
products conform to
their specified
requirements. This
includes the assessment
of potential
subcontractors and
verification of purchased
products.
CMM
This is addressed in
Software Subcontract
Management.
ISO 9000/9001 and CMM
PurchaserSupplied
Product
ISO 9000/9001
CMM
Requires that any
purchaser-supplied
material be verified and
maintained.
Describing the use of
purchased software. It
does so in the context of
identifying off-the-shelf
or
reusable software as part
of planning.
ISO 9000/9001 and CMM
ISO 9000/9001
Product
Identification
and Trace
ability
CMM
Software Product
Requires that the product
Engineering states the
be identified and
specific need for
traceable during all
consistency and trace
stages of production,
ability between software
delivery, and installation.
work products.
ISO 9000/9001 and CMM
ISO 9000/9001
Process Control
Servicing
CMM
Requires that production The procedures defining
processes be defined and the software production
planned. This includes
process are distributed
carrying out production
throughout the key
under controlled
process areas in the
conditions, according to
various Activities
documented instructions.
Performed practices.
Requires that servicing
activities be performed
as specified.
Intended to be applied in
both the software
development and
maintenance
environments,
ISO 9000/9001 and CMM
Training
ISO 9000/9001
CMM
Requires that training
needs be identified and
that training be
provided, since selected
tasks may require
qualified personnel.
Records of training are
maintained.
Identified in the training
and orientation practices
in the Ability to Perform
common feature.
ISO 9000/9001 and CMM
Quality Records
ISO 9000/9001
CMM
Requires that quality
records be collected,
maintained, and
dispositioned.
Specifically pertinent to
this clause are the testing
and peer review
practices in Software
Product
Engineering.
Companies have ISO 9001 certification
AIR LIQUIDE
ALLIED SIGNAL AEROSPACE
AMERICAN PACIFIC
CORPORATION
AMERICAN TANK &
FABRICATING CO.
APPLIED MATERIALS, INC.
ARCO CHEMICAL COMPANY
ASHLAND CHEMICAL CO.
ASME INTERNATIONAL
ASQ
AT&T
BABCOCK & WILCOX
BACARDI-MARTINI CANADA,
INC
BAKER HUGHES MINING TOOLS,
INC.
BARCLAYS BANK, PLC.
BASF CORPORATION
BAXTER DIAGNOSTICS, INC.
BAYER CORPORATION
BECKMAN INSTRUMENTS
BELL SOUTH
TELECOMMUNICATIONS, INC.
BELL TECHNOLOGIES
BOEING
BOEING AEROSPACE
OPERATIONS
BOSE CORPORATION
Companies have ISO 9001 certification
BRISTOL- MYERS SQUIBB
BRITISH STANDARDS
INSTITUTION, INC.
BURLINGTON PERFORMANCE
WEAR
CAMPBELL SOUP COMPANY
CANADA GENERAL STANDARDS
BOARD
CARRIER CORPORATION
CATERPILLAR, INC.
CHRYSLER CORPORATION
CIBA VISION
CREATIVE LABS
CUTLER & HAMMER
DAIMLER CHRYSLER
DIEBOLD, INC.
DOW CHEMICAL
DRAVO
DUNLOP TIRE CORPORATION
DURACELL
E.I. DUPONT
EASTMAN KODAK
EATON CORPORATION
ENTELA, INC., QSRD
ERICSSON, INC.
ESSILOR OF AMERICA, INC.
ESTEE LAUDER COMPANIES
EXXON CHEMICAL COMPANY
Companies have ISO 9001 certification
EXXON COMPANY, INC.
FISERV
FORD
FUJITSU COMPUTER PRODUCTS
GE AIRCRAFT ENGINES
GILLETTE
GOODYEAR TIRE
GOODYEAR TIRE & RUBBER
COMPANY
HARBISON-WALKER
REFRACTORIES
HASBRO USA GAME MFG.
HEWLETT PACKARD
HYUNDAI ELECTRONICS
AMERICA
IBM
INGERSOLL-RAND INTL. SALES
INC
KOHLER
KRUPP WERNER & PFLEIDERER
CORPORATION
LOCKHEED MARTIN AIR
TRAFFIC MANAGEMENT
LOCKHEED MARTIN CANADA
LOCKHEED MARTIN ENERGY
SYSTEMS
References
 Williaw E. Lewis, “Software Testing And Continuous Quality Improvement”, Third Edition,
CRC Press, 2009.
 K. Naik and P. Tripathy: “Software Testing and Quality Assurance”, Wiley, 2008.
 Ian Sommerville, Software Engineering, 8th edition, 2006.
 Aditya P. Mathur,“Foundations of Software Testing”, Pearson Education, 2009.
 D. Galin, “Software Quality Assurance: From Theory to Implementation”, Pearson Education,
2004
 David Gustafson, “Theory and Problems of Software Engineering”, Schaum’s Outline Series,
McGRAW-HILL, 2002.
 Michael R. Liu, Handbook of Software Reliability Engineering, IEEE Computer Society Press,
McGraw-Hill, 2005.
50
51