Cleanroom Software Engineering
Andy Moyer
Software Engineering
University of Wisconsin, Platteville
[email protected]
Abstract
Cleanroom Software Engineering is a process which aims to achieve or approach zero defects
with statistically provable quality. The idea came from the manufacturing of semiconductors. In
Cleanroom development, correctness and verification replaces unit testing and debugging. When
the coding is completed, the system test is run without any debugging. The process is built upon
the incremental process to create small, easily solvable problems that eventually add up to solve
the main problem. The Cleanroom process focuses on the design of the program rather than
debugging code; focusing on bug prevention rather than bug detection. Through the core
techniques the code is written, reviewed, and revised before ever being compiled or debugged.
Introduction
There are many processes developed for the production of software; all of which aim to improve
reliability and quality. The Cleanroom Software Engineering process is no exception. The
process borrows the idea from production of electronic components such as semiconductors;
which are produced in rooms which are under strict control to keep potential defects, such as
dust, from damaging the product during production. The Cleanroom process aims to keep all or
nearly all defects out of the code before building and running of code. While the ideal of the
process sounds appealing, the process itself is involved and expensive upfront [1].
What Is Cleanroom Software Engineering?
In order to begin to understand how Cleanroom Software Engineering is performed, it must first
be understood what Cleanroom is. In its most simple definition, and like any other software
development process, Cleanroom Software Engineering is a set of principles and practices for the
specification, development, and certification of software-intensive systems [2]. Cleanroom is a
theory-based, team oriented process developed by Harlan Mills in the late 1970's and early
1980's. Harlan Mills was an employee of IBM during the time he developed the process.
Influenced by Edsger Dijkstra on structured programming, Nicholas Wirth on stepwise
refinement, and David Parnas on modular design, Mills developed a process and scientific
foundations for Cleanroom [3]. Mills was also influenced by two fundamental principles: first is
that programs are rules for mathematical functions and second, there are an infinite set of ways
programs can execute and require statistical sampling for testing [3].
Goal of the Cleanroom Process
The goals of Cleanroom Software Engineering are very ambitious. It aims to achieve or
approach zero defects. This is a daunting task and the basis for the process. To achieve this
goal, the Cleanroom process focuses almost entirely on defect prevention by creating correct,
verifiable software using an incremental development process [2].
Properties of Cleanroom
Like any other process, Cleanroom has a set of properties that set it apart from the rest and define
it as Cleanroom. One of the first properties is also an important consideration in choosing any
process, and that is the cost. Cleanroom is a very pricy process 'up-front'. A developer must
spend a lot of time and money in the defect prevention of every stage in the lifecycle. The
biggest time consumer of the process is the use of statistical methods to ensure quality.
Beginning with the specification of requirements all the way to the testing, a formal proof is used
to ensure no defects are injected [1].
Like any process, it is heavily team oriented. What makes Cleanroom a little different is that
general specification, development, and certification are done by separate small teams, about
three to eight people, each using the incremental model. The teams are generally small to
minimize coordination and simplify communication. Depending on the size of the project
however, the size and number of teams may be different. Small scale projects will be able to get
by with a single team for all three areas. For medium scale projects, an initial team can be
formed and as the project progresses, new teams can be formed from the original teams to break
the project down as necessary. For large scale projects however, it is necessary to form each of
the teams from the beginning [3].
One key idea is that the development team has no access to compilers. The development team
must design and code the entire system without ever running or even compiling the code
themselves. They rely heavily on code reviews to find bugs. They will read over code, line by
line, looking for syntax errors and proving the code correct. Once they believe the code is
correct, the code is then passed onto the verification team and they will build and run the code
and return any feedback of bugs found to the development team [3].
The Cleanroom Process
The Cleanroom process defines four stages of development in an effort to achieve its goals;
incremental development under statistical process control, function-based specification design
and verification, correctness verification, and statistical testing and software verification. Each
stage has a unique ability to be used separately or together and can be introduced in any order
[3].
Incremental Development Under Statistical Process Control
Based on the engineering principle of controlled iteration in product development, this stage
focuses on incremental development in a series of smaller, cumulative steps. Each pass of each
increment builds on the previous pass as well as everything from the previous pass. The major
benefit from approaching a problem in this manner is that it is a lot easier logistically to control
small portions of a large problem. It is sound practice to break the larger problem at hand into
smaller more manageable problems in order to reduce the inherent risks associated with
managing large problems [3].
The incremental development brings an organized and orderly sequence of development cycles.
Each step goes through many phases of the software life cycle. However, one of the most
important pieces is customer visibility. At the end of each increment, some amount of end-user
functionality should be available for the customer to see. When the customer can see their
system slowly build up they can give direct immediate feedback about the functionality
implemented. This is helpful because the customer is not overwhelmed with a completed system
where it may be hard to find or even understand some of the functionality. With feedback at
each increment the system has a better chance of becoming what the customer expects. The
major condition for this to work is that once a function is approved by the customer, it absolutely
must not change through successive iterations [3].
Function-Based Specification, Design, and Verification
The objectives behind this are to expand the specification into implementation via small steps
and to maintain intellectual control by designing data and control structures at appropriate levels
of abstraction. Specification begins with the external view better known as the black box view.
This is expanded upon and becomes a state machine view or the state box. The state box is
further refined into the clear box where procedure is defined. The three boxes are collectively
called the box structure in Cleanroom. The boxes are used to perform stepwise refinement of the
system from specification to the code. The first step is to expand the high level black box to a
state box and then to clear box. The clear box is then further refined into more black boxes.
This is repeated until the system can be broken up into manageable size problems. While
breaking the system down farther, each box and each level expanded is verified for correctness
[3].
The Black Box
The black box specifies the required behavior of the system by mapping the inputs to the
outputs. None of the internal behavior is mentioned at this point, only the view of the system
from the user stand point. The user is not specifically a person, but may also be a hardware unit
or even another program. The black box is a state free, procedure free representation of a
function and must be complete, consistent, and traceably correct. The system is usually formally
specified using formal specifications [3].
The State Box
The state box is a derivative of the black box and is the first step toward implementation. This
shows some of the data implementation but hides process implementation. It describes how an
input is transformed in each state of the system to become the output. The state box is
represented as a finite state machine [3].
The Clear Box
The clear box is a derivative of the state box. This is where the data and process implementation
are represented. The goal is to stepwise refine procedures and prove them correct. This means
that even though the clear box is the final derivative in the box structure, new black boxes can be
found and then further derived to simplify some of the complex functions. Clear boxes have
flexibility to be defined in either design languages or the target language for the system [3].
Correctness Verification
Correctness verification is the primary debugging process in the Cleanroom process. As stated
above, each box structure is subject to correctness verification. It aims to determine that the box
structures correctly implement the design. It attempts to remove any errors that were introduced
during development. Through formal methods, it completely reviews the code for completeness,
consistency, and correctness [3].
Not only is each box structure verified but each step in the incremental process is verified. The
step is verified against the requirements and also with the previous step to ensure that the
functionality that has been developed, proven, and approved by the customer remains unchanged
after making new additions to the system [3].
It's important to understand not only what kinds of things are put through verification but also
what is being verified. The verifiers are looking if the box structures behave as defined, the
correctness of each refinement, and if the response mapping defined in a previous step is
preserved in the current step [3].
Statistical Testing and Software Verification
Cleanroom process testing methods are based off of the idea that when a larger system has a
population model that is too large to logically test every case, sampling must be used. During
this stage, a usage model is developed to represent the population of all possible system uses.
Below is an example of a simple usage diagram. They define the states of the system, which
state the system can go to from the current state, and the probability the system will go to that
state.
Figure 1: Example of a Usage Model [3]
When the usage model is developed, it has high reuse throughout the process. To begin with,
test cases can be generated based off of the usage model. These models allow themselves to be
expanded upon as well, thus allowing the test cases to be expanded. The system is not limited to
only one usage model either, and generally multiple models are created. Once the usage models
and test cases are defined, each increment of compilable functionality is statistically tests to
ensure that it meets quality standards, certify a level of reliability that the product will deliver in
the field, and provide feedback for quality control and process improvement [3].
The Cleanroom outlines three steps that are taken for statistical testing and verification. The first
step is to plan the test. This is where a stratification plan which describes each layer to be
developed. Also the sampling plan is outlined in which the stimuli are chosen to best represent
the system. Finally a description of which physical testing resources are devoted to each
increment of the Cleanroom model. The second step is to develop usage models required by the
test plan. Finally, the last step is to develop and validate the usage distribution.
Case Studies
Several case studies have been performed to evaluate the Cleanroom process. The information
below describes the setup and results of two case studies. The first case study was performed by
NASA and the second case study was performed by the Department of Defense.
NASA Case Study
In March of 1990 NASA performed a study on the Cleanroom process and compared it to their
typical process. Their objective was to analyze the application on a large ground support system
that was used in support of an attitude determination requirements on the Coarse/Fine Attitude
Determination Subsystem (CFADS) of the Upper Atmosphere Research Satellite (UARS). The
completed code contained roughly 34,000 lines of FORTRAN code [4].
Figure 2: Cleanroom Process used in NASA Case Study [4]
NASA split the development processes into two separate teams each developing code from the same set
of requirements. One of the more interesting aspects of the team construction was the experience
difference. As seen below in Figure 3, the average experience of the application for the typical process
managers have about .8 years more experience than the managers of the Cleanroom process and the
technical staff has about .2 years more experience on the typical process compared to the Cleanroom
process team. Also the managers on the typical process have two times more experience overall than the
managers in the Cleanroom process and the technical staff on the typical process have more than double
the overall experience than the Cleanroom process team [4].
Figure 3: Experience comparison of the typical process team to the Cleanroom Process team [4]
NASA Case Study Results
Despite the clear experience difference, there were some very good data exposed in favor of the
Cleanroom process. To begin, the program developed using the typical process had a failure rate
of about six errors per thousand lines of code. The Cleanroom process program had almost half
as many failures with about 3.3 errors per thousand lines of code. It is pretty evident that the
experience difference did not affect the outcome of the results. Even though the Cleanroom
process team had less experience they were still able to produce a more quality product. NASA
concluded that one of the key differences was that in the Cleanroom process, the impact of
unstable requirements were lessened by concentrated efforts in team reviews [4].
Also, below in Figure 4 is a good representation of how much time is needed upfront in the
Cleanroom process. The Cleanroom process team did not start coding until 50% of the way
through the project calendar. The typical process team had started at about 25%. However the
Cleanroom process was able to develop the code much faster, and consistently as well as
producing higher quality [4].
Figure 4: Timeline of project development of both processes
Figure 4: Timeline of project development of both processes [4]
Department of Defense Case Study
The Department of Defense also performed a case study using a research and development
program known as the STARS program. Their goals were to attempt to stretch the life of old
systems by re-engineering and enhancing software [1].
DoD Case Study Results
The DoD, like NASA, used two separate teams to attempt the same problem. They had
measured a few metrics, the productivity measured in lines of code per person month, and the
failure rate, which are just a couple to look at. The typical process productivity was about 121
lines of code per person month and the failure rate was 23~27 failures per thousand lines of code.
The Cleanroom process was almost five times more productive measured at 559 lines of code per
person month and the failure rate was only one failure per thousand lines of code. Again the
results favor the Cleanroom process showing a more productive development as well as a more
quality product [1].
A Critique
In 1997, Beizer challenged some of the claims made by supporters of Cleanroom, in particular
the elimination of unit testing. He argues that this contradicts known testing theory and common
sense as that it is impossible to find bugs without actually compiling code [1].
Conclusion
So in conclusion, the Cleanroom process is a heavily formal proof based process aiming to
reduce the amount of defects to zero or near zero. The four processes of the Cleanroom model
define a sound basis for almost any kind of project. Through the case studies, it has shown
potential for the Cleanroom to be a viable option in large scale projects. However this does not
go without saying that there are not any problems with the process as the absence of unit testing
is not easily over looked.
References
1. Cleanroom Software Engineering. Retrieved from University of Texas at Arlington website:
http://www.uta.edu/cse/levine/fall99/cse5324/cr/clean/page1.html
2. DACS, The Data and Analysis Center for Software. Retrieved January 5, 2009 Found at:
www.dacs.dtic.mil/databases/url/key.php?keycode=64
3. Prowell, Stacy et al. (1993). "Cleanroom Software Engineering: Technology and Process".
Reading, MA: Addison Wesley Longman, Inc.
4. Green, Scott et al. (1990). "The Cleanroom Case Study in the Software Engineering
Laboratory: Project Description and Early Analysis". NASA. Retrieved from
http://ntrs.nasa.gov/archive/nasa/casi.ntrs.nasa.gov/19910008271_1991008271.pdf