Historical Perspective in Optimising
Software Testing Efforts

Ashish Jolly
S A Jain Institute of Management & Technology
Jain College Road, Ambala City (Haryana)


Chief scientist, Caper Jones, of Software Productivity Research in Massachusetts, estimates that 60 percent of the United State's software work force is dedicated to fixing software errors that could have been avoided. In addition, he states there are only 47 days in a calendar year dedicated to doing actual development or enhancement of software applications. Most of the software engineering work is wasted due to fixing errors or working on projects that are eventually cancelled. It appears product reliability has fallen to the waste side. The 1980s emphasized quality control, however by the 1990s innovation at top speed has taken top priority.

The current era of desktop computing will soon pass evolving to a new era of globalization. Software globalization will form the new infrastructure of our society. It will require desktop computers to be an essential business tool rather than the occasional productivity enhancer. In essence, this new era will require high dependability and reliability due to software computing now performing life critical functions (Morris, 2000). Therefore, there must be a better method to software development without creating numerous defects.

One approach to developing quality software with minimal defects is Cleanroom Software Engineering (CSE).

Cleanroom Software Engineering (CSE) Defined

In days gone by, software development was rife with poor practices. These included ad-hoc, cut-and-try debugging (often called "unit testing" in practice), bottom-up delivery (code first, documentation last if ever), and unstructured coding practices.

Those dark days saw many projects with characteristics like:

  • Blaming testing for late deliveries: code enters test "on time" and never seems to leave.
  • One "guru," indispensable to a project and hostage to its continued maintenance.
  • Uncertainty about a delivery date until late in the game.
  • Eleventh-hour integration attempts to put together something, anything, for a demo or trade show.
  • Unexpectedly low reliability resulting in high support costs.
  • Limited re-use due to unreliability and poor documentation of components.
  • Market pressure demands for simultaneous improvements in quality and productivity.

Harlan Mills and his colleagues from IBM developed the CSE methodology in the early 1980s. They were part of IBM's Federal Defense System where software failures could mean millions of dollars and most importantly, human lives. This software methodology follows the same analogy as cleanroom fabrication of semiconductors. Instead of trying to clean dirt off the semiconductor wafers after production, the object is to prevent the dirt from getting into the production environment in the first place.

This realization led Mills and his team to suggest a software engineering approach that was radical, even shocking at the time. Cleanroom software engineering, he proposed, should emphasize defect prevention over defect removal, and must include development and testing practices supportive of that emphasis.

CSE focuses on theory-based engineering practices of software development and certification while using the Capability Maturity Model for Software (CMM) as its framework. This framework (i.e. CMM) describes the key elements of an effective software process such as planning, engineering, managing development and maintenance. In addition, CMM establishes a means to measure an organization's software development process by maturity levels. Nevertheless, the main point is that CSE encompasses CMM as part of its software development process.

CSE spans the entire software life cycle. It has an 80-20 life cycle. This means 80 percent of the time is dedicated to design while 20 percent is writing code. In contrast, the traditional life cycle is 40-20-40 where 40 percent is design, 20 percent is coding, and 40 percent is unit testing. There is absolutely no unit testing as part of the CSE process. The untested software product is expected to work. If it does not, the defects are analyzed to determine how the software development process should be improved. Then the defective product is discarded. According to Harlan Mills, the most important tool to CSE is the wastebasket.

CSE is based on two main principles. They are the design and testing principles. The design principle consists of mathematical functional verification of the product. Here reviewers of the software product use mathematical functions to argue the correctness of the original design. The testing principle focuses on certifying the reliability of software product (i.e. measuring the quality). This technique uses statistical testing based on expected system usage rather than trying to remove defects.

The Truth About Cleanroom: Principles and Practices

The focus of Cleanroom involves moving from traditional, craft-based software development practices to rigorous, engineering-based practices. It is well-documented that significant differences in cost are associated with errors found at different stages of the software life cycle. By detecting errors as early as possible, Cleanroom reduces the cost of errors during development and the incidence of failures during operation; thus the overall life cycle cost of software developed under Cleanroom can be expected to be far lower than industry average.

The four main components to CSE include the following: incremental development; formal methods of specification and design; correctness verification; and Statistical Usage Based Testing and Certification. Figure 1 in Appendix A shows the CSE Model. Each of these components is necessary to develop software code that prevents errors from the beginning of development.

Incremental Development

Using cleanroom methodology, software products are developed in a series of functional increments that sum up to the final deliverable product. These increments represent operational user functions. The most stable requirements are implemented first, with further steps of requirements resolution performed with each successive increment. The integration of these increments is done top down. This approach allows early assessment of the product quality and gives continuous feedback to management as to the progress of development. When the final increment is integrated, the system is complete.

As each increment is integrated, the evolving system of increments undergoes a new step in statistical usage testing for quality certification. Statistical measures of quality provide feedback for reinforcement or improvement of the development process as necessary. Early increments serve as system prototypes, providing an opportunity to elicit feedback from customers to validate requirements and functionality. It's not a bad approach for the marketing group either, who can demonstrate core parts of the product early. As inevitable changes occur, incremental development provides a framework for revising schedules, resources, and function, and permits changes to be incorporated in a systematic manner.

Incremental Development Benefits

Incremental development has many advantages over bottom-up and traditional "waterfall" lifecycles:

  • risks are addressed and mitigated systematically;
  • the cost of test scaffolding and drivers for artificial interfaces is reduced;
  • early quality measurement is meaningful because the "real" interfaces are tested;
  • every test case is a rehearsal of actual product use;
  • a user view of the system is seen early in the process;
  • high-level interfaces are thoroughly tested from the first increment onward;
  • a running system exists early, and can be delivered if necessary.

Formal Methods of Specification and Design

Cleanroom software engineering is based on a set of formal specifications describing the external behavior of the system. In the past, there was little incentive to devote much effort to the specification process. The Cleanroom approach puts much more emphasis on producing a formal specification model. Cleanroom specifications should provide a complete and precise description of the system behavior. Although the Cleanroom approach requires formal methods for specification and design, practical usage has suggested that the level of formality will vary from project to project and even within a single project.

(i) Function Specification Process: The purpose of the Function specification process is to (a) specify the complete functional behavior of the software in all possible circumstances of use and

(b) obtain agreement with the customer on the specified function.

Function Specification is based on Software Requirements. Function specification activities include:

  • Defining the format and notation for specification.
  • Defining software boundaries and stimulus/response interfaces with hardware, other software and human users.
  • Specifying the required external behavior of the software in the black box function form.

The object-based technology of box structures is an effective specification technique for Cleanroom development. Box structures map system stimuli (inputs) and stimulus histories (previous inputs) into responses (outputs).

A black box is a specification of external behavior of the function for all possible circumstances. Inputs to the black box are called stimuli and outputs are responses. The responses from the black box are driven by the stimulus history and the current stimulus. We consider all cases, including unexpected and erroneous stimuli. This inclusion forces the developer to consider all possible cases of external behavior instead of a simple English language description of what is required that can be easily misinterpreted and incomplete. If you have used other development methodologies, you will find that this is not too different. Structured analysis and design uses data flows instead of stimuli and responses and the object-oriented approach (OO) uses messages as shown in Figure 3. A small mindset change is required to start thinking in terms of stimuli/response, but a small one.

A state box is derived from and verified against a corresponding black box. A state box maps the current stimulus and current state into a response and a new state. A clear box is derived from and verified against a corresponding state box. In the clear box, the procedures required to implement the state box transition function are defined, possibly introducing new black boxes for further decomposition into state or clear boxes.

Box structures of the kind used in cleanroom fixes some of the problems found in object-oriented (OO) approaches. One example is the attempt to encapsulate data in OO. Encapsulating data at the right level of abstraction can be difficult: too high a level degrades modularity and too low a level can lead to redundancies in the data. The cleanroom approach forces the designer to invent only the data needed at that time in the design. This box-structured approach also allows for intellectual control in an object-based development environment and forces the concept of top-down refinement and verification. The aim in doing all this is to achieve the transition from specification to code in small enough increments so that each step is obviously correct.

Box-structured systems are developed according to the following principles:

    (1) all data to be defined and retained in a design are encapsulated in boxes,
    (2) all processing is defined by sequential and concurrent use of boxes, and
    (3) each use of a box occupies a distinct place in the usage hierarchy of the system.

Clear boxes play an important role in the hierarchy by defining and controlling the correct operation of box services at the next level of refinement.

The logic behind proceeding in small increments (stepwise refinement) is based on the fact that the human intellect can only understand complexities when they are linked in close, simple relationships. Anyone who has gone into a legacy system to make a "simple" change and broke something else can better understand this. Cleanroom, if done correctly, can prevent this by allowing incremental development and verification where each black box can be considered totally independent and fixes or updates can be more isolated.

Referential Transparency is a popular mathematical principle applied in Cleanroom specification. With referential transparency, reference to the object gives the entire effect of the object itself. Referential transparency of box structure hierarchies concerns the completeness of specification of the behavior of lower level boxes at each step in system decomposition.

(ii) Architectural Specification Process: The purpose of the architectural specification process is to define the conceptual model, the structural organization and the execution characteristics of the software. The Cleanroom aspect of architecture specification is in decomposing of the history-based black box function specification into state-based state box and procedure-based clear box descriptions.

Correctness Verification

During software design and development, the software development team conducts a rigorous verification activity on the design and code using strict correctness verification methods that prove that the software meets the specification. Verification reviews are held by the team to formally or informally verify the software using a set of correctness proofs that are standard in a structured programming environment. This method allows the verification of programs of any size by reducing the verification process to a manageable number of verification checks. For well-specified, well-structured software, proof of software correctness can be done by direct assertion. Correctness verification is done before the software is ever executed, so that the developers are not permitted to get into a "debugging" mode of operation.

This type of correctness verification can result in a near zero defect level. This process also scales to programs of any size, because all programs have a finite number of control constructs. This method of correctness verification allows developers to deliver software without unit testing and produces better code than the old unit testing way. "No unit testing" does not, however, mean "no use of the machine." It is essential to use the machine for experimentation, to evaluate algorithms, to benchmark performance, and to understand and document the semantics of interfacing software.

Elimination of unit testing motivates tremendous determination in developers to ensure that the code they deliver for independent testing is error-free on first execution. But there is a deeper reason to adopt correctness verification-it is more efficient and effective than unit testing. Programs of any size can contain an essentially infinite number of possible execution paths and states, but only a minute fraction of those can be exercised in unit testing. Correctness verification, however, reduces the verification of programs to a finite and complete process.

Statistical Usage Based Testing and Certification

In the Cleanroom process, statistical usage testing for certification replaces coverage testing for debugging. Testing is carried out by the certification team based on anticipated usage by customers. The cleanroom concept asserts that this type of unit testing should be private to the programmer. Unit testing is actually dangerous because it tends to find many of the superficial defects and lets some of the other more important errors slip through. Cleanroom uses a formal statistical approach to testing that can indicate the quality of the software and stopping criteria as well. This approach differs from the traditional approach, in which testers assume there are errors in the software and set out to find as many as possible, with the faulty assumption that if enough errors are found and fixed, the software quality would improve. Cleanroom operates under the realistic assumption that it is not possible to test quality into a product. Certification (the cleanroom term for testing) is performed to certify the software reliability, not test the software in the classical sense.

Statistical usage testing involves testing software the way the users use it. This type of testing focuses on external system behavior, not the internals of the software. Statistical usage testing all starts with test planning and the development of a usage model. Planning involves developing a usage specification, which defines usage scenarios of the product. These usage scenarios can be both correct and incorrect uses of the product. These usage scenarios are used for test case generation, and testing is done using user profiles.

Coverage testing can provide no more than subjective evidence of reliability. Thus, if many errors are found, does that mean that the code is of poor quality and many errors remain, or that most of the errors have been discovered? Conversely, if few errors are found, does that mean that the code is of good quality, or that the testing process is ineffective? Statistical testing provides scientifically valid measures of reliability, such as mean-time-to-failure (MTTF), as a basis for objective management decision-making regarding software and development process quality.

Cleanroom testing involves the following four steps: (1) usage modeling, (2) generation of test cases, (3) execution of test cases, and (4) reliability estimation.

Advantages of Cleanroom Software Engineering

CSE has at least three main advantages. They are improved quality, increased productivity, and improved software maintainability. CSE improves quality by preventing errors in the first place. Thus, there is a reduction in the overall cost of the program. For example, from 1987 to 1994, software code developed with CSE only had 2.3 defects per thousand lines of code measured from the first time of execution. In contrast, industry averages using traditional software development methods were estimated at 10 defects per thousand lines of code after unit testing.

Another advantage CSE has over traditional software development methods is productivity. Productivity is increased due to the reduced time required to debug and rework software.

Lastly, software developed with CSE has clear, well-defined specifications with a less-complicated design allowing for easier maintenance. This is because CSE incorporates team reviews as part of the process and uses the box structure method for creating specifications. In a team review situation, there is not the crutch of relying on unit testing to find defects.


The typical software lifecycle is about 40% design, 20% code, and 40% unit testing. The cleanroom lifecycle is 80% design and 20% code; no unit test. Cleanroom spans the entire software development life cycle.

Cleanroom represents the first practical attempt at defining a development process that builds software without introducing defects. Cleanroom software engineering differs from conventional views in that it places importance on design specification and mathematically based proofs of correctness. In addition, the Cleanroom approach relies heavily on statistical usage testing to verify that the software will work without producing high-impact errors for the users in the field. The Cleanroom approach is unique in that it replaces unit testing and debugging with correctness verification and quality certification. The goal of the Cleanroom approach is to be able to build high quality software without increasing the project's cycle time.

If anything is new to CSE, it is the attitude that near zero-defect software is possible. This paper provided an in-depth review of CSE including a description of what it is and the advantages of using it. CSE is still thought of as some mythical software development process. However, more companies need to make quality software besides innovation as top priority. Further research is still needed on CSE. More companies need to pilot and publish how CSE was integrated in their current software development process.

Developing high quality software that is within budget and does not require a massive customer service staff should be on any software company's objective list. CSE is no silver bullet, however it does provide a more scientific and measurable process that takes the art out of software development.

Appendix A

CSE Model

Figure 1. CSE Model (Oshana & Linger, 1999)

Ashish Jolly
S A Jain Institute of Management & Technology
Jain College Road, Ambala City (Haryana)

Source: E-mail May 4, 2005


For Faculty Column Page 1: Article No. 1 to 99 Click here

For Faculty Column Page 2: Article No. 100 Onward Click here



Important Note :
Site Best Viewed in Internet
Explorer in 1024x768 pixels
Browser text size: Medium