Criteria for Completion of Testing
18.1.4 Criteria for Completion of Testing
A classic question arises every time software testing is discussed: "When are we done testing—how do we know that we've tested enough?" Sadly, there is no definitive answer to this question, but there are a few pragmatic responses and early attempts at empirical guidance.
? One response to the question is: "You're never done testing, the burden simply
When are
we done
shifts from you (the software engineer) to your customer." Every time the cus-
testing?
tomer/user executes a computer program, the program is being tested. This sober- ing fact underlines the importance of other software quality assurance activities. Another response (somewhat cynical but nonetheless accurate) is: "You're done test- ing when you run out of time or you run out of money."
Although few practitioners would argue with these responses, a software engi- neer needs more rigorous criteria for determining when sufficient testing has been conducted. Musa and Ackerman [MUS89] suggest a response that is based on sta- tistical criteria: "No, we cannot be absolutely certain that the software will never fail, but relative to a theoretically sound and experimentally validated statistical model, we have done sufficient testing to say with 95 percent confidence that the probabil- ity of 1000 CPU hours of failure free operation in a probabilistically defined environ- ment is at least 0.995."
F I G U R E 18.3
Failure
Data collected during testing
intensity as a function of execution time
Predicted failure intensity, l(t)
Failures per test hour
Execution time, t
Using statistical modeling and software reliability theory, models of software fail- ures (uncovered during testing) as a function of execution time can be developed [MUS89]. A version of the failure model, called a logarithmic Poisson execution-time model, takes the form
(18-1) where
f(t) = (1/p) ln [l 0 pt + 1]
f(t) = cumulative number of failures that are expected to occur once the software has been tested for a certain amount of execution time, t, l 0 = the initial software failure intensity (failures per time unit) at the begin-
ning of testing, p = the exponential reduction in failure intensity as errors are uncovered
and repairs are made.
The instantaneous failure intensity, l(t) can be derived by taking the derivative of f(t)
(18-2) Using the relationship noted in Equation (18-2), testers can predict the drop-off of
l(t) = l 0 / (l 0 pt + 1)
errors as testing progresses. The actual error intensity can be plotted against the pre- dicted curve (Figure 18.3). If the actual data gathered during testing and the loga- rithmic Poisson execution time model are reasonably close to one another over a number of data points, the model can be used to predict total testing time required to achieve an acceptably low failure intensity.
By collecting metrics during software testing and making use of existing software reliability models, it is possible to develop meaningful guidelines for answering the question: "When are we done testing?" There is little debate that further work remains to be done before quantitative rules for testing can be established, but the empirical approaches that currently exist are considerably better than raw intuition.
PA R T T H R E E C O N V E N T I O N A L M E T H O D S F O R S O F T WA R E E N G I N E E R I N G
Parts
» The Concurrent Development Model
» SUMMARY Software engineering is a discipline that integrates process, methods, and tools for
» PEOPLE In a study published by the IEEE [CUR88], the engineering vice presidents of three
» THE PROCESS The generic phases that characterize the software process—definition, development,
» THE PROJECT In order to manage a successful software project, we must understand what can go
» METRICS IN THE PROCESS AND PROJECT DOMAINS
» Extended Function Point Metrics
» METRICS FOR SOFTWARE QUALITY
» INTEGRATING METRICS WITHIN THE SOFTWARE PROCESS
» METRICS FOR SMALL ORGANIZATIONS
» ESTABLISHING A SOFTWARE METRICS PROGRAM
» Obtaining Information Necessary for Scope
» An Example of LOC-Based Estimation
» QUALITY CONCEPTS 1 It has been said that no two snowflakes are alike. Certainly when we watch snow
» SUMMARY Software quality assurance is an umbrella activity that is applied at each step in the
» R diagram 1.4 <part-of> data model; data model <part-of> design specification;
» SYSTEM MODELING Every computer-based system can be modeled as an information transform using an
» Facilitated Application Specification Techniques
» Data Objects, Attributes, and Relationships
» Entity/Relationship Diagrams
» Hatley and Pirbhai Extensions
» Creating an Entity/Relationship Diagram
» SUMMARY Design is the technical kernel of software engineering. During design, progressive
» Data Modeling, Data Structures, Databases, and the Data Warehouse
» Data Design at the Component Level
» A Brief Taxonomy of Styles and Patterns
» Quantitative Guidance for Architectural Design
» Isolate the transform center by specifying incoming and outgoing
» SUMMARY Software architecture provides a holistic view of the system to be built. It depicts the
» The User Interface Design Process
» Defining Interface Objects and Actions
» D E S I G N E VA L U AT I O N
» Testing for Real-Time Systems
» Organizing for Software Testing
» Criteria for Completion of Testing
» The Transition to a Quantitative View
» The Attributes of Effective Software Metrics
» Architectural Design Metrics
» Component-Level Design Metrics
» SUMMARY Software metrics provide a quantitative way to assess the quality of internal product
» Encapsulation, Inheritance, and Polymorphism
» Identifying Classes and Objects
» The Common Process Framework for OO
» OO Project Metrics and Estimation
» Event Identification with Use-Cases
» SUMMARY Object-oriented analysis methods enable a software engineer to model a problem by
» Partitioning the Analysis Model
» Designing Algorithms and Data Structures
» Program Components and Interfaces
» SUMMARY Object-oriented design translates the OOA model of the real world into an
» Testing Surface Structure and Deep Structure
» Deficiencies of Less Formal Approaches 1
» What Makes Cleanroom Different?
» Design Refinement and Verification
» SUMMARY Cleanroom software engineering is a formal approach to software development that
» Structural Modeling and Structure Points
» Describing Reusable Components
» SUMMARY Component-based software engineering offers inherent benefits in software quality,
» Guidelines for Distributing Application Subsystems
» Middleware and Object Request Broker Architectures
» An Overview of a Design Approach
» Consider expert Web developer will create a complete design, but time and cost can be appropriate
» A Software Reengineering Process Model
» Reverse Engineering to Understand Data
» Forward Engineering for Client/Server Architectures
» SUMMARY Reengineering occurs at two different levels of abstraction. At the business level,
Show more