A SOFTWARE TESTING LIFECYCLE

3.2 A SOFTWARE TESTING LIFECYCLE

So far, we have seen many testing processes, including unit testing, integration testing, reliability testing, acceptance testing, certification testing, and regression testing. We will review other forms of testing in the next chapter. While they may differ in many ways, these forms of testing all follow a generic process, which includes the following phases:

• Preparing a Test Environment: With the possible exception of regression testing, which takes place during the operations and maintenance, most testing happens

28 A SOFTWARE TESTING LIFECYCLE

in a development environment rather than its actual operating environment. Hence, it is incumbent on the test engineer to create a test environment that mimics the operational environment as faithfully as possible. This may be a non- trivial task, involving such steps as simulating the operational environment; creating stubs for missing parts of the operational environment; and simulating workload conditions by creating fictitious demands on the system.

• Generating Test Data: Ideally, to test a program, we would like to execute it on all possible inputs or combinations of inputs or combinations of inputs and inter- nal states (if the program carries an internal state), and observe its behavior under these circumstances. Unfortunately, that is totally unrealistic for all but the most trivial programs. Hence, the challenge for the program tester is to find a set of test data that is small enough to be feasible, yet large enough to be representative. What do we mean by representative: we mean that if the program executes successfully on the test data, then we can be fairly confident that it would execute successfully on any input data (or more generally any configuration of input data and internal state). Quantitatively, if we let S be the set of all possible configurations of inputs and internal states, and D the subset of S that includes all the configurations on which the program was tested successfully; and if we let σ and δ be, respectively, the events “the program runs successfully on all elements of S” and “the program runs successfully on all elements of D,” then we want the conditional probability Π(σ | δ) to be as close to 1.0 as possible. This, in general, is a very difficult problem; hence Dijkstra’s often-cited quote that “ testing can be used to prove the presence of faults, never their absence.” While one can hardly argue with this premise, we will see in later chapters that while testing cannot be used to prove a program correct, it can be used to establish lesser goals, that are useful nevertheless, such as: to estimate the reliability of

a program; to estimate the fault density of the program; to certify that the relia- bility of a program exceeds a required threshold; or, if used in conjunction with fault diagnosis and removal, to enhance the reliability of a program.

In practice, test data is generated by means of what is called a test selection criterion. This is a condition that characterizes elements of S that are in D. It is not difficult to generate a test selection criterion that produces a small set D; what is very difficult is to generate a test selection criterion that is representative of S (a much harder case to make). The generation of a test selection criterion is one of the most important attributes of software testing; it is also a very difficult decision to make, as we discuss in Part III of the book, and the aspect of software testing that has mobilized the greatest share of researcher attention. Selecting what data to run the product on determines the fate of the test process, in the sense that it affects the extent to which the test achieves its goal. We can identify three broad cate- gories of test selection criteria, which are as follows: ○ Functional criteria of test data selection: These criteria consist in generating

test data by inspecting the specification of the software product; the goal of these criteria is to exercise all the functionalities and services that the product is supposed to deliver.

3.2 A SOFTWARE TESTING LIFECYCLE

○ Structural criteria of test data selection: These criteria consist in generating test data by inspecting the source code of the product; the goal of these criteria is to

exercise all the components of the product. ○ Random test data selection: These criteria consist in generating test data randomly over all of S (the combination of the input space and the internal state space); but this is usually done according to a specific usage pattern. In practice, the com- bined configurations of inputs and internal states are not equally likely to occur; some may be more likely to occur than others; in order for this type of selection criterion to be effective, we need to have a probability distribution function over S that quantifies the likelihood of occurrence of any element of S in normal usage. By adhering to the same probability distribution during the testing phase, we ensure that whatever behavior is observed during the testing phase is likely to arise during field usage; another advantage of this approach is that test data can be gen- erated automatically (using random data generators), so that a software product can be tested on much more inputs than if test data were generated by hand.

• Generating an Oracle: Whenever we test a software product, we need an oracle, that is, an agent that tells us, for each execution, whether the software product behaved correctly or not. The most obvious candidate for an oracle is the specification that the software product is meant to satisfy; and the safest way to implement an oracle is to write a certified Boolean function that takes as input the test data and the output of the program and rules on whether the observed input/output behavior satisfies the specification. But there are several situations where this ideal solution is impractical, or unnecessary, which are as follows: ○ First, the specification may be so complex that writing a Boolean function to test it

is difficult and/or error-prone; if the Boolean function that represents the oracle is more complex than the program under test, then this solution defeats the purpose of the test, and may in fact mislead us into the wrong conclusions and actions.

○ Second, it may be unnecessary to test the program against all the clauses of the specification: we may be interested in testing safety properties of the software

product, in which case the oracle will only reflect safety critical requirements; or we may verify the correctness of the program against some aspects of the specification using alternative means (e.g., static analysis).

○ Third, the process of storing the test data prior to each test and executing the oracle after each test may be prohibitively expensive in terms of computer

resources, compelling us to consider more cost-effective options. ○ Fourth, there are cases where we want to use an oracle that is in fact stronger (more demanding) than the specification: when the goal of the test is to find faults, it is not sufficient to know that the program satisfies the specification; rather it is necessary to check that the program computes the exact function that the designers intended it to compute; any deviation from this function may be an indication of a fault in the program.

• Generating a Termination Condition: Any test process aims to achieve a goal: For example, unit testing aims to diagnose faults in the program unit before

30 A SOFTWARE TESTING LIFECYCLE

integrating it into the project’s configuration; integration testing aims to diagnose faults in the design of the system or the specifications in the system’s unit; reliability testing aims to estimate the reliability of the software product, or to remove enough faults from the system to raise its reliability beyond a required threshold; acceptance testing aims to establish the dependability of the software product to the satisfaction of the customer (or to the terms of the development contract); and so on. The Termination Condition of a test is the condition that characterizes the achievement of the goal, that is, the condition that we test to know that we have achieved the goal of the test, hence we can terminate the test.

• Producing a Test Driver: The test driver is the process whereby the program is executed on the selected test data, its output is tested against the selected test oracle, and a course of action is defined for the case of a successful test and the case of an unsuccessful test, until the condition of termination is realized. If test data is generated automatically (e.g., using random test data generation, or by reading from

a predefined test data repository), and if the termination condition can be checked on the fly (e.g., generating a predefined number of test data samples, or exhausting a file of test data), and if the analysis of the test outcome can be done off-line, then the test driver can be automated. A generic pattern for such a test driver may be as follows:

void function testdriver() {statetype state, initstate; while (! testTermination())

{generateTest(state); initstate=state;

Program(state); // candidate program // modifies state if (oracle(initstate, state)) {successfultest(initstate);} else

{unsuccessfultest(initstate);}}

cout << “test report”;} At each iteration, the driver generates a new test data sample, stores it into var-

iable initstate, then lets the program under test run, modifying the current state but keeping the initial state intact. Then, the test oracle is called to check the execution of the program on the current test data sample, and depending on the outcome of the test, takes some action; if the test is successful, it may record the initial state on which the test was successful, or simply increment a counter recording the number of successful tests; if the test is unsuccessful it may write

a failure report on some file intended for the analysis of the test outcome. • Executing the Test: This phase consists merely of executing, whether by hand or automatically, the test driver that is defined in the previous phase. • Analyzing Test Outcome: The whole test would be in vain if we did not have a phase in which we analyze the outcome of the test and draw the conclusion that is called for, depending on the goal of the test. If the goal of the test is to find faults, then this phase consists in analyzing the outcome of the test to identify faults and

3.2 A SOFTWARE TESTING LIFECYCLE

consists in determining whether the product can be deemed acceptable; if the goal of the test is to estimate reliability, then this phase consists in computing the estimated reliability on the basis of the observed successes and failures of the program under test; and so on.

The process of executing the test varies according to the type of test, but broadly follows the process depicted in Figure 3.3. There are instances where the test loop exits

Termination

Yes

Analyzing test

condition

outcome, exit

satisfied?

No

Produce next test data

Record successful execution

Execute the product on the test data

Take relevant action

Depending on test driver

32 A SOFTWARE TESTING LIFECYCLE

whenever an unsuccessful execution is encountered; in such cases, the fault that may have caused the failure of the program is diagnosed, then removed, and the test resumes by reentering the loop. In other instances, an unsuccessful test does not disrupt the loop, but does cause a record to be stored to document the circumstances of the fail- ure. These cases will be explored in Chapter 7, when we discuss a taxonomy of testing methods.