COMPARISON OF TESTING TECHNIQUES

5.9 COMPARISON OF TESTING TECHNIQUES

So far we have discussed two major techniques for generating test data from source code, namely control flow–based path selection and data flow–based path selection. We also explained a few criteria to select paths from a control flow graph and data flow graph of a program. Programmers often randomly select test data based on their own understanding of the code they have written. Therefore, it is natural to

127 (All-paths)*

5.9 COMPARISON OF TESTING TECHNIQUES

(All-c-uses/Some-p-uses)*

(All-p-uses/Some-c-uses)*

(All-c-uses)* (All-defs)*

(All-p-uses)*

Figure 5.6 Relationship among FDF (feasible data flow) testing criteria. (From ref. 4. © 1988 IEEE.)

compare the effectiveness of the three test generation techniques, namely random test selection, test selection based on control flow, and test selection based on data flow. Comparing those techniques does not seem to be an easy task. An acceptable, straightforward way of comparing them is to apply those techniques to the same set of programs with known faults and express their effectiveness in terms of the following two metrics:

• Number of test cases produced • percentage of known faults detected

Ntafos [5] has reported on the results of an experiment comparing the effec- tiveness of three test selection techniques. The experiment involved seven math- ematical programs with known faults. For the control flow–based technique, the branch coverage criterion was selected, whereas the all-uses criterion was cho- sen for data flow testing. Random testing was also applied to the programs. The data flow testing, branch testing, and random testing detected 90%, 85.5%, and 79.5%, respectively, of the known defects. A total of 84 test cases were designed to achieve all-uses coverage, 34 test cases were designed to achieve branch cover- age, and 100 test cases were designed in the random testing approach. We interpret the experimental results as follows:

A programmer can randomly generate a large number of test cases to find most of the faults. However, one will run out of test cases to find some of the remaining faults. Random testing does not look to be ineffective, but

128 CHAPTER 5 DATA FLOW TESTING

Total number of faults in a program

Reduce this gap

Number of faults detected by using a technique Random

Control flow–

Data flow–

New testing

testing based testing

based testing

techniques

Figure 5.7 Limitation of different fault detection techniques.

it incurs higher costs than the systematic techniques, namely, the control flow and the data flow techniques.

• Test selection based on branch coverage produces much fewer test cases than the random technique, but achieves nearly the same level of fault detection as the random technique. Thus, there is significant saving in the cost of program testing.

• The all-uses testing criterion gives a programmer a new way to design more test cases and reveal more faults than the branch coverage criterion.

• All these techniques have inherent limitations which prevent them from revealing all faults. Therefore, there is a need to use many different testing techniques and develop new techniques. This idea is depicted in Figure 5.7. Our goal is to reduce the gap between the total number of faults present in a program and the faults detected by various test generation techniques.