TEST GENERATION REQUIREMENTS
8.3 TEST GENERATION REQUIREMENTS
The purpose of a test operation is to run a candidate program on sample inputs and observe its behavior in order to draw conclusions about the quality of the program. In general, we cannot make certifiable claims about the quality of the candidate pro- gram unless we have observed its execution on all possible inputs under all possible circumstances. Because this is most generally impossible, we must find substitutes; this is the focus of test data generation.
Given a candidate program p and a specification R whose domain is X, we are inter- ested to choose a subset T of X such that we can achieve the goal of the test by execut- ing the candidate program on T rather than on X. Clearly, the requirement that T must satisfy depends on the goal of the test. As we remember from Chapter 7, we identify several possible goals of testing, including the following:
• Finding and removing faults • Proving the absence of faults • Estimating the frequency of failures • Ensuring the infrequency of failures
8.3 TEST GENERATION REQUIREMENTS 149
These goals impose different requirements on T, which we review below: •
A Logical Requirement: Set T must be chosen in such a way that if there exists an input x in X such that a candidate program p fails when executed on x, then there exists t in T such that program p fails when executed on t. This requirement provides, in effect, that set T is sufficiently rich to detect all possible faults in candidate programs.
A Stochastic Requirement: The reliability observed on the execution of a candi- date program p on T is the same as (or an approximation of ) the reliability observed on the execution of the candidate program p on X. So that any reliability claim made on the basis of observations made during the testing phase, when input data are limited to T, will be borne out during the operation phase, when the input ranges over all of X.
A Sufficient Stochastic Requirement: The reliability observed on the execution of a candidate program p on T is lower than, or the same as, the reliability observed on the execution of a candidate program p on X. So that any reliability claim made on the basis of observations made during the testing phase, when input data is limited to T, will be matched or exceeded during the operation phase, when the input ranges over all of X.
Given that the set of states that yield a successful execution of a candidate program p is dom R P , the logical requirement can be expressed as follows:
X dom R P T dom R P This requirement provides, in effect, that if program p has a fault, test data set T will
expose it; as such, this requirement serves the first goal, whose focus is to expose all faults of the program. In order for T to serve the second goal of testing, it needs to satisfy the following condition:
X dom R P, which provides, in effect, that if program p executes successfully on T, then it executes
T dom R P
successfully on X (hence is correct). The following proposition provides that these two conditions are equivalent.
Proposition: Test Data Adequacy Let R be a specification on space S, whose domain is X and let p be a program on space S, and T a subset of X. The following two conditions are equivalent:
• X dom R P
T dom R P
• T dom R P
X dom R P.
150 TEST GENERATION CONCEPTS
Proof We proceed by equivalence:
X dom R P
T dom R P
{De Morgan’s laws} T dom R P=
X dom R P=
by set theory, A B= is equivalent to A B T dom R P
X dom R P
{by set theory, A=A T dom R P
X dom R P
QED
This proves that a test set T is adequate for the first goal of testing (finding and removing faults) if and only if it is adequate for the second goal of testing (proving the absence of faults). Figures 8.2, 8.3, and 8.4, respectively, show three configura- tions of X, dom R P and T: a case where a test data T is certainly inadequate, a case where test data T may be adequate, and a set of two cases where test data T is certainly adequate. The cases where T is provably adequate are both of limited practical interest: one arises when the program is correct, the other arises when T is all of S.
The test data T of Figure 8.2 is certainly inadequate: The left-hand side of the impli- cations in proposition test data adequacy is valid, but the right-hand side is not. Test data T of Figure 8.3 may be adequate.
X = dom(R)
dom(R∩P)
Figure 8.2 Inadequate test data.
8.3 TEST GENERATION REQUIREMENTS 151
X = dom(R)
dom(R∩P)
Figure 8.3 Possibly adequate test data.
dom (R) dom (R) = dom(R∩P)
=T
T dom(R∩P)
Figure 8.4 Certainly adequate test data.
Figure 8.4 shows cases where T is certainly adequate: (a) if p is correct or (b) if T is exhaustive test. The testing goals can be mapped onto test data requirements as shown in the following table:
Testing goals Test data requirements
Finding and removing faults Logical requirement Proving the absence of faults
Logical requirement Estimating the frequency of failures
Stochastic requirement Ensuring the infrequency of failures
Stochastic requirement Sufficient stochastic requirement
152 TEST GENERATION CONCEPTS
Note that the stochastic requirement logically implies the sufficient stochastic requirement; hence, any test data set that satisfies the former satisfies the latter; for the same reason, the former can be used whenever the latter is adequate. Note that we focus our attention on X, the domain of the specification, rather than the domain (or the state space) of the program, since candidate programs are judged for their behavior on the domain of the specification they are supposed to satisfy.
In practice, it is very difficult to generate test data that meets these requirements, especially the logical requirement; nevertheless, these requirements serve a useful function, in that they define the goal that we must attain as we generate test data, and a yardstick by which we judge the quality of our choices. In the next section, we discuss the concept of a test selection criterion, which is a criterion by which we characterize test data to satisfy some requirement among the three introduced in this section.