TEST GENERATION CRITERIA

8.4 TEST GENERATION CRITERIA

Given a test generation requirement, among the three we have discussed earlier, it is common to generate a test generation criterion, that is, a condition on set T that dictates how to generate it in such a way as to satisfy the requirement.

For the logical requirement, the most compelling (and most common) criterion is to partition the domain of the specification by an equivalence relation, say EQ, and to mandate that T contain one representative element for each equivalence class of

X modulo EQ. Formally, these conditions are written as follows: • EQ T = X

• T×T EQ I The first condition provides that each equivalence class of X modulo EQ is

represented by at least one element of T, as illustrated in Figure 8.5 (where the equivalence classes are represented by the quadrants). As for the second condition, it merely provides that T contains no unnecessary elements, that is, no two elements of the same equivalence class.

The rationale for this criterion depends on the definition of EQ: Ideally, EQ is defined in such a way that all the elements of the same equivalence class of X modulo EQ have the same fault diagnosis capability, that is, either the candidate program runs successfully on all of them or it fails on all of them; hence, there is no reason to test candidate programs on more than one element per equivalence class. Formally, this condition can be written as follows:

EQ s, s s dom R P s dom R P We refer to this condition (on EQ) as the condition of partition testing, and we have

the following proposition.

8.4 TEST GENERATION CRITERIA 153

X/EQ

Figure 8.5 Partitioning the domain of the specification.

Proposition: Partition Testing Let R be a specification whose domain is X, and let EQ be an equivalence relation on X. If relation EQ satisfies the condition of partition testing, then any set T that satisfies the condition EQ T = X necessarily satisfies the logical requirement of test selection.

Proof. We must prove: T dom R P

X dom R P To this effect, we assume the left-hand side and prove the right-hand side. By

hypothesis, we have X = EQ T , from which we infer, by the left-hand side of the aforementioned implication:

X EQ dom R P

On the other hand, from the condition of partition testing, we infer

X EQ dom R P,

where EQ = s,s s dom R P s dom R P Now, we compute EQ dom R P: EQ dom R P

154 TEST GENERATION CONCEPTS

= {definition} s s s dom R P

s, s

EQ

= {substitution} s s s dom R P

s dom R P s dom R P = {logical equivalence}

ss dom R P ss dom R P s dom R P s dom R P {simplification} ss dom R P {identity } dom R P

QED

The condition EQ T = X provides, in effect, that each element of X is related to at least one element of T; in other words, each equivalence class of X modulo EQ has

I, it is not needed for the proof of this proposition, because it only limits the size of T (it provides that no more than one representative per equivalence class is needed in T). In practice, the hypoth- esis that EQ does indeed satisfy this property is usually hard to support, and the criterion is only as good as the hypothesis.

a representative in T. As for the condition T × T EQ

For the stochastic requirement and the sufficient stochastic requirement, the most common generation criterion that we invoke provides that set T has the same prob- ability distribution as the expected usage pattern of the software product. The rationale for this criterion is to imitate the operating conditions of the software product to the largest extent possible, so that whatever behavior we observe during testing is sure to

be borne out during the product’s operation. Given a specification R whose domain is X, we consider the following criterion on a subset T of X: We let χ be a random variable on X that reflects the usage pattern of candidate programs in operation, and we let θ be a random variable on T that reflects the distribution of test data during the testing phase. Then the probability distribution p θ of θ over T must be identical to the probability distribution p χ of χ over X, in the following sense:

A,A X p θ x dx =

p χ χ dχ

xA

xA

8.5 EMPIRICAL ADEQUACY ASSESSMENT 155

Figure 8.6 Mimicking a probability distribution.

The probability of occurrence of a test data point in any sub set A of X is identical to the probability of occurrence of an actual input data point in subset A during typical system operation. As an additional requirement, T must also be large enough so that observations of the candidate program on T provide a statistically significant sample of the program’s behavior. In practice, depending on how large (or how dense) T is, it may be impossible to define a probability distribution p θ that mimics exactly the prob- ability distribution p χ ; we then let distribution p θ approximate the probability distri- bution p χ (Fig. 8.6).

In order to mimic the probability distribution p χ , set T must have more elements where p χ is high than where p χ is low. This matter will be revisited in Chapter 9, when we discuss random test generation.