THEORY OF WEYUKER AND OSTRAND

2.3 THEORY OF WEYUKER AND OSTRAND

A key problem in the theory of Goodenough and Gerhart is that the reliability and validity of a criterion depend upon the presence of faults in a program and their types. Weyuker and Ostrand [18] provide a modified theory in which the validity and reliability of test selection criteria are dependent only on the program specifi- cation , rather than a program. They propose the concept of a uniformly ideal test selection criterion for a given output specification. In the theory of Goodenough and Gerhart, implicit in the definitions of the predicates OK(d ) and SUCCESS- FUL(T ) is a program P . By abbreviating SUCCESSFUL() as SUCC(), the two predicates are rewritten as follows:

OK(P, d) : Define a predicate OK(P , d ) which expresses the acceptability of result P (d ). Thus, OK(P , d ) = true if and only if P(d) is an acceptable

outcome of program P . SUCC(P, T) : For a given T ⊆ D, T is a successful test for a program

P , denoted by SUCC(P , T ), if and only if, ∀t ∈ T , OK(P, t). Thus, SUCC(T ) = true if and only if, ∀t ∈ T , OK(P, t).

With the above definitions of OK(P , d ) and SUCC(P , T ), the concepts of uniformly valid criterion, uniformly reliable criterion, and uniformly ideal test selection are defined as follows.

38 CHAPTER 2 THEORY OF PROGRAM TESTING

Uniformly Valid Criterion C : Criterion C is uniformly valid iff (∀P )[(∃d ∈ D)(¬OK(P , d)) ⇒ (∃T ⊆ D)(C(T ) & ¬SUCC(P , T ))]

Uniformly Reliable Criterion C : Criterion C is uniformly reliable iff

(∀P )(∀T 1 , ∀T 2 ⊆ D)[(C(T 1 ) & C(T 2 )) ⇒ (SUCC(P , T 1 ) ⇔ SUCC(P , T 2 )) ]

Uniformly Ideal Test Selection : A uniformly ideal test selection criterion for

a given specification is both uniformly valid and uniformly reliable. The external quantifier (∀P) binding the free variable P in the definition of uni-

formly valid criterion C essentially means that the rest of the predicate holds for all programs P for a given output specification. Similarly, the external quantifier (∀P) binding the free variable P in the definition of uniformly reliable criterion C means that the rest of the predicate holds for all programs P for a given output specification.

Since a uniformly ideal test selection criterion is defined over all programs for

a given specification, it was intended to solve all the program-dependent difficulties in the definitions given by Goodenough and Gerhart. However, the concept of uniformly ideal test selection also has several flaws. For example, for any significant program there can be no uniformly ideal criterion that is not trivial in the sense of selecting the entire input domain D . A criterion C is said to be trivially valid if the union of all tests selected by C is D . Hence, the following theorems.

Theorem.

A criterion C is uniformly valid if and only if C is trivially valid. Proof. Obviously a trivially valid criterion is valid. Now we need to show that a

criterion C which is not trivially valid cannot be uniformly valid for a given output specification. For any element d not included in any test of C , one can write a program which is incorrect for d and correct for D − {d}.

Theorem.

A criterion C is uniformly reliable if and only if C selects a single test set.

Proof. If C selects only one test, it is obviously reliable for any program. Now, assume that C selects different tests T 1 and T 2 and that t ∈ T 1 but t / ∈T 2 .A program P exists which is correct with respect to test inputs in T 2 but incorrect on t . Thus, the two tests yield different results for P , and C is not reliable.

Now, we can combine the above two theorems to have the following corollary.

Corollary.

A criterion C is uniformly valid and uniformly reliable if and only if

C selects only the single test set T = D.

39 An important implication of the above corollary is that uniform validity and

2.4 THEORY OF GOURLAY

uniform reliability lead to exhaustive testing —and exhaustive testing is considered to be impractical. Next, the above corollary is reformulated to state that irrespective of test selection criterion used and irrespective of tests selected, except the entire

D , one can always write a program which can defeat the tests. A program P is said to defeat a test T if P passes T but fails on some other valid input. This is paraphrasing the well-known statement of Dijkstra that testing can only reveal the presence of errors, never their absence [16].

Reliability and validity of test selection criterion are ideal goals, and ideal goals are rarely achieved. It is useful to seek less ideal but usable goals. By set- tling for less ideal goals, we essentially accept the reality that correctness of large programs is not something that we strive to achieve.

Weyuker and Ostrand [18] have introduced the concept of a revealing crite- rion with respect to a subdomain, where a subdomain S is a subset of the input domain D . A test selection criterion C is revealing for a subdomain S if when- ever S contains an input which is processed incorrectly then every test set which satisfies C is unsuccessful. In other words, if any test selected by C is success- fully executed, then every test in S produces correct output. A predicate called REVEALING(C , S ) captures the above idea in the following definition:

REVEALING(C, S) iff (∃d ∈ S)(¬OK(d)) ⇒ (∀T ⊆ S)(C(T ) ⇒ ¬SUCC(T )) The key advantage in a revealing criterion is that it concerns only a subset of the

input domain, rather than the entire input domain. By considering a subset of the input domain, programmers can concentrate on local errors. An important task in applying the idea of a revealing criterion is to partition the input domain into smaller subdomains, which is akin to partitioning a problem into a set of subproblems. However, partitioning a problem into subproblems has been recognized to be a difficult task.