SELECTING A SPECIFICATION

12.1 SELECTING A SPECIFICATION

In Chapter 11, we have discussed how to map a specification against which we want to test a program into a test oracle; in this section, we discuss how to choose a specification. One may argue that the question of what specification to test a program against is moot, since we do not get to choose the specification. We argue that while we do not in general have the luxury of deciding what specification we must test a program against, we do have some latitude in choosing how to deploy different ver- ification methods against different components of a complex, compound specifica- tion. Indeed, each family of verification methods (fault avoidance, fault removal, fault tolerance) works best for some type of specifications and works much less for others; the law of diminishing returns advocates that we use a wide range of methods, where each method is deployed against the specification components that are best adapted to it. We consider each broad family of methods and briefly characterize the properties of the specifications that are best adapted to it:

• Fault Avoidance/Static Analysis of Program/Static Verification of Program Cor- rectness. We argue that specifications that are reflexive and transitive are very

well adapted to static verification methods. Imagine that one has to prove the correctness of a complex program g with respect to a specification R that is repre- sented by a reflexive transitive relation. If g is structured as a sequence of two

subprograms, say g = g 1 ;g 2 , then to prove that G refines R, it suffices to prove that G 1 and G 2 refine R (since R is transitive). Likewise, we find that if g is an

Software Testing: Concepts and Operations, First Edition. Ali Mili and Fairouz Tchier. © 2015 John Wiley & Sons, Inc. Published 2015 by John Wiley & Sons, Inc.

254 TEST DRIVER DESIGN

if-then statement, then to prove that G refines R, it suffices to prove that the then- branch of g refines R; and that if g is an if-then-else statement, then to prove that

G refines R, it suffices to prove that each branch of the statement refines R; and finally that if g is a while statement, then in order for G to refine R, it suffices that the loop body of g refines R. More generally, the reflexivity and transitivity of R greatly simplify the inductive arguments that are at the heart of many algorithms, whereby reflexivity supports the basis of induction and transitivity supports the inductive step. As far as axiomatic program proofs are concerned (using Hoare’s logic), it is well known that the most difficult aspects of such proofs (and the main obstacle to their automation) is the need to generate intermediate assertions and invariant assertions. When the specification at hand is reflexive and transitive, these asser- tions often take the simple form

s 0 ,s R,

where s 0 is the initial state and s is the current state. A small illustration of this situation is given in Chapter 6, where a reflexive transitive relation (prm(s0,s)) is uniformly used as an intermediate assertion and as an invariant assertion throughout the program, and all the verification conditions have the same assertion as precondition and postcondition.

• Fault Removal/Software Testing. The main criterion that a specification must sat- isfy to be an adequate choice for testing is the criterion of reliability: It must pro-

duce an oracle that can be implemented reliably, as a faulty oracle may throw the whole test off-balance and may insert faults into the software product, rather than remove existing faults.

• Fault Tolerance. In order to be an adequate specification for fault tolerance, a specification has to meet the following criteria: first, lend itself to a simple, reli-

able oracle (the same criterion as for testing, for the same rationale); second, it has to lend itself to an efficient run-time execution (since it may have to be checked at run-time to detect errors); third, and most importantly, it has to refer to current states rather than current and past states. In practice, this third require- ment means that the specification is represented by an inverse vector, that is, a relation R such that LR = R. Such relations refer to the current state but not to any previous state; they offer the advantage that they can be checked by looking exclusively at the current state and spare us from the burden of storing previous states at designated checkpoints and from checking complex binary relations; hence they represent a savings in terms of memory space and processing time.

To illustrate the kind of rationale that governs the mapping of sub-specification to methods, we consider the following examples:

• We consider the example of the sorting routine discussed in Chapter 6, where we showed that such a program is difficult to prove using static methods, and

12.2 SELECTING A PROCESS 255

difficult to test, but that that it is easy to prove its correctness with respect to some part of the specification and very easy and efficient to test it against the other part of the specification.

• We consider the specification of a program to perform a Gaussian elimination of

a system of linear equations defined by a square matrix of size N and a column vector of size N. The specification that we consider for this program has the form:

Gauss = Eq Tri,

where Eq means that the original system of equations has the same set of solu- tions as the final system of equations and Tri means that the final system of equa- tions is triangular. It is difficult to prove the correctness of the program with respect to specification Gauss (since this requires that we deal with several nested loops, we check the start and end values of several index variables, we worry about the logic for finding optimal pivots, etc.); it is very complex, inef- ficient, and unreliable to test the program using an oracle derived from specifi- cation Gauss (due to the difficulty and inefficiency of ensuring that two systems of equations have the same set of solutions); it gives us a great return on invest- ment (in terms of required effort vs. achieved impact) to verify the correctness of

a candidate program with respect to specification Eq (since that can be done by merely checking that the system of equations is never modified except by repla- cing an equation by a linear combination of the equation with others) and to test it using an oracle derived from specification Tri (since this can be done by a simple scan of the lower half of the current matrix, without reference to any previously saved data). Note that as an equivalence relation, Eq is reflexive and transitive (hence satisfies the properties we have identified as making a specification ade- quate for proving correctness).

Hence when we say that the first step in designing a test driver is to decide what specification we want to test the program against, we mean it. The foregoing discus- sions illustrate in what sense and to what extent the software engineer does have some latitude in making this decision.