DILEMMAS OF ORACLE DESIGN

11.1 DILEMMAS OF ORACLE DESIGN

All the test data in the world do not help us unless we have an oracle, a correct oracle, to reliably check whether the candidate program behaves according to its specifica- tion. The choice of an oracle is both critical and difficult.

• It is critical to have a reliable oracle because otherwise we run the risk of over- looking faults (if failures are not detected) and the risk of acting on faulty diag- noses (if correct behavior is reported to be incorrect).

• It is difficult and error prone to monitor the behavior of a program by having a human operator watch its inputs and outputs; but developing automated oracles poses challenges of its own, which we discuss in this section.

The general framework in which an oracle is invoked can be written as follows (where g is the program under test and S is the space of the program):

main () {s = testdata; s_init=s; g; // modifies s, keeps s_init intact assert(oracle(s_init,s));}

Software Testing: Concepts and Operations, First Edition. Ali Mili and Fairouz Tchier. © 2015 John Wiley & Sons, Inc. Published 2015 by John Wiley & Sons, Inc.

236 TEST ORACLE DESIGN

The design of the oracle is subject to the following criteria: • Simplicity. The oracle must be simple enough that we can ensure its reliability

with a great degree of confidence. In particular, it must be significantly simpler to write and analyze than candidate programs. If the oracle is as complex as the program being tested, then it is as likely to have faults as the candidate program; this, in turn, defeats the purpose of using the oracle to test the program.

• Strength. Ideally, we want the oracle to capture all the clauses of the requirements specification, so as to test all the relevant functional properties of the program.

But this may prove too complex (hence violating the first criterion), too ineffi- cient (in terms of resource usage), or too impractical (requiring a great deal of context, for example). Hence we may often have to settle for capturing a subset of the target requirements.

The criteria of strength and simplicity must be balanced against each other to reach

a tradeoff where we derive an oracle that is sufficiently strong to be useful, yet suf- ficiently simple to be reliable. This tradeoff is very easy when it is easier to check that

a final state is correct than to compute a correct final state. Consider for example a program that computes the roots of a quadratic equation: The space of this program is defined by real variables a, b, c that represent the coefficients of the quadratic equa- tion, and real variables x1and x2 that hold the roots. A specification of the quadratic equation may be written as follows:

R= s, s positive b 2 -4ac root x1 root x2 positive x2 -x1 This specification provides that the quadratic equation is assumed to have two dis-

tinct roots, and that candidate programs are expected to produce the smaller root in x1 and the larger root in x2 . To account for possible loss of precision in computer arith- metic, predicates positive and root are defined with respect to selected precision thresholds; for example,

• positive x ≡ x > ε , •

root x ≡ ax 2 + bx + c < ε , for some small positive constant, ε. Whereas the program computes the roots of the

quadratic equation constructively, the oracle merely checks that the computed values are indeed roots of the equation, and that they are distinct (hence the program is return- ing all the roots, rather than twice the same root). We can think of many other exam- ples where computing a result is significantly more complex than checking that a computed result is correct:

• Consider the problem of computing the roots of a polynomial of higher degree, or the roots of a functional that has no polynomial form altogether,

11.1 DILEMMAS OF ORACLE DESIGN 237

where no constructive solutions are known: candidate programs may com- pute the roots by some iterative method of successive approximations; yet all the oracle has to do is simply to check that the delivered values are indeed roots to the equation.

• Solving a large system of linear equations may be a complex affair (e.g., Gaussian elimination), where we have to worry not only about the correct- ness of the algorithm, but also about controlling round-off errors; yet check- ing that the solution is correct amounts to little more than multiplying a matrix by a vector.

• Computing the inverse of a N × N matrix involves complex calculations (e.g., N systems of linear equations), which are made all the more complex by the need to control round off errors; yet checking that the solution is correct amounts to little more than computing the product of two matrices.

• Computing the eigenvalues and eigenvectors of a N × N matrix involves solving polynomials of degree N, followed by solving systems of N linear equations, where again the control of runaway round-off errors is a major concern; yet checking that the solution is correct amounts to little more than multiplying a matrix by a vector.

In all these cases, it is possible to test a complex program using a simple, reli- able oracle. There are ample cases, however, where checking that a final program state is correct is not much easier than generating a correct final state; in such cases, we may have to settle for generating an oracle that tests only some aspects of the target requirements specification, deferring the other aspects to other ver- ification/quality assurance methods. As a simple illustrative example, consider the specification of a gcd program on natural variables x and y. The specification can

be written as:

R= s, s x > 0 y > 0 x = gcd x,y If we are given the initial values of variable x and y, and the final value x of

variable x, we have no easy way to tell whether x is the gcd of x and y, except possibly to compute the gcd of x and y independently and compare it with x . But doing so defeats the purpose of the oracle because it violates the requirement of simplicity: if the oracle is as complex as the program we are testing, we have no reason to trust the correctness of the oracle more than the correctness of the can- didate program. As a substitute, we may want to settle for checking that x is a common divisor of x and y, and renounce checking that it is the greatest common divisor. The specification that corresponds to such a scaled-down oracle can then

be written as:

R= s,s x > 0 y > 0 x modx = 0 y mod x = 0

238 TEST ORACLE DESIGN