LOGICAL CLAIMS

13.1 LOGICAL CLAIMS

13.1.1 Concrete Testing

We consider the scenario where we run a candidate program g on space S against an oracle derived from specification R, and we find that the program runs successfully on all elements of the test data set T; then we ask the question: what claims can

be made about program g? Before we answer this question, we need to specify precisely what we mean by runs successfully (in reference to a program under test). The first interpretation we adopt is that whenever the test oracle is executed, it returns true; we have no assurance that the oracle will ever be invoked (in particular, if the program under test does not terminate), but we know that if it is ever invoked, it returns true. Under this interpretation of a successful test, we observe that in order for the oracle to be invoked, the program g has to terminate its execution, that is, the initial state has to be an element of dom(G). Hence we write:

t Tt dom G

oracle t,G t

We rewrite this expression, replacing oracle(t, G(t)) by its expression as a function of the specification R (refer to the derivation of oracles from specifications, discussed in Chapter 9):

t,G t R By logical simplification, we transform this as follows: t Tt dom G t dom R

t Tt dom G

t dom R

t,G t R By changing the quantification, we rewrite this as follows: t St T

t,G t R By logical simplification, we rewrite this as follows: t St dom G t T t dom R

t dom G t dom R

t,G t R By tautology, we rewrite this as follows: t St dom G t T t dom R

t T t,G t R If we let R be the pre-restriction of R to T, we can write this as: t St dom G t dom R

t,G t R

282 TEST OUTCOME ANALYSIS

In other words, we have proven that program g is partially correct with respect to R , the pre-restriction of R to T. Whence the following proposition:

If we test program g on space S using an oracle derived from specification R on test data T and the program runs successfully for all the test data in T, we can conclude that pro- gram g is partially correct with respect to R , the pre-restriction of R to T.

Because the domain of R is typically a very small subset of the domain of R, this property is in effect very weak, in general. Yet, logically, this is all we can claim from the successful execution of the test; it is possible that the successful test gives us some confidence in the quality/reliability of the software product, but that is not

a logical property.

13.1.2 Symbolic Testing

Symbolic testing is, in general very complex, for not only does it involve complex control structures such as loops, loops with complex loop bodies, nested loops, and so on, but it may also involve complex data structures. In order to model complex data structures, we need a rich vocabulary of relevant abstractions, as well as an adequate axiomatization pertaining to these abstractions.

Not only is symbolic testing complex and error prone, it is often wasteful and inefficient: Indeed, it is very common for programs to be much more refined than the specification they are intended to satisfy; hence by trying to compute the function of a program in all its minute detail, we may be dealing with functional detail that is ultimately irrelevant to the specification that the program is written to satisfy and even more irrelevant to the specification that the program is being tested against. Consider the example of a binary search, which searches an item x in an ordered array a by means of two indices l and h (for low and high), and imagine having to characterize the final values of l and h as a function of a and x; it is very difficult, as it depends on very minute details of the program (whether strict or loose inequalities are used) on whether x is in a or not, and on the position of x with respect to cells of a; and yet it is also of little relevance as the most important outcome of the search is to determine whether x is in a, and eventually at what location. Performing the symbolic execution of a binary search just to check whether it satisfies the specification of a search is like going through an 8000 feet pass to climb a 3000 feet peak.

If we do overcome the complexity, the error proneness, and the possibility of excessive (and irrelevant) detail that come with full blown symbolic execution of a program, then our reward is that we can prove any property we wish about the program in question, with respect to any specification we wish to consider. As a reminder, we present below a brief summary of the properties that we may want to prove about a candidate program g on space S once we have computed its function G.

13.1 LOGICAL CLAIMS 283

Given a program g on space S whose function is G, and given a specification R on S, • Program g is correct with respect to R if and only if:

dom R G = dom R

• Program g is partially correct with respect to R if and only if:

dom R G = dom R dom G

• Program g is defined with respect to R if and only if:

dom R dom G = dom R

13.1.3 Concolic Testing

From a cursory analysis, it appears that • Concrete testing, to the extent that it is carried out without fault removal, typi-

cally produces a very weak logical statement, pertaining to partial correctness with respect to a typically weak specification.

• Symbolic Execution, to the extent that it is deployed in full, enables us to prove any property we wish with respect to any specification, but it is very difficult to deploy, is very prone to errors, and forces us to deal with functional detail that may well be irrelevant to whatever property we wish to prove.

These two methods can be compared and contrasted in the following table.

Method Process

Assumption

Scope

Assessment Advantages

Drawbacks

Concrete Dynamic

Ease of Weak claims testing

Faithful

No

deployment and analysis

execution

reflection of

limitation

actual operating conditions

Symbolic Static

Arbitrarily Difficulty/ execution analysis of

Rules used for

Limited to

strong claims complexity/ the source

symbolic

aspects of

with respect to error code, impact

execution

programs

arbitrary proneness of of execution

consistent with that are

specifications deployment on symbolic

actual

easy to

behavior of

model

data

computer

284 TEST OUTCOME ANALYSIS

Concolic testing is a technique that combines concrete testing and symbolic testing in an effort to achieve a greater return on investment than each method taken individually; the name concolic is in fact derived from combining the beginning of concrete with the ending of symbolic.

Concolic testing is essentially a form of concrete testing, where the program is executed on actual test data, but it uses symbolic execution techniques to compute the path conditions of various execution paths through the program, hence improves the coverage of the test. By focusing on execution paths rather than closed form control structures, and by targeting the derivation of path conditions rather than fully detailed path functions, concolic testing obviates the main difficulties of symbolic test- ing. On the other hand, by taking a systematic approach to the derivation of path con- ditions, it aims to achieve a degree of efficiency by ensuring that each new concrete test data exercises a new execution path, rather than a previously visited execution path.