ORACLES FOR STATE-BASED PRODUCTS

11.3 ORACLES FOR STATE-BASED PRODUCTS

In the previous section, we have discussed how to choose a specification against which we test a program and then how to derive a test oracle from a specification. In particular, we have focused our attention on two possible specifications:

A specification that is appropriate for acceptance testing, which is the weakest specification that a user is willing to accept as a criterion for considering that the contract (between the software provider and the software user) is fulfilled.

A specification that is appropriate for fault removal, which is the strongest pos- sible specification that the candidate program must fulfill, reflecting the intent of the programmer and the minute details of the program.

In this section, we consider software products that are based on an internal state. As we have seen in Chapter 4, such products can be specified by means of relations from input histories to outputs. The main advantage of this specification model is that it absolves us from talking about system states, leaving this matter as a design decision rather than a specification decision.

We consider a specification of the form (X, Y, R), where R is a relation from H = X ∗ to Y and we let g be a candidate implementation of the specification, in the form of

a class (in the object oriented programming (OOP) sense). If we are interested in test- ing class g for the purpose of fault removal, then we can specify each of its methods in terms of how it affects the system state and how it generates outputs accordingly.

11.3 ORACLES FOR STATE-BASED PRODUCTS 243

Each individual method can be viewed as a simple software component mapping an initial state into a final state; testing such components falls under the model we discussed in the previous section. Hence we focus our attention in this section on testing a state-based system against a specification of the form (X, Y, R), where R is a relation from H = X ∗ to Y. We assume that such specifications are represented by means of axioms and rules, as we discuss in Chapter 4. The question then becomes: how do we test a candidate implementation against such a specification? More specifically, how do we map such an axiomatic specification into an oracle? In the following section we discuss, in turn, how we generate oracles from axioms and how we generate oracles from rules. In the discussions that follow, we assume that implementation g is a class that has a method for each symbol in X; for the sake of simplicity, we assume that each method has the same name as the cor- responding symbol; we postfix method names with parentheses, even when they have no parameters.

11.3.1 From Axioms to Oracles

In the notation we introduced in Chapter 4, axioms have the form • Rh=y where h is an (elementary) input history that ends with a VX symbol (representing a

method that returns a value but does not change the state) and y is the corresponding output. History h can then be written as

• h = h vop, where vop is a VX symbol. In order to test implementation g against this axiom, we

write the following sequence of code:

vtype y; y=y0; // y0: output specified by the axiom

g.m1(); g.m2(); g.m3(); … g.mk(); // sequence h’

if (g.vop==y) {successfultest;} else {unsuccessfultest;}

where vtype is the data type returned by operation vop, and y is the output value provided by the axiom. As an illustration, we consider the following axioms from the stack specification discussed in Chapter 4 and generate an oracle for each.

• stack(init.top)=error.

itemtype y; y=error; // data type returned by top

g.init(); // sequence h’

if (g.top==y) {successfultest;} else {unsuccessfultest;}

244 TEST ORACLE DESIGN

• stack(init.h.push(a).top)=a.

itemtype y; y=a;// data type returned by top g.init(); g.xx(); g.yy(); … g.zz(); // any sequence of AX methods g.push(a);

// arbitrary a

if (g.top==y) {successfultest;} else {unsuccessfultest;}

• stack(init.size)=0.

int y; y=0; // data type returned by size

g.init(); // sequence h’

if (g.size==y) {successfultest;} else {unsuccessfultest;}

• stack(init..empty)=true.

bool y; y=true; // data type returned by empty

g.init(); // sequence h’

if (g.empty==y) {successfultest;} else {unsuccessfultest;}

• stack(init.push(a).empty)=false.

bool y; y=false; // data type returned by empty g.init(); g.push(a); // sequence h’, arbitrary a if (g.empty==y) {successfultest;} else {unsuccessfultest;}

11.3.2 From Rules to Oracles

The vast majority of rules in axiomatic specifications has the form of an equality between the images of two histories and expresses the property that two histories are equivalent for all subsequent input sequences. Typically the two histories are ordered (one is more complex than the other) and such rules can be used to infer the output of the complex history from the output of the simpler history. We focus on such rules first and then we consider other forms of rules.

Such rules can be written in generic form as:

hRhh=Rhh,

and can be interpreted as follows: for any input sequence h, the input sequence h . h yields the same outcome as the input sequence h . h; in other words, the histories

h and h are equivalent now (if h is empty) and at any time in the future (if h is not empty). Examples of such rules, in the stack specification given in Chapter 4, include the following:

11.3 ORACLES FOR STATE-BASED PRODUCTS 245

• stack(h .init.h) = stack(init.h). • stack(init.pop.h) = stack(init.h). • stack(init.h.push(a).pop.h+) = stack(init.h.h+). • stack(init.h.top.h+) = stack(init.h.h+). • stack(init.h.size.h+) = stack(init.h.h+). • stack(init.h.empty.h+) = stack(init.h.h+).

Some rules have h+ (nonempty sequences) instead of h (possibly empty), but they could be converted into rules with h by replacing h+ by xxx.h for each symbol xxx in

X. Hence we make no distinction between rules that end with an arbitrary history h and rules that end with a nonempty history h + . The same input sequence may lend itself to more than one rule, yielding a different oracle for each rule, as we discuss below. As an example, we consider the following input sequence:

• init.pop.push(a).size.push(b).pop.top.push(c). We leave it to the reader to check that this input sequence lends itself to the

following rules: • The Init-Pop Rule that reduces the sequence to

○ init.push(a).size.push(b).pop.top.push(c). • The VX Rule (for size and top) that reduces the sequence to ○ init.pop.push(a).push(b).pop.push(c). • The Push-Pop Rule that reduces the sequence to ○ init.pop.push(a).size.top.push(c).

Each one of these rules provides that the original sequence places the stack in the same state as the simpler input sequence; since we want to write the oracle by inspect- ing the specification rather than candidate implementations (and we want the same oracle to work for all possible implementations), we abstain from referring to states. The question that arises then is: how can we say that two states are identical if we cannot refer to the states? The answer is that, as an approximation, we consider that two states are identical if all the VX operations return the same values at these two states. Hence if we have a rule of the form:

• R(init.h .h) = R(init.h .h), where h is simpler than h , then the general template for an oracle that is derived from

the above rule is the following segment: g.init(); g.m1(); g.m2(); … g.mk(); // sequence init.h’

if oracle() {successfultest();} else {unsuccessfultest();}

246 TEST ORACLE DESIGN

where oracle() is defined as follows:

bool oracle() {vx1type vx1; vx2type vx2; vx3type vx3; // VX types vx1 = g.vop1(); vx2 = g.vop2(); vx3 = g.vop3();

// storing the current state, following init.h’

g.init(); g.m1’(); g.m2’(); … g.mh (); // sequence

init.h”

return ((vx1==g.vop1()) && (vx2==g.vop2()) && (vx3==g.vop3()));}

As an illustration, we consider the following input sequence: • init.pop.push(a).size.push(b).pop.top.push(c). and we generate oracles to test it, according to various applicable rules. • The Init-Pop Rule. In order to test this sequence against the Init-Pop rule, we

apply the code pattern shown above, which we specialize to this rule.

itemtype a, b, c, v; int n; // working variables g.init(); g.pop(); g.push(a); n=g.size(); g.push(b); g.pop(); v=g.top(); g.push(c); if oracleinitpop(){successfultest();}

else {unsuccessfultest();}

where we define oracle() as follows:

bool oracleinitpop() {bool sempty; int ssize; itemtype stop;

// VX values sempty=g.empty(); ssize=g.size(); stop=g.top(); g.init(); g.push(a); n=g.size(); g.push(b); g.pop(); v=g.top(); g.push(c); return ((sempty == g.empty()) && (ssize==g.size) &&

(stop==g.top()));}

• The VX Rule (for size). In order to test this sequence against the VX rule for size,

we apply the code pattern shown above, which we specialize to this rule.

itemtype a, b, c, v; int n; // working variables g.init(); g.pop(); g.push(a); n=g.size(); g.push(b); g.pop(); v=g.top(); g.push(c); if oracleVXsize(){successfultest();}

else {unsuccessfultest();}

11.3 ORACLES FOR STATE-BASED PRODUCTS 247

where we define oracle() as follows:

bool oracleVXsize() {bool sempty; int ssize; itemtype stop;

// VX values sempty=g.empty(); ssize=g.size(); stop=g.top(); g.init(); g.pop(); g.push(a); g.push(b); g.pop(); v=g.top(); g.push(c); return ((sempty == g.empty()) && (ssize==g.size) &&

(stop==g.top()));}

• The VX Rule (for top). In order to test this sequence against the VX rule for top, we apply the code pattern shown above, which we specialize to

this rule.

itemtype a, b, c, v; int n; // working variables g.init(); g.pop(); g.push(a); n=g.size(); g.push(b); g.pop(); v=g.top(); g.push(c); if oracleVXtop(){successfultest();} else {unsuccessfultest();}

where we define oracle() as follows:

bool oracleVXtop() {bool sempty; int ssize; itemtype stop;

// VX values sempty=g.empty(); ssize=g.size(); stop=g.top(); g.init(); g.pop(); g.push(a); n=g.size(); g.push(b); g.pop(); g.push(c); return ((sempty == g.empty()) && (ssize==g.size) &&

(stop==g.top()));}

• The Push-Pop Rule. In order to test this sequence against the Push-Pop rule, we apply the code pattern shown above, which we specialize to this rule.

itemtype a, b, c, v; int n; // working variables g.init(); g.pop(); g.push(a); n=g.size(); g.push(b); g.pop(); v=g.top(); g.push(c); if oraclepushpop(){successfultest();}

else {unsuccessfultest();}

where we define oracle() as follows:

bool oraclepushpop() {bool sempty; int ssize; itemtype stop;

// VX values sempty=g.empty(); ssize=g.size(); stop=g.top();

248 TEST ORACLE DESIGN

g.init(); g.pop(); g.push(a); n=g.size(); v=g.top(); g.push(c); return ((sempty == g.empty()) && (ssize==g.size) &&

(stop==g.top()));}

Whereas some rules provide that distinct input histories are equivalent, other rules describe how the value of a VX method depends on the structure of the input history. We write their general form as follows:

• C(R(init.h.vop), R(init.h .vop)), where C is a binary predicate between values returned by vop. As such, these rules are

potentially applicable to any input sequence that ends with a vop symbol. The general format of their oracle can be written as follows, where we assume that sequence h is more complex than sequence h ’:

g.init(); g.m1(); g.m2(); … g.mk(); // sequence init.h if oraclevoprule() {successfultest();} else {unsuccessfultest();}

where oraclevoprule() is defined as follows:

bool oraclevoprule() {vxtype vx; vx = g.vop();

// store R(init.h.vop)

g.init(); g.m1’(); g.m2’(); … g.mh (); // sequence init.h’

return (C(vx, g.vop);}

As an illustration, we consider the following input sequence • init.pop.push(a).push(b).size.push(a).top.push(c).size

and we generate an oracle for it on the basis of the size rule. As we recall, the Size Rule of the stack specification provides:

• stack(init.h.push(a).size) = 1 + stack(init.h.size). We find:

itemtype a, b, c, v; int n; // working variables g.init(); g.pop(); g.push(a); g.push(b); n=g.size(); g.push(a); v=g.top(); g.push(c);

// sequence init.h if oraclesize() {successfultest();} else {unsuccessfultest();}

11.3 ORACLES FOR STATE-BASED PRODUCTS 249

where oraclesize() is defined as follows:

bool oraclesize() {int ssize; ssize = g.size();

// store R(init.h.vop) g.init(); g.pop(); g.push(a); g.push(b); n=g.size(); g.push(a); v=g.top();

// sequence init.h’

return (ssize==g.size()+1);}

To illustrate the generation of oracles from the empty rules, we consider the fol- lowing input sequence:

• init.pop.empty.push(a).push(b).size.pop.push(a).size.push(c).empty. Using the two empty rules (copied from Chapter 4): • stack(init.h.push(a).h .empty)

stack(init.h.h .empty) • stack(init.h.h .empty)

stack(init.h.pop.h .empty)

From these rules, we generate the following oracles:

itemtype a, b, c, v; int n; bool e; // working variables g.init(); g.pop(); e=g.empty(); g.push(a); g.push(b); n=g.size(); g.pop; g.push(a); n=g.size(); g.push(c); // sequence init.h if oracleempty1() {successfultest();}

else {unsuccessfultest();}

where oracleempty1() is defined as follows:

bool oracleempty1() {bool sempty; sempty = g.empty();

// store R(init.h.vop) g.init(); g.pop(); e=g.empty(); g.push(a); g.push(b); n=g.size();

g.pop; n=g.size(); g.push(c); // sequence init.h’

return (!(sempty) || (g.empty()));}

// stack(init.h.push(a).h .empty) →

// stack(init.h.h .empty)

Whereas the first empty rule provides that removing a push operation makes the stack more empty (so to speak), the second empty rule provides that adding a pop operation also makes the stack more empty. Its oracle can be defined as follows, for the selected input sequence:

250 TEST ORACLE DESIGN

itemtype a, b, c, v; int n; bool e; // working variables g.init(); g.pop(); e=g.empty(); g.push(a); g.push(b); n=g.size(); g.push(a); n=g.size(); g.push(c); // sequence init.h if oracleempty2() {successfultest();}

else {unsuccessfultest();}

where oracleempty2() is defined as follows:

bool oracleempty2() {bool sempty; sempty = g.empty();

// store R(init.h.vop) g.init(); g.pop(); e=g.empty(); g.push(a); g.push(b); n=g.size();

g.pop(); g.push(a); n=g.size(); g.push(c); // sequence init.h’

return (!(sempty) || (g.empty()));}

// stack(init.h.h .empty) →

// stack (init.h.pop.h .empty)

So far we have used axioms and rules to generate test data and design oracles; but in fact, test data generation ought to be driven by coverage criteria. In Chapter 9, we had explored ways to generate test data for state-based software products, using the criteria that all states and all state transitions be visited at least once. In Chapter 12, we will see how the data generated in Chapter 9 can be combined with the oracles introduced herein to build test drivers.