TEST GENERATION FOR STATE BASED SYSTEMS

9.3 TEST GENERATION FOR STATE BASED SYSTEMS

A state-based system is a system whose output depends not only on its (current) input but also on an internal state, which is itself dependent on past inputs. As we discussed in Chapter 4, such systems can be specified in relational form by means of the follow- ing artifacts:

• An input space X, from which we derive a set H of sequences of X • An output space Y •

A relation R from H to Y. It is common to specify such systems in a way that makes their internal state

explicit, by means of the following artifacts: • An input space X

• An output space Y • An internal state space, Σ • An output function, ω from X × Σ to Y •

A state transition function, θ from X × Σ to Σ. We argue that it is possible to map a specification of the form (X,Y,R) into a

specification of the form (X,Y,Σ,ω,θ), as follows: •

X and Y are preserved.

9.3 TEST GENERATION FOR STATE BASED SYSTEMS 177

• We define the equivalence relation E on H (=X*) by: E= h,h h,yhh,y R

h h ,y R In other words, the pair (h, h ) is in E if and only if they are equivalent histories, in

the sense that they produce the same output now (for h empty) and in the future (for h not empty). Using this equivalence relation, we define the internal state space of the system as the quotient (H/E), that is, the set of equivalence classes of

H modulo E. Whereas the output of the system depends on its input history, this does not mean that the system must remember its input history in all its detail; rather it must only remember the equivalence class of its input history; this is exactly the internal state of the system.

• The output function is defined as follows:

h σ h x, y R Given a current internal state σ and a current input symbol x, we let h be an arbitrary

ω = x,σ , y

element of σ, and we let y be an image of (h. x) by R. Because σ is an equivalence class of H modulo E, the choice of h within σ is immaterial, by definition.

• The state transition function is defined as follows:

θ = x,σ ,σ hσσ=Ehx,

where E(h. x) designates the set of images of (h. x) by relation E, which is the equivalence class of (h. x) modulo E.

We have made the observation that even software systems that carry an internal state can, in theory, be modeled by a mere relation from an input set (structured as the set of lists formed from the input space) to an output space. This observation is important because it means that we can, in theory, select test data for such a system in the same way as we do for a system that has no internal state, except for considering the special structure of the input set.

As an illustration of the mapping between the (X, Y, R) model of specification and the (X, Y, Σ, ω, θ) model, we consider the specification of the stack given in Chapter 4 in the (X, Y, R) format and we discuss (informally) the terms of its (X, Y, Σ, ω, θ) specification.

X: The input space of the stack is defined as

X = init, pop, top, size, empty push × itemtype, where itemtype is the data type of the items we envision to store in the stack. We

distinguish, in set X, between inputs that affect the state of the stack (namely: AX = init, push, pop and inputs that merely report on it (namely: VX =

top, size, empty ). ◦ From the set of inputs X, we build the set of input histories, H, defined as, H = X ∗ .

178 FUNCTIONAL CRITERIA

• Y: The output space includes all the values returned by all the inputs of VX, namely:

Y = itemtype error integer boolean •Σ : The equivalence relation E includes two histories h and h if and only if any

subsequent history h produces the same outcomes. Examples of histories that are equivalent modulo relation E include, for instance:

◦ pop.top.init.pop.push(a). ◦ init.push(a). ◦ init.pop.top.size.push(a).empty.push(b).top.pop. ◦ init.push(a).push(b).push(c).push(d).top.pop.pop.pop.

All these histories belong to the same equivalence class modulo E, that is, they are part of the same state; we represent this state by its simplest element, which is

◦ init.push(a). •ω : The output function of the stack maps an internal state σ and a current input x into an output y by virtue of the equation:

h x,y R,

where h is an element of σ. If we consider the internal state σ that is the equiv- alence class of init.push(a), and we let h = init.push(a) be a representative ele- ment of state σ, then the output of the stack for internal state σ and input x is characterized by the following equation:

init push a x,y stack

For x = top, y = ω σ, x = a. For x = size, y = ω σ, x = 1 For x = empty, y = ω σ, x = false. For x = pop, push(_), or init, y = ω σ,x is arbitrary (an arbitrary element of Y), since these are elements of AX that change the state of the stack but generate no output.

•θ : The state transition function of the stack maps an internal state σ and a current input x into a new internal state σ by virtue of the equation:

σ =Ehx,

where h is an element of σ. If we consider the internal state σ that is the equiv- alence class of init.push(a), and we let h = init.push(a) be a representative ele- ment of state σ, then the next state of the stack for internal state σ and input x is characterized by the following equation:

σ = E init push a x

9.3 TEST GENERATION FOR STATE BASED SYSTEMS 179 The following table shows the result of applying function θ to the init.push(a)

and to the input sympols of AX.

x θ (init.push(a), x)

init init push (b)

init.push (a).push(b) pop

init

Because formally, state-based systems can be specified by binary relations, then in theory we can apply to them any criterion that we apply to simple input/output systems which are also specified by (homogeneous) binary relations. But in recognition of the special structure that implementations of state-based systems have (in terms of an input space, an internal space, and an output space), it makes sense to formulate test data selection criteria accordingly. Among the test data selection criteria that we may adopt, we mention the following:

• Select Test Data to Visit Every State. The question that arises with this criterion is that the specification of the product is defined in terms of input sequences

and outputs and has no cognizance of internal states; also, even if we derive the states from the trace specifications as we have discussed above, the implementation does not have to adopt these very states. We obviate this dilemma by considering that, for an outside observer, a state is characterized by the values of all the VX operations at that state. In other words, to check that the software product is in the right state (without looking at the internal data of the product) with respect to the specification, we generate the sequence of operations that leads us to that state, and then we append to it in turn all the

VX operations of the specification. The procedure that generates data according to this criterion proceeds as follows:

◦ Partition the set of histories into equivalence classes modulo the equivalence relation E defined above.

◦ Choose an element of each equivalence class, and add to it, in turn, all the VX operations of the specification.

In practice, relation E may have an infinity of equivalence classes, forcing us to take supersets of it by merging in the same equivalence class input sequences for which we suspect that candidate programs have the same behavior (i.e., they succeed for both or fail for both); we will see examples of this merger below. This criterion is illustrated in the Figure 9.2, where the quadrants represent

equivalence classes of H* modulo E, h 1 ,h 2 , …, h 8 represent the elements of the equivalence classes, and v 1 ,v 2 , …, v i represent the VX operations of the specification.

180 FUNCTIONAL CRITERIA

h 8 h 3 .v 1 h 3 .v 2 ... h 3 . v i

h 7 h 4 .v 1 h 4 .v 2 ... h 4 .v i

Figure 9.2 Visiting each state.

• Select Test Data to Perform Every State Transition. We perform state transitions by appending AX operations to existing sequences for all the AX operations of the specification, then appending in turn all the VX operations to identify the new state, and check its validity with respect to the specification. This criterion is illustrated in the Figure 9.3, where the quadrants represent equivalence classes

of H* modulo E, h 1 ,h 2 , …, h 8 represent the elements of the equivalence classes, and v 1 ,v 2 , …, v i represent the VX operations of the specification, and a 1 ,a 2 , …, a j represent its AX operations.

From this discussion we infer that the set of test data that we ought to generate to visit all the states and traverse all the state transitions is the union of two terms:

1. X ∗

E × VX (by virtue of the criterion of visiting all the states) and

2. X ∗

E × AX × VX (by virtue of traversing all the state transitions).

9.3 TEST GENERATION FOR STATE BASED SYSTEMS 181

h 3 .a 2 .v 1 h 3 .a 2 .v 2 ... h 3 .a 2 .v i H

h 7 h 3 .a 1 .v 1 h 3 .a 1 .v 2 ... h 3 .a 1 .v i

Figure 9.3 Visiting each state transition.

Whence the following test data generation criterion:

If we want to generate test data for a program p to satisfy the logical requirement with respect to a state-based specification R from input space X to output space Y, we let E be the quivalence relation defined on H = X ∗ by

h h',y R, and we generate test data T as: T=X ∗

Whereas the formula of T (above) provides for composing sets (X*/E), AX, and VX by means of the Cartesian product, we in fact mean to obtain a set of sequences

182 FUNCTIONAL CRITERIA

constructed by concatenation of elements of these sets; this is illustrated in the example below.

While AX and VX are usually small, (X*/E) is usually infinite, requiring some additional assumptions to replace it by a finite/small subset. As an illustration, we con- sider again the stack specification, and we argue that two input sequences of this specification are equivalent if and only if they are reducible to the same sequence of the form:

init push t 1 push t 2 push t 3 … push t k

This set of states is clearly infinite, given that k can be arbitrarily large, and t 1 ,t 2 ,t 3 , … ,t k , can take arbitrary values. If we assume that the success or failure of a stack implementation will not depend, perhaps, on the value that we store in the stack at any position, then we can represent states as follows, where push − n represents a

sequence of n instances of push() for arbitrary values on the stack, then we can write the states of the stack as:

init push − k

This set is also infinite, since k may take arbitrary natural values; hence further assumptions are needed. We may assume (with some risk) that any implementation of the stack that works for stacks of size 3 works for stacks greater than 3, provided we have assurances that overflow is not an issue (either because our stack size is bounded or because our supply of memory is unbounded). Hence (X*/E) can be approximated with the following set:

init, init push , init push push , init push push push This gives the following test data, which we present in two different tables that

correspond to the two terms of the union in the formula of T:

(X*/E)

init

init.push (–)

init.push ( ). push( ) init.push ( ). push( ).

push ()

VX top

init.top

init.push (a). init.push (–).

init.push (–).push(–).

top

push (a).top

push (a).top

size

init.size

init.push (a). init.push (–).

init.push (–).push(–).

size

push (a).size

push (a).size

empty

init.empty

init.push (a). init.push (–).

init.push (–).push(–).

empty

push (a).empty

push (a).empty

9.3 TEST GENERATION FOR STATE BASED SYSTEMS 183

AX VX X ∗ E × AX × VX

init

init.push (–)

init.push ( ). init.push ( ).

push ()

push ( ). push( )

init top

init.push (–). init.push (–). top

init.init.

init.push (a).

init.top

push (a).init. push (–).push(a).

top

init.top

size

init.init.

init.push (–). init.push (–). size

init.push (a).

init.size

push (a).init. push (–).push(a).

size

init.size

empty

init.init.

init.push (–). init.push (–). empty

init.push (a).

init.empty

push (a).init. push (–).push(a).

empty

init.empty

push top

init.push (b).

init.push (–). init.push (–). top

init.push (a).

push (b).top

push (a).

push (–).push(a). push (b).top

push (b).top

size

init.push (–). init.push (–).push push (b).size

init

init.push (a).

push (b).size

push (a).

(–).push(a).push push (b).size

(b).size

empty

init.push (b).

init.push (–). init.push (–). empty

init.push (a).

push (b).

push (a).

push (–).push(a).

empty

push (b).empty push (b).empty

pop top

init.pop.top

init.push (a).

init.push (–). init.push (–).push

pop.top

push (a).pop. (–).push(a).

top

pop.top

size

init.pop.size

init.push (a).

init.push (–). init.push (–).

pop.size

push (a).pop. push (–).push(a).

size

pop.size

empty

init.push (–). init.push (–). empty

init.pop.

init.push (a).

pop.empty

push (a).pop. push (–).push(a).

empty

pop.empty

How do we test an implementation using this test data? Simply by declaring an instance of the class that implements the specification R and by writing sequences of method calls that represent the data shown in this table. For example,

#include <iostream>

#include “stack.cpp”

using namespace std; /* state variables */ stack s; itemtype t; int z; bool e; // to store outputs of VX operations

184 FUNCTIONAL CRITERIA

int main () { s.init(); t=s.top(); cout << t; // test datum: init.top // … … … s.init(); s.push(c); s.push(d); s.push(a); s.push(b); // test datum: init.push(_).push(_).push(a).pop.empty

As for the question of whether these executions took place according to the specification, it will be addressed when we discuss oracle design, in Chapter 11.