RANDOM TESTING

9.7 RANDOM TESTING

In the random testing approach, test inputs are selected randomly from the input domain of the system. We explain the idea of random testing with a simple example √

of computing X , where X is an integer. Suppose that the system will be used in an environment where the input X takes on all values from the interval [1, 10 8 ] with equal likelihood and that the result must be accurate to within 2 × 10 −4 . In order to test this program, one can generate uniformly distributed pseudorandom

integers within the interval [1, 10 8 ]. Then we execute the program on each of these inputs t and obtain the output z t . For each t, we compute z t and z 2 t and compare z 2 t with t. If any of the outputs fails to be within 2 × 10 −4 of the desired results, the

program must be fixed and the test repeated. Based on the above example, random testing can be summarized as a four-step procedure [19]:

Step 1: The input domain is identified. Step 2: Test inputs are selected independently from the domain. Step 3: The system under test is executed on these inputs. The inputs constitute

a random test set. Step 4: The results are compared to the system specification. The test is a failure

if any input leads to incorrect results; otherwise it is a success. Random testing corresponds to simple random sampling from the input

domain [20]. If the distribution of the selected inputs (step 2) is the same as the distribution of inputs in the expected-use scenario (the operational profile), then statistical estimates for the reliability of the program can be obtained from test outcomes. Random testing gives us an advantage of easily estimating software reliability from test outcomes. Test inputs are randomly generated according to an

253 operational profile, and failure times are recorded. The data obtained from random

9.7 RANDOM TESTING

testing can then be used to estimate reliability. Other testing methods cannot be used in this way to estimate software reliability.

A large number of test inputs are typically required to get meaningful sta- tistical results. Consequently, some kind of automation is required to generate a large number of inputs for random testing. For effective generation of a large set of inputs for statistical estimation, one needs to know the operational profile of the system. On the other hand, the expected results (step 4) are usually not obvious. Computing expected outcomes becomes difficult if the inputs are randomly chosen. Therefore, the technique requires good test oracles to ensure the adequate evalu- ation of test results. A test oracle is a mechanism that verifies the correctness of program outputs. The term test oracle was coined by William E. Howden [21]. An oracle provides a method to (i) generate expected results for the test inputs and (ii) compare the expected results with the actual results of execution of the implemen- tation under test (IUT). In other words, it consists of two parts: a result generator to obtain expected results and a comparator. Four common types of oracles are as follows [22]:

• Perfect Oracle: In this scheme, the system (IUT) is tested in parallel with a trusted system that accepts every input specified for the IUT and always produces the correct result. A trusted system is a defect-free version of the IUT.

• Gold Standard Oracle:

A previous versions of an existing application

system is used to generate expected results, as shown in Figure 9.9. • Parametric Oracle: An algorithm is used to extract some parameters from

the actual outputs and compare them with the expected parameter values, as shown in Figure 9.10.

• Statistical Oracle: It is a special case of a parametric oracle. In a statis- tical oracle, statistical characteristics of the actual test results are verified.

Golden implementation

Golden result

Test case

Test input

input

Comparator

Pass or fail

Implementation

Actual result

under test

Figure 9.9 Gold standard oracle.

Pass or fail Test case

Test input

input Actual parameters

Implementation Actual result

Output

converter Figure 9.10 Parametric oracle.

under test

254 CHAPTER 9 FUNCTIONAL TESTING

Moreover, the actual test results are random in the case of randomized software and random testing. Therefore, it is not possible to give an exact expected value. In this scheme, the expected statistical characteristic is compared with the actual test results. A statistical oracle does not check the actual output but only some characteristics of it. Therefore, a statistical oracle cannot decide whether or not a single test case passes. If a failure occurs, identification of the failure cannot be attributed to the success of

a single test case; rather, the entire group of test cases is credited with the success. The decision of a statistical oracle is not always correct. In other words, at best the probability for a correct decision can be given. Figure 9.11 shows the structure of a statistical oracle [23]. It consists of a statistical analyzer and a comparator. The statistical analyzer computes var- ious characteristics that may be modeled as random variables and delivers it to the comparator. The comparator computes the empirical sample mean and the empirical sample variance of its inputs. Furthermore, expected val- ues and properties of the characteristics are computed by the comparator based on the distributional parameters of the random test input.

Adaptive Random Testing In adaptive random testing the test inputs are selected from the randomly generated set in such a way that these are evenly spread over the entire input domain. The goal is to select a small number of test inputs to detect the first failure. A number of random test inputs are generated, then the “best” one among them is selected. We need to make sure the selected new test input is not too close to any of the previously selected ones. That is, the selected test inputs should be distributed as spaced out as possible.

An adaptive random testing technique proposed by Chen et al. [24] keeps two sets, namely, T and C, as follows:

• The executed set T is the set of distinct test inputs that have been selected and executed without revealing any failure.

• The candidate set C is a set of test inputs that are randomly selected. Initially the set T is empty, and the first test input is randomly selected from

the input domain. The set T is then incrementally updated with the selected ele- ment from the set C and executed until a failure is revealed. From the set C, an element that is farthest away from all the elements in the set T is selected as the next test input. The criterion “farthest away” can be defined as follows. Let

Comparator Random

Pass or fail

Distributional parameters

test input

Characteristics

generator

Implementation Actual results

Statistical

Test case

under test

analyzer

inputs

Figure 9.11 Statistical oracle.

255 T = {t 1 ,t 2 ,...,t n } be the executed set and C = {c 1 ,c 2 ,...,c k } be the candidate

9.8 ERROR GUESSING

set such that C ∩ T = φ. The criterion is to select the element c h such that, for all

min dist(c h ,t i

)≥ min dist(c j ,t i ) i=1

i=1

where dist is defined as the Euclidean distance. In an m-dimensional input domain, for inputs a = (a 1 ,a 2 ,...,a m ) and b = (b 1 ,b 2 ,...,b m ) , dist(a, b) = m

i=1 (a i −b i ) 2 . The rationale of this criterion is to evenly spread the test input by maximizing the minimum distance between the next test input and the already

executed test cases. It should be noted that there are various ways to construct the candidate set

C giving rise to various versions of adaptive random testing. For example, a new candidate set can be constructed of size 10 each time a test input is selected. Empir- ical study shows that adaptive random testing does outperform ordinary random testing by 50% [24]. In the above comparison the performance metric is the size of the test suite used to detect the first failure.