2 Software testing objectives Direct objectives

Frame 9.2 Software testing objectives Direct objectives

■ To identify and reveal as many errors as possible in the tested software. ■

To bring the tested software, after correction of the identified errors and retesting, to an acceptable level of quality.

■ To perform the required tests efficiently and effectively, within budgetary and scheduling limitations.

Indirect objective

■ To compile a record of software errors for use in error prevention (by corrective and preventive actions).

It should be noted that omission of the frequently stated goal “to prove that the software package is ready” is not accidental. This goal inherently contradicts the first operative objective mentioned, and may influence or, stated more accurately, bias the choice of tests and/or test cases. Myers (1979) neatly summarized the issue: “If your goal is to show the absence of errors you won’t discover many. If your goal is to show the presence of

182 The wording of the second objective reflects the fact that bug-free soft- ware is still a utopian aspiration. Therefore, we prefer the phrase “acceptable

9 level of quality”, meaning that a certain percentage of bugs, tolerable to the Softw

users, will remain unidentified upon installation of the software. This per- centage obviously varies by software package and user, but must be lower for

are tes

high failure risk packages.

ting – str

9.2 Software testing strategies

Although test methodologies may vary, often greatly, these are applied with-

a in the framework of two basic testing strategies: tegies

To test the software in its entirety, once the completed package is avail- able; otherwise known as “big bang testing”.

To test the software piecemeal, in modules, as they are completed (unit tests); then to test groups of tested modules integrated with newly com- pleted modules (integration tests). This process continues until all the package modules have been tested. Once this phase is completed, the entire package is tested as a whole (system test). This testing strategy is usually termed “incremental testing”.

Furthermore, incremental testing is also performed according to two basic strategies: bottom-up and top-down. Both incremental testing strategies assume that the software package is constructed of a hierarchy of software modules. In top-down testing, the first module tested is the main module, the highest level module in the software structure; the last modules to be tested are the lowest level modules. In bottom-up testing, the order of test- ing is reversed: the lowest level modules are tested first, with the main module tested last.

Figure 9.1 illustrates top-down and bottom-up testing of an identical software development project composed of 11 modules. In the upper part, Figure 9.1(a), the software development process and its subsequent testing are carried out bottom-up, in four stages, as follows:

Stage 1: Unit tests of modules 1 to 7.

Stage 2: Integration test A of modules 1 and 2, developed and tested in stage 1, and integrated with module 8, developed in the current stage.

Stage 3: Two separate integration tests, B, on modules 3, 4, 5 and 8, inte- grated with module 9, and C, for modules 6 and 7, integrated with module 10.

Stage 4: System test is performed after B and C have been integrated with module 11, developed in the current stage.

Stage 4

M11

Integration B Stage 3

M9

M10

Softw

Integration A

are tes

Stage 2

M8

ting str

Stage 1 M1

M7 Integration C

(a) Bottom-up testing

tegies

Integration D Integration C Integration B

Integration A

Stage 3 M8

Stage 5 M1

(b) Top-down testing Figure 9.1: Bottom-up (a) and top-down (b) testing – an illustration

In Figure 9.1(b), software development and testing are carried out top-down in six stages. It should be apparent that the change of testing strategy introduces major changes into the test schedule. The testing will be per- formed as follows:

Stage 1: Unit tests of module 11.

Stage 2: Integration test A of module 11 integrated with modules 9 and

10, developed in the current stage.

Stage 3: Integration test B of A integrated with module 8, developed in the current stage.

9 ■ Softw Stage 4: Integration test C of B integrated with modules 6 and 7, devel-

oped in the current stage.

Stage 5: Integration test D of C integrated with modules 1 and 2, devel-

are tes

oped in the current stage.

Stage 6: System test of D integrated with modules 3, 4 and 5, developed

ting – str

in the current stage. The incremental paths shown in Figure 9.1 are only two of many possible

paths. The path in the examples is “horizontally sequenced” (“breadth first”),

a although one could choose a path that is “vertically sequenced” (“depth tegies

first”). If we were to alter the horizontal path of the top-down sequence shown in Figure 9.1(b), to a vertical sequence, testing would be performed thus:

Stage 1: Unit tests of module 11.

Stage 2: Integration test A of the integration of module 11 with module

9, developed in the current stage.

Stage 3: Integration test B of A with module 8, developed in the current stage.

Stage 4: Integration test C of B with modules 1 and 2, developed in the current stage.

Stage 5: Integration test D of C with module 10, developed in the current stage.

Stage 6: Integration test E of integration D with modules 6 and 7, devel- oped in the current stage.

Stage 7: System test is performed after E has been integrated with mod- ules 3, 4 and 5, developed in the current stage.

Other path possibilities involve clustering of modules into one testing stage. For example, for the top-down path of Figure 9.1(b), one might cluster mod- ules 8, 1 and 2, and/or modules 10, 6 and 7.

Stubs and drivers for incremental testing Stubs and drivers are software replacement simulators required for modules not available when performing a unit or an integration test.

A stub (often termed a “dummy module”) replaces an unavailable lower level module, subordinate to the module tested. Stubs are required for top- down testing of incomplete systems. In this case, the stub provides the results of calculations the subordinate module, yet to be developed (coded), is designed to perform. For example, at stage 3 of the top-down example shown in Figure 9.1(b), upper module 9, which activates module 8, is avail- able; it has been tested and corrected at stage 2 of the testing. Stubs are required to substitute for the subordinate level modules 1 and 2, which have not been completed. This test setting is presented in Figure 9.2(a).

(a) Implementing top-down

tests (Stage 3 testing of the

M9

example shown in Figure 9.1)

Softw

M8

Module

on test

are tes

(b) Implementing bottom-up

Stub of

Stub of

tests (Stage 2 testing of the

M1

M2

ting str

example shown in Figure 9.1)

Driver of

M9

tegies

M8

Module

on test

M1

M2

Figure 9.2: Use of stubs and drivers for incremental testing – examples

Like a stub, a driver is a substitute module but of the upper level mod- ule that activates the module tested. The driver is passing the test data on to the tested module and accepting the results calculated by it. Drivers are required in bottom-up testing until the upper level modules are developed (coded). For example, at stage 2 testing of the bottom-up example shown in Figure 9.1(a), the lower level subordinate modules 1 and 2 are available; they have been tested and corrected at stage 1 of the testing. A driver is required to substitute for upper level module 9, which has not been completed. This test setting/scenario is shown in Figure 9.2(b).