Output correctness tests are, in most cases, among the tests that consume the Softw

9 Output correctness tests are, in most cases, among the tests that consume the Softw

greater part of testing resources. In those frequent cases where output correctness tests alone are performed, they consume all testing resources. Implementation of

are tes other classes of tests depends on the nature of the software product and its future

users as well as on the developer’s procedures and decisions. The output correctness tests apply the concept of test cases. Improved

ting – str choice of test cases can be achieved by the efficient use of equivalence class

partitioning, a method to be discussed here. Equivalence class partitioning is a black box method aimed at increasing the efficiency of testing and, at the same time, improving coverage of poten-

a tial error conditions. An equivalence class (EC) is a set of input variable tegies

values that produce the same output results or that are processed identical- ly. EC boundaries are defined by a single numeric or alphabetic value, a group of numeric or alphabetic values, a range of values, and so on. An EC that contains only valid states is defined as a “valid EC”, whereas an EC that contains only invalid states is defined as an “invalid EC”. In cases where a program’s input is provided by several variables, valid and invalid ECs should be defined for each variable.

According to the equivalence class partitioning method, test cases are defined so that each valid EC and each invalid EC are included in at least one test case. Test cases are defined separately for the valid and invalid ECs. In defining a test case for the valid ECs, we try to cover as many as possible “new” ECs (i.e., classes not included in any of the former test cases) in that same test case. Test cases are added as long as there are uncovered ECs. As

a result of this process, the total number of required test cases to cover the valid ECs is equal to and in most cases significantly below the number of valid ECs. Note that in defining invalid ECs, we must assign one test case to each “new” invalid EC, as only one invalid EC can be included in a test case.

A test case that includes more than one invalid EC may not allow the tester to distinguish between the program’s separate reactions to each of the invalid ECs. Hence, the number of test cases required for the invalid ECs equals the number of invalid ECs.

Compared to the use of a random sample of test cases, equivalence class- es save testing resources because they eliminate duplication of the test cases defined for each EC. Importantly, as the equivalence class method is a black box method, equivalence class partitioning is based on software specification documentation, not on the code. Systematic constructing of equivalence class- es for a program’s input variables increases the coverage of possible valid and error conditions of input and thus improves the testing plan’s effectiveness. Further improvement of testing effectiveness and efficiency is achieved by test- ing for the boundary values of ECs, a subject we elaborate next.

Test cases and boundary values According to the definition of equivalence classes, one test case should be sufficient for each class. However, when equivalence classes cover a range of Test cases and boundary values According to the definition of equivalence classes, one test case should be sufficient for each class. However, when equivalence classes cover a range of

9.5 Bl

these cases, the preparation of three test cases – for mid range, lower bound-

ary and upper boundary values – is recommended.

ac Example – the Golden Splash Swimming Center

bo

The following example illustrates the definition of (valid and invalid)

equivalence classes and the corresponding test case values. The software

te

module in question calculates entrance ticket prices for the Golden Splash

sting

Swimming Center.

The Center’s ticket price depends on four variables: day (weekday, week- end), visitor’s status (OT = one time, M = member), entry hour (6.00–19.00, 19.01–24.00) and visitor’s age (up to 16, 16.01–60, 60.01–120). The entrance ticket price table is shown in Table 9.7.

The equivalence classes and the corresponding test case values for the above example are presented in Tables 9.8 and 9.9.

A total of 15 ECs were defined for the ticket price module: nine valid ECs and six invalid ECs. The test cases that correspond to these ECs apply the representing values listed in Table 9.8. The test cases for these ECs, including their boundary values, are presented in Table 9.9.

A total of 15 test cases cover all the defined ECs, including the respec- tive EC boundary values:

Three test cases for the valid ECs (for our example a total of nine valid ECs were defined).

Six test cases for the boundary value ECs (in our example, boundary test- ing is applicable for only two of the four input variables).

Six test cases for invalid ECs (for our example a total of six invalid ECs were defined).

Table 9.7: Entrance ticket price table – the Golden Splash Swimming Center

Mon, Tue, Wed, Thurs, Fri Sat, Sun Visitor’s status

Entry hour

Ticket prices – $

Visitor’s age

Table 9.8: Equivalence classes – the Golden Splash Swimming Center ticket price module

9 Variable

Valid

Representing values

Invalid Representing

Softw

equivalence

equivalence values for classes

Values for Boundary

valid ECs values

classes invalid ECs

are tes

Day of week (1) Mon, Tue,

Mon

(1) Any alpha- Mox

Wed, Thurs, Fri

numeric value (not a day)

ting – str Sat

(2) Sat, Sun

Other than 88 status

Visitor’s (1) OT

OT

OT or M Entry hour

(2) M

(1) 6.00–19.00 7.55 6.00, 19.00 (1) Hours < 6.00 4.40

tegies &@

a (2) 19.01–24.00 20.44 19.01, 24.00 (2) Any alpha-

numeric values (not time)

Visitor’s age (1) 0.0–16.00

(1) Any alpha- TTR (2) 16.01–60.00

numeric value

(not an age) (2) Ages > 120.0

Table 9.9: Test cases – the Golden Splash Swimming Center ticket price module

Test case

Visitor’s Test case type

Test case Day of

Visitor’s Entry

results For valid ECs

For invalid ECs 10 Mox

OT

7.55 8.4 Invalid day

11 Mon

88 7.55 8.4 Invalid visitor status

12 Mon

OT

4.40 8.4 Invalid entry hour

8.4 Invalid entry hour

Invalid visitor age

15 Mon

OT

7.55 150.1 Invalid visitor age

Though the equivalence class method is applied mainly with correctness 201 tests, it may be used for other operation factor testing classes as well as for

9.5 Bl

revision and transition factor testing classes.

9.5.2 Other operation factor testing classes

ac k

Apart from output correctness tests, operation factor testing classes include

ox

the following classes of tests:

te sting

Quality requirements factor

Test class

Correctness

(1) Documentation tests (2) Availability (reaction time) tests

Reliability

Reliability tests

Efficiency Stress tests (load and durability tests) Integrity

Software system security tests Usability

(1) Training usability tests (2) Operational usability tests

Documentation tests Documentation testing, though neglected in many cases, should be consid- ered as important as code testing or design documents inspections. An erroneous user manual or programmer manual can lead to mistakes during program operation and maintenance that may incur damages equivalent in severity to those caused by software bugs.

Common components of documentation, supplied by the developer, are:

Functional descriptions of the software system. This overview enables the potential user to decide whether the system is suitable for his needs or not.

Installation manual. This document includes a detailed description of the installation process and hardware requirements as well as instructions for interfacing with equipment and with other software packages – if such interfaces are part of the system specifications. In commercial software packages (COTS software), the installation manual usually includes also customization instructions.

User manual. “How to get started” instructions, detailed instructions for applying the various system functions, recovery instructions for common operator mistakes and system errors are all covered in the user manual. In many cases, the user manual is supplied as a computerized help manual.

Programmer manual. This type of documentation is supplied for custom- made software. It includes the information required for maintaining the system (bug corrections, adaptation to changing requirements and

202 software improvement), program structure, description of program logic including algorithms, and so on. Users and customers of COTS software

9 do not require a programmer’s manual as they do not carry out mainte- Softw

nance individually.

are tes

Document testing plans should include the following three components:

Document completeness check. Based on the requirements specification

ting – str

and the detailed design reports, its purpose is to check whether all the required documents have been completed as specified and as intended by the designer.

a Document correctness tests. Correctness tests determine whether the tegies

instructions listed in the user document are correct. Implementation of correctness tests requires designing a test case file in a manner that close- ly resembles the methodology of output correctness tests discussed in Section 9.5.1.

Document style and editing inspection. Refers to document clarity and its agreement with documentation standards in cases this requirement is specified in the contract.

Availability tests Availability is defined as reaction time – the time needed to obtain the requested information or the time required for firmware installed in com- puterized equipment to react. Availability is of highest importance in on-line applications of frequently used information systems. The failure of firmware software to meet availability requirements (i.e., retarded reaction time) can make the equipment useless.

It is relatively difficult to test availability, especially for information sys- tems planned to serve a large population of users, and for real-time systems planned to treat high-frequency events. This difficulty stems from the need to carry out the tests under regular operation load as well as under maximal load conditions as specified in the requirement specifications. It should be noted that the availability requirements for regular and maximal workloads are usually different. The required characteristics of the availability testing environment support combining of this class of tests with load tests (stress tests) and performing the adjusted computerized combined tests. (For dis- cussion of load tests, see below.)

Reliability tests The software system reliability requirement deals with features that can be translated as events occurring over time, such as average time between fail- ures (e.g., 500 hours), average time for recovery after system failure (e.g., 15 minutes), or average downtime per month (e.g., 30 minutes per month). Reliability requirements are to be in effect during regular full-capacity oper- ation of the system. It should be noted that in addition to the software factor, Reliability tests The software system reliability requirement deals with features that can be translated as events occurring over time, such as average time between fail- ures (e.g., 500 hours), average time for recovery after system failure (e.g., 15 minutes), or average downtime per month (e.g., 30 minutes per month). Reliability requirements are to be in effect during regular full-capacity oper- ation of the system. It should be noted that in addition to the software factor,

9.5 Bl

Like availability testing, reliability testing is especially difficult as it

requires operation of the full range of software applications conducted under regular workload conditions. To be practical, such tasks should be carried

ac

out only after computerized simulations have been run to obtain average val-

ues, and only once the system is completed. With respect to resources, the

major constraint on performing tests of this type is the scope of resources

ox

required, which is vast as testing may continue for hundreds of hours and a

te

comprehensive test case file must be constructed.

sting

Statistical reliability testing offers a much less expensive and much speedi- er option for the assessment of reliability on the basis of statistical models. Much literature is available on the subject, just a few major sources being Myers (1976), Musa et al. (1990) and Musa (1998). However, despite their widespread use and practical benefits, statistical reliability tests have been sub- jected to criticism ever since their emergence. The main issue debated is the extent to which statistical models represent real-life software system operation.

Stress tests The class of stress tests subsumes two main types of tests: load tests and durability tests. It is possible to perform these tests only subsequent to soft- ware system completion. Durability tests, however, can generally be carried out only after the firmware or the information system software has been installed and is ready for testing.

Stress tests: load tests Load tests relate to the functional performance of the system under maximal operational load: maximal transactions per minute, hits per minute to an Internet site and the like. Load tests, which are usually conducted for loads higher than those indicated in the requirements specification, are of utmost importance for software systems planned to serve simultaneously a large population of users. In most working software systems, the maximal load figure combines several types of transactions. Due to its variability, the best way to explain the process is through an example.

Example “Music in the Air”, a network of music stores, run a service on the Internet that registers requests for price quotations and orders. On weekdays, the average rate of customer hits is 10 per minute for orders and 20 per minute for price quotations. The maximum loads recorded on Saturday afternoon are 30 per minute for orders and 100 per minute for price quotations. The maximal load defined in the software specifications, which takes future growth into account, is 60 per minute for orders and 200 per minute for price quotation. The loads for which the program will be tested are 75 per minute for orders and 250 per minute for price quotations. So, this explains how the test loads were chosen for the example.

204 Manual performance of load tests is impractical for most software sys- tems, and is therefore carried out by computerized tests based on

9 comprehensive simulations of high loads, again similar to the procedures Softw

adapted for availability testing. These load simulations enable us to measure the expected reaction times as a function of a range of loads. They thus allow

are tes

us to ascertain whether upgrading is necessary and which changes should be made to allow the software system to meet the planned requirements. For more about computerized load testing, see Section 10.3 in the next chapter

ting – str

which deals with computerized software testing. Stress tests: durability tests

a Durability tests are carried out in physically extreme operating conditions tegies

such as high temperatures, humidity, and high-speed driving along unpaved rural roads, as detailed in the durability specification requirements. Hence, these durability tests are typically required for real-time firmware integrated into systems such as weapon systems, long-distance transport vehicles, and meteorological equipment. Durability issues for firmware include firmware responses to climatic effects such as extreme hot and cold temperatures, dust, road bumps, and extreme operation failures resulting from sudden electrical failure, voltage “jumps” in the supply mains, sudden cutoffs in communica- tions, and so on. Information system software durability tests focus on operation failures resulting from sudden electrical failures, voltage “jumps” in the supply mains and sudden cutoffs in communications.

Software system security tests Software security components of software systems are aimed at preventing unauthorized access to the system or parts of it, detection of unauthorized access and the activities performed by the penetration, and the recovery of damages caused by unauthorized penetration cases.

The main security issues dealt with by these tests are:

Access control, where the usual requirement is for control of multi-level access (usually by a password mechanism). Of special importance here are the firewall systems that prevent unauthorized access to Internet sites.

Backup of databases and software files and recovery in cases of system failure.

Logging of transactions, system usage, access trials, and so forth. The challenge of creating and breaking into security systems has bred a spe-