TEST DESIGN FACTORS

11.1 TEST DESIGN FACTORS

The central activity in test design is to identify inputs to and the expected outcomes from a system to verify whether the system possesses certain features. A feature is a set of related requirements. The test design activities must be performed in

a planned manner in order to meet some technical criteria, such as effectiveness, and economic criteria, such as productivity. Therefore, we consider the following factors during test design: (i) coverage metrics, (ii) effectiveness, (iii) productivity, (iii) validation, (iv) maintenance, and (v) user skill. In the following, we give motivations for considering these factors.

Coverage metrics concern the extent to which the DUT is examined by a test suite designed to meet certain criteria. Coverage metrics lend us two advantages. First, these allow us to quantify the extent to which a test suite covers certain aspects, such as functional, structural, and interface of a system. Second, these allow us to measure the progress of system testing. The criteria may be path testing, branch testing, or a feature identified from a requirement specification.

Each test case is given an identifier(s) to be associated with a set of require- ments. This association is done by using the idea of a coverage matrix. A coverage matrix [A ij ] is generated for the above idea of coverage [1]. The general structure of the coverage matrix [A ij ] is represented as shown in Table 11.1, where T i stands for the ith test case and N j stands for the j th requirement to be covered; [A ij ] stands for coverage of the test case T i over the tested element N j . The complete set of test cases, that is, a test suite, and the complete set of tested elements of the

coverage matrix are identified as T c = {T 1 ,T 2 , .., T q } and N c = {N 1 ,N 2 , .., N p }, respectively.

A structured test case development methodology must be used as much as possible to generate a test suite. A structured development methodology also min- imizes maintenance work and improves productivity. Careful design of test cases

Software Testing and Quality Assurance: Theory and Practice , Edited by Kshirasagar Naik and Priyadarshi Tripathy Copyright © 2008 John Wiley & Sons, Inc.

322 CHAPTER 11 SYSTEM TEST DESIGN

TABLE 11.1 Coverage Matrix [A ij ] Test Case

Requirement Identifier

Identifier

in the early stages of test suite development ensures their maintainability as new requirements emerge. The correctness of the requirements is very critical in order to develop effective test cases to reveal defects. Therefore, emphasis must be put on identification and analysis of the requirements from which test objectives are derived. Test cases are created based on the test objectives. Another aspect of test case production is validation of the test cases to ensure that those are reliable. It is natural to expect that an executable test case meets its specification before it is used to examine another system. This includes ensuring that test cases have ade- quate error handling procedures and precise pass–fail criteria. We need to develop

a methodology to assist the production, execution, and maintenance of the test suite. Another factor to be aware of is the potential users of the test suite. The test suite should be developed with these users in mind; the test suite must be easy to deploy and execute in other environments, and the procedures for doing so need to be properly documented. Our test suite production life cycle considers all six factors discussed above.