TEST OBJECTIVE IDENTIFICATION

11.4 TEST OBJECTIVE IDENTIFICATION

The question “What do I test?” must be answered with another question: “What do I expect the system to do?” We cannot test the system comprehensively if we do not understand it. Therefore, the first step in identifying the test objective is to read, understand, and analyze the functional specification. It is essential to have

a background familiarity with the subject area, the goals of the system, business processes, and system users for a successful analysis. Let us consider our previously revised requirement: The software image must

be easy to upgrade/ downgrade for 100 network elements. The test engineer needs to ask one question: What do I need to know to develop a comprehensive set of test objectives for the above requirement? An inquisitive test engineer may also ask the following questions:

• Do we have to upgrade the software image sequentially on each of the network elements or at the same time on all the 100 elements? Or, do we proceed in a batch of 20 elements at a time?

• What is the source of the upgrade? Will the source be on an element management server?

• What does “easy” mean here? Does it refer to the length of time, say, 200 seconds, taken by the upgrade process?

• Can we have a mix of old and new software images on different network elements on the same network? In other words, is a new software image compatible with the old image?

• If we support old and new software images on different network ele- ments on the same network, then the EMS should be capable of managing two versions of a software installed on the same network. Is it possi- ble for an EMS to manage network elements running different software versions?

• To what release will the software be downgraded? Suppose the software image is upgraded to the nth release from the (n − 1)th release. Now,

if the software image is to be downgraded, to what release should it be downgraded—(n − 1)th or (n − 2)th release?

335 • While a system is being upgraded, do we need to observe the CPU utiliza-

11.5 EXAMPLE

tion of the network elements and the EMS server? What is the expected CPU utilization?

We critically analyze requirements to extract the inferred requirements that are embedded in the requirements. An inferred requirement is one that a system is expected to support but is not explicitly stated. Inferred requirements need to be tested just like the explicitly stated requirements. As an example, let us consider the requirement that the system must be able to sort a list of items into a desired order. One obvious test objective is: Verify that the system can sort an unsorted list of items. However, there are several unstated requirements not being verified by the above test objective. Many more test objectives can be identified for the requirement:

• Verify that the system produces the sorted list of items when an already sorted list of items is given as input.

• Verify that the system produces the sorted list of items when a list of items with varying length is given as input.

• Verify that the number of output items is equal to the number of input items.

• Verify that the contents of the sorted output records are the same as the input record contents.

• Verify that the system produces an empty list of items when an empty list of items is given as input.

• Check the system behavior and the output list by giving an input list con- taining one or more empty (null) records.

• Verify that the system can sort a list containing a very large number of unsorted items.

The test objectives are put together to form a test group or a subgroup after they have been identified. A set of (sub)groups of test cases are logically com- bined to form a larger group. A hierarchical structure of test groups as shown in Figure 11.2 is called a test suite. It is necessary to identify the test groups based on test categories and refine the test groups into sets of test objectives. Individual test cases are created for each test objective within the subgroups; this is explained in the next section with an example. Test groups may be nested to an arbitrary depth. They may be used to aid system test planning and execution, which are discussed in Chapters 13 and 13, respectively.