Selecting Test Cases for Final Test Cycle Though it is desirable to reexecute all the test cases in the final test cycle, a

12.7.4 Selecting Test Cases for Final Test Cycle Though it is desirable to reexecute all the test cases in the final test cycle, a

lack of time and additional cost may not allow the system test team to do so. The concept of regression testing is applied only in the final test cycle. In our approach, test cases are selected based on (i) the results of their prior execution; (ii) their membership in certain test groups: basic, functionality, robustness, interoperability, stress, scalability, performance, and load and stability; and (iii) their association with software components that have been modified.

Test cases are selected in three steps: In the first step, the test suite is parti- tioned into four different bins—red, yellow, green, and white—based on certain criteria which are described in the selection procedure given below. The red bin is used to hold the test cases that must be executed. The yellow bin is used to hold the test cases that are useful to execute. The green bin is used to hold the test cases that will not add any value to regression testing and thus can be skipped. The rest of the test cases are included in the white bin. In other words, the test cases for which no concrete decision can be made in the first step are put in the white bin. In the second step, test cases from the white bin are moved to the other bins by considering the software components that were modified during system testing and the test cases that are associated with those components. Finally, in the third step, the red and yellow bins are selected for regression testing. In the following, we present the test selection procedure:

Step 1: The test suite is partitioned into red, yellow, green, and white bins as follows:

• Red: The red bin holds the following kinds of test cases that must be executed:

Test cases that failed at least once in the previous test cycles. Test cases from those test groups for which RCA was conducted by

the development team in the previous test cycles. Test cases from the stress, scalability, and load and stability test groups.

These test cases are more likely to reveal system crash defects. They are selected to ensure that the final build is stable and the probability of the system crashing at the customer site is extremely low.

Test cases from the performance test category. The performance char- acteristic must be measured against the final build that is going to

be released to the customers. • Yellow: The yellow bin holds the test cases that are useful to execute.

This bin includes those test cases whose objectives are similar to the objectives of the test cases in the red bin. For example, let a test case with the following objective be in the red bin: While software up-gradation is in progress, the CPU utilization should not exceed 60%. Then, a test case with the following objective is put in the yellow bin: Verify that software image can be upgraded to the nth release from the (n − 1)th release in less than 300 seconds. The condition, that is, less

370 CHAPTER 12 SYSTEM TEST PLANNING AND AUTOMATION

than 60% CPU utilization, tells us when to execute the upgradation test.

• Green: The green bin holds the test cases that will not add any value to regression testing and thus can be skipped. This bin includes those test cases whose objectives are implicitly covered by the execution of the test cases in the red and yellow bins. For example, if a test case with the objective “Verify that a software image can be upgraded to the nth release from the (n − 1)th release in less than 300 seconds” is in the yellow bin, then a basic test case with the objective “The software can

be upgraded to the nth release from the (n − 1)th release” is included in the green bin.

• White: The test cases for which no concrete decision can be made in the first step are put in the white bin. This includes the rest of the test

cases not falling in the red, yellow, or green bin. Step 2: Test cases from the white bin are moved to the other bins by considering

the software components that were modified during the system testing and the test cases that are associated with those components. The software developers identify all the software components that have been modified after the start of the first test cycle. Each test case from the white bin is mapped to the identified software components. This mapping is done by analyzing the objective of the test case and then checking whether the modified code of the identified software components is exercised by executing the test cases. The test cases that are mapped to more than one, one, or zero software components are moved to the red, yellow, or green bin, respectively.

Step 3: Test cases from the red and yellow bins are selected for regression testing as follows:

• All the test cases from the red bin are selected for the final test cycle. • Depending on the schedule, time to market, and customer demand, test

cases from the yellow bin are selected for the final test cycle. Remark 2. It is useful to understand the selection strategy explained above. The

red bin holds the test cases that must be selected in the final test cycle. The test cases falling in the “must execute” category are (i) the test cases which had failed in the previous test cycles; (ii) the test cases from the groups for which RCA was performed in the previous test cycles; (iii) test cases from the stress, scalability, and load and stability test groups; and (iv) the test cases which concern modified software components. We remind the reader that RCA for a test group is performed if too many test cases had failed from that group. We put emphasis on a test group by selecting its test cases in the final test cycle. Test cases from the stress, scalability, and load and stability groups are also included in the final test cycle by default, even though those test cases may not concern the modified components. The rationale for their inclusion is that one must run a final check on the stress, scalability, load, and stability aspects of software systems—such as

371 servers, Internet routers, and base stations in wireless communication— which are

12.7 TEST EXECUTION STRATEGY

likely to serve thousands of simultaneous end users.