Prioritization of Test Cases Prioritization of test cases means ordering the execution of test cases according to

12.7.5 Prioritization of Test Cases Prioritization of test cases means ordering the execution of test cases according to

certain test objectives. Formulating test objectives for prioritization of individual test cases is an extremely difficult task. Here, we discuss test prioritization in terms of groups of test cases with common properties. An example of a test objective for prioritization is: Execute the maximum number of test cases without being blocked. Furthermore, a major concern from software developers is that test engineers report critical defects (e.g., defects related to system crash) toward the end of test cycles. This does not give them enough time to fix those defects. In addition, blocking defects need to be fixed earlier in the test cycles in order to execute the blocked test cases. Therefore, we need to prioritize the execution of tests to detect critical defects early in the test cycles. In a multicycle-based test execution strategy, it is desirable to have different test objectives for prioritization in different test cycles for three reasons: (i) initially the quality level of the system under test is not very high, (ii) the quality of the system keeps improving from test cycle to test cycle, and (iii) a variety of defects are detected as testing progresses. Below we explain test prioritization in individual test cycles.

Test Prioritization in Test Cycle 1 Principle. Prioritize the test cases to allow the maximum number of test cases to

completely execute without being blocked. Test engineers execute their assigned test cases in different test environments.

Each engineer prioritizes the execution of their subset of test cases as follows:

A high priority is assigned to the test cases in the basic and functionality test groups.

A medium priority is assigned to the robustness and interoperability test groups.

A low priority is assigned to the test cases in the following groups: docu- mentation, performance, stress, scalability, and load and stability tests.

The basic tests give prima facie evidence that the system is ready for more rig- orous tests. The functionality tests provide a comprehensive testing over the full range of the requirements within the capabilities of the system. Both of these test groups are given high priority in the first test cycle to ensure that any functionality defects are fixed first. Functionality defects can block the execution of other test group. Stress, performance, scalability, load, and stability tests need complex con- figurations across different platforms, operating systems, and database management systems. Execution of these tests depends on the outcome of the interoperability and robustness tests. Therefore, interoperability and robustness test cases are executed

372 CHAPTER 12 SYSTEM TEST PLANNING AND AUTOMATION

next to flush out any issues that may block the execution of stress, performance, scalability, load, and stability tests.

Test Prioritization in Test Cycle 2 Principle. Test cases which failed in the previous test cycle are executed early

in the test cycle. In the second test cycle, the test cases are reassigned to the test engineers

based on their interest and expertise. The process described in Section 12.7.4 is used to distribute all the test cases into three different bins: red, yellow, and green. In this step, we are not selecting a subset of the test suite. Instead, the idea of partitioning a test suite is further used in prioritizing test cases. Each test engineer prioritizes the execution of test cases in their subset as follows:

A high priority is assigned to the test cases in the red bin.

A medium priority is assigned to the test cases in the yellow bin.

A low priority is assigned to the test cases in the green bin.

Test Prioritization in Test Cycle 3 Principle. Test prioritization is similar to that in the second test cycle, but it is

applied to a selected subset of the test cases chosen for regression testing. Once again, the test cases are reassigned based on interest and expertise

among the test engineers. Then each test engineer prioritizes the execution of test cases in their assigned subset as follows:

A high priority is assigned to the test cases in the red bin.

A low priority is assigned to the test cases in the yellow bin.

The reader may recall from the discussion of Section 12.7.4 that the test cases in the green bin are not executed in the final test cycle.