MODELING TEST RESULTS

11.7 MODELING TEST RESULTS

A test suite schema can be used by a test manager to design a test suite after a test factory is created. A test suite schema, as shown in Table 11.7, is used to group test cases for testing a particular release. The schema requires a test suite ID, a title, an objective, and a list of test cases to be managed by the test suite. One also identifies the individual test cases to be executed (test cycles 1, 2, 3 and/or regression) and the requirements that the test cases satisfy. The idea here is to gather a selected number of released test cases and repackage them to form a test suite for a new project.

Test engineers concurrently execute test cases from a selected test suite on different test beds. The results of executing those test cases are recorded in the test factory database for gathering and analyzing test metrics. In a large, complex system with many defects, there are several possibilities of the result of a test execution, not merely passed or failed. Therefore, we model the results of test execution by using a state transition diagram as shown in Figure 11.7, and the corresponding schema is given in Table 11.8. Figure 11.7 illustrates a state diagram of a test case result starting from the untested state to four different states: passed, failed, blocked, and invalid.

348 CHAPTER 11 SYSTEM TEST DESIGN

TABLE 11.7 Test Suite Schema Summary Field

Description

test_suite_id Unique identifier assigned by originator test_suite_title

Test suite title, one-line title for test suite test_suite_objective

Objective of test suite, short description. tests

Reference list of test case identifiers test_id

Test case ID, selected

test_title Test case title, read only, filled in when test case is created test_category

Category name (performance, stress, interoperability,

functionality, etc.)

tester

Engineer responsible for testing

sw_dev Software developer responsible for this test case who will assists test engineer in execution of test case

priority

Priority of test case

requirement_ids Requirement identifier, read only, filled in when test case is

created

cycle 1 –3 Check box to indicate test is cycle 1, 2, or 3 test case regression

Check box to indicate test is regression test case

TABLE 11.8 Test Result Schema Summary Field

Description

tc_id Reference to test case identifier record test_title

Test case title, read only, filled in when result is created test_category

Test category name (performance, stress, interoperability,

functionality, etc.)

status State of test case result: passed, failed, blocked, invalid or untested; initially status of test case is “untested”

run date

Date test case was run

time Time spent in executing test case tester

Name of person who ran test

release

Software release (03.00.00)

build Software integration number (1 –100). defect_ids

List of defects which cause test to fail; value can come from

bug tracking database

test_suite

Test suite this results pertains to

The execution status of a test case is put in its initial state of untested after designing or selecting a test case. If the test case is not valid for the current software release, the test case result is moved to the invalid state. In the untested state, the test suite identifier is noted in a field called test_suite_id. The state of the test result, after execution of a test case is started, may change to one of the following states:

11.8 TEST DESIGN PREPAREDNESS METRICS

Passed

Untested Failed

Blocked

Invalid Figure 11.7 State transition diagram of test case result.

passed, failed, invalid, or blocked. A test engineer may move the test case result to the passed state from the untested state if the test case execution is complete and satisfies the pass criteria.

If the test execution is complete and satisfied the fail criteria, a test engineer moves the test result to the failed state from the untested state and associates the defect with the test case by initializing the defect_ids field. The test case must

be reexecuted when a new build containing a fix for the defect is received. If the reexecution is complete and satisfies the pass criteria, the test result is moved to the passed state.

The test case result is moved to a blocked state if it is not possible to com- pletely execute it. If known, the defect number that blocks the execution of the test case is recorded in the defect_ids field. The test case may be reexecuted when a new build addressing a blocked test case is received. If the execution is complete and satisfies the pass criteria, the test result is moved to the passed state. On the other hand, if it satisfies the fail criteria, the test result is moved to the failed state. If the execution is unsuccessful due to a new blocking defect, the test result remains in the blocked state and the new defect that blocked the test case is listed in the defect_ids field.