SYSTEM TEST REPORT

13.9 SYSTEM TEST REPORT

During the test execution cycles test reports are generated and distributed as dis- cussed in Sections 13.4.2 and 13.4.4. A final summary report is created after completion of all the test cycles. The structure of the final report is outlined in Table 13.14.

The introduction to the test project section of the test report summarizes the purpose of the report according to the test plan. This section includes the following information:

• Name of the project • Name of the software image • Revision history of this document • Terminology and definitions • Testing staff • Scope of the document • References

The summary of test results section summarizes the test results for each test category identified in the test plan. In each test cycle, the test results are summarized in the form of the numbers of test cases passed, failed, invalid, blocked, and untested and the reasons for not executing some test cases. A test case may not be tested because of several reasons, such as unavailability of equipment and difficulty in creating the test scenario using the available test bed setup. The latter may only be possible at a real beta testing site. For each test cycle, the following data are given in this section:

• Start and completion dates • Number of defects filed • Number of defects in different states, such as irreproducible, FAD, closed,

shelved, duplicate, postponed, assigned, and open

TABLE 13.14 Structure of Final System Test Report 1. Introduction to the test project

2. Summary of test results 3. Performance characteristics 4. Scaling limitations 5. Stability observations 6. Interoperability of the system 7. Hardware/software compatible matrix 8. Compliance requirement status

439 Problems still remaining in the system, including any shortcoming or likeli-

13.10 PRODUCT SUSTAINING

hood of failure, are summarized in this section in terms of their levels of severity and their potential impact on the customers. For example, if the password is case insensitive, then it should be stated so. As another example, if an operating system puts a limit of 20 on the number of windows that can be simultaneously opened, then it should be stated that the user should not open more than 20 windows simultaneously; exceeding the limit may cause the system to crash.

In the performance characteristics section of the document, the system response time, throughput, resource utilization, and delay measurement results are described along with the test environments and tools used in the measurement process.

The limits of the system are stated in the scaling limitation section of the document based on the findings of scalability testing. Once again the facilities and tools that are used in scaling testing are described here.

The stability observations section states the following information: • Uptime in number of hours of the system

• Number of crashes observed by different groups, such as software devel- oper, ST, and SIT, in each week of the test cycles

• Descriptions of symptoms of the defects causing system crash that are in the irreproducible state

We state in the interoperability section of the document those third-party software and hardware systems against which the system interoperability tests were conducted. The list may include software and hardware against which the system will not interoperate.

A table shows what software versions are compatible with what hardware revisions in the hardware/software compatible matrix section. Generation of this table is based on the different kinds of hardware systems and different versions of the software used during system test cycles.

Finally, in the compliance requirement status section of the document compli- ance statistics are summarized by generating them from the requirements database. The following reports are generated by creating appropriate queries on the database:

• Requirements that are in compliance, partial compliance, or noncompliance with the software image

• Requirements that are verified by testing, analysis, demonstration, and inspection

• Requirements that are covered by each test case of the test suite along with the status of the test case