TEST ENVIRONMENT

12.6 TEST ENVIRONMENT

It is necessary to plan for and design the test environment, also called a test bed or

a test laboratory, to make the execution of system-level test cases effective. It is a challenge to design test environments which contain only a small proportion of the equipment and facilities used in actual operation because of budget limitations. The central idea in using a small proportion of equipment and facilities is to do more with less . The objective here is to achieve effective testing whereby most of the defects in the system are revealed by utilizing a limited quantity of resources. One has to be innovative in designing a test bed such that the test objectives are fulfilled by executing the test cases on the test bed. One must consider alternatives, or, at least, a scaled-down version of the deployment environment from the standpoint of cost-effectiveness. Efforts must be made to create a deployment environment by using simulators, emulators, and third-party traffic generation tools. Such tools are found to be useful in conducting scalability, performance, load, and stress testing. An emulator may not be an ideal substitute for real equipment, but as long as it satisfies the purpose, it is worth investing in. We explained in Chapter 8 that there are different categories of test cases designed at the system level. Therefore, multiple test environments are constructed in practice for the following reasons:

• To run scalability tests, we need more resources than needed to run func- tional tests.

• Multiple test beds are required to reduce the length of system testing time. Preparing for a test environment is a great challenge in test planning. This is

especially true in testing distributed systems and computer networks where a vari- ety of equipment are connected through communication protocols. For example, such equipment includes user computers, servers, routers, base stations in a wireless

359 network, authentication servers, and billing servers. It may take several months to

12.6 TEST ENVIRONMENT

set up an effective test bed for large, complex systems. It requires careful planning, procurement of test equipment, and installation of the equipment in a test facility different from the software development facilities so that system testing is per- formed effectively. Developers have their own test environments to perform unit tests and integration tests. However, a separate, dedicated system test laboratory, different from the ones used in unit and integration testing, is essential for the following reasons:

• Test engineers need to have the ability to reconfigure a test environment. • Test activities need not interfere with development activities or live oper-

ations. • Increased productivity is achieved by having a dedicated test laboratory.

A central issue concerning setting up of a system test laboratory is the justification to procure the equipment. Note that there is a need to justify each item to be pro- cured. A good justification for procuring the equipment can be made by answering the following questions:

• Why do we need this equipment? • What will be the impact of not having this equipment? • Is there an alternative to procuring this equipment?

The technical leader of the system test engineering group should gather some facts and perform some preparation activities in order to get answers to these questions. The following items are part of a good fact gathering process:

• Reviews the system requirements and the functional specification • Participates in the review processes to better understand the system and

raise potential concerns related to the migration of the system from the development environment to the deployment environment

• Documents his or her findings The following preparation activities are conducted to support the development of

a system test bed: • Obtain information about the customer deployment architecture, includ-

ing hardware, software, and their manufacturers. For example, the real deployment network diagram along with the software configuration is use- ful in designing a scaled-down version of the system in the laboratory. The manufacturer names will be handy in procuring the exact equipment for interoperability and compatibility testing.

• Obtain a list of third-party products to be integrated with the SUT. Iden- tification of the external products is important because of the need for performing interoperability testing.

• List third-party test tools to be used to monitor, simulate, and/or generate real traffic. This traffic will be used as input to the SUT.

360 CHAPTER 12 SYSTEM TEST PLANNING AND AUTOMATION

• Identify the third-party software tools to be used under licenses. • Identify the hardware equipment necessary to support special features

specified in the requirement/test objectives, such as high availability and backup/recovery exercises within the test environment.

• List the number of hardware copies to carry out system testing if the project involves new hardware.

• Analyze the functional, performance, stress, load, and scalability test objec- tives to identify elements of the test environment that will be required to support those tests.

• Identify the security requirements for the test environment. Ensure that the security test cases can be executed using the test environment and an intruder cannot disrupt the stress and stability tests that may be running overnight or over the weekends.

• List the small, but necessary networking gears that may be required to set up the test laboratory, such as switches, terminal servers, hubs, attenuators, splitters, personal computers, servers, and different kinds and sizes of cables to interconnect these gears.

• List any other accessories required to facilitate system testing, such as racks, vehicles, and special shielding to prevent radiation.

After the above fact gathering and researching activities, the team leader develops

a schematic diagram of one or more test beds in terms of the following two items: • High-level graphic layout of test architectures

• Table of types of equipment, their quantities, and their descriptions to support the test architecture

The equipment list should be reviewed to determine the equipment available in-house and those that need to be procured. The list of equipment to be procured constitutes a test equipment purchase list. The list needs to include quantities required, unit price information, including maintenance cost, and justification, as outlined in the template shown in Table 12.2. The team leader must specify, for each item in the justification column, the justification for the item and the impact it will have on system testing in terms of quality and a time-to-market schedule. The team leader may obtain a quote from the suppliers to get an accurate unit price. The test team leader needs to keep track of the equipment received and their installation after the budget is

TABLE 12.2 Equipment Needed to be Procured Equipment to Procure

Quantity

Unit Price

Maintenance Cost Justification

361 approved and the orders are placed. The leader needs to ensure that these activities

12.7 TEST EXECUTION STRATEGY

are on track to meet the overall software project schedule.