5 The main types of automated tests

Frame 10.5 The main types of automated tests

■ Code auditing ■

Coverage monitoring ■

Functional tests ■

Load tests ■

Test management

A code auditor can verify the following:

Does the code fulfill code structure instructions and procedures? – Module size. Some code auditors calculate for the tested code com-

plexity metrics, such as McCabe’s cyclomatic complexity metrics – Levels of loop nesting – Levels of subroutine nesting – Prohibited constructs, such as GOTO.

Does the coding style follow the coding style procedures?

10 – Naming conventions for variables, files, etc. – Unreachable code lines of program or entire subroutines.

Sof

Do the internal program documentation and help support sections follow

tw

the coding style procedures?

are tes

Format and size of comments: – Location of comments in the file

ting – impl – Help index and presentation style. Coverage monitoring

Coverage monitors produce reports about the line coverage achieved when implementing a given test case file. The monitor’s output includes the per-

ement

centage of lines covered by the test cases as well as listings of uncovered lines. These features make coverage monitoring a vital tool for white-box tests.

ation Functional tests

Automated functional tests often replace manual black-box correctness tests. Prior to performance of these tests, the test cases are recorded into the test case database. The tests are then carried out by executing the test cases through the test program. The test results documentation includes listings of the errors identified in addition to a variety of summaries and statistics as demanded by the testers’ specifications.

After the corrections have been completed, re-testing the whole program or parts of it (“regression tests”) is commonly required. Automated regres- sion tests performed for the whole program verify that the error corrections have been performed satisfactorily and that the corrections have not uninten- tionally introduced new errors in other parts of the program. The regression tests themselves are performed with the existing test case database; hence, these tests can be executed with minimal effort or professional resources. An additional automated testing tool that supports functional tests, the output comparator, is of great help in the regression test stage. The automated com- parison of outputs of successive tests, together with the results of the functional testing tools, enables testers to prepare an improved analysis of the regression test results and to help developers to discover the causes of the errors detected in those tests. It is quite common for a program to require three or four regression tests before its quality level is considered satisfactory.

Load tests The history of software system development contains many sad chapters of systems that succeeded in correctness tests but severely failed – and caused enormous damage – once they were required to operate under standard full load. The damage in many cases was extremely high because the failure occurred “unexpectedly”, when the systems were supposed to start provid- ing their regular software services. The most spectacular failures tend to take Load tests The history of software system development contains many sad chapters of systems that succeeded in correctness tests but severely failed – and caused enormous damage – once they were required to operate under standard full load. The damage in many cases was extremely high because the failure occurred “unexpectedly”, when the systems were supposed to start provid- ing their regular software services. The most spectacular failures tend to take

10.3 Autom

simultaneous events.

For load tests to be performed, the maximal load environment must first

be created. If executed manually, the tests must be conducted when the sys- tem is under maximal user load, a condition that is impractical in most cases and impossible in others. Therefore the only way to carry out load tests for medium-sized and large systems is by means of computerized simulations

ated tes

that can be programmed to closely approach real load conditions. The load tests themselves are based on scenarios of the maximal load sit- uations – composed of events or transactions and their frequencies – that the

ting

software system is expected to confront and deal with. This allows automat-

ed load testing (stress tests) to be combined with availability and efficiency tests, which likewise require maximal load environment for their execution. At this point, “virtual users and virtual events” come into play. For oper- ating scenarios devised for load testing, virtual users and virtual events are generated and operated in a hardware and communication environment defined by the system planner. A virtual user or event emulates the behavior of a human user or a real event. Its behavior is “constructed” by applying real outputs captured from real user applications, that are then used as inputs for the simulation. The simulation’s required loads and frequencies are also created by computerization. The simulation then produces outputs similar to those captured from real-life users at the frequencies and with the user mix defined by the scenario. These outputs serve as inputs for the test-

ed software. The tests are carried out with the final approved version of software and with the planned hardware and communication configuration. The computerized monitoring of the load tests produces software system performance measurements in terms of reaction time, processing time, and other desired parameters. These are compared with the specified maximal load performance requirements in order to evaluate how well the software system will perform when in daily use. Usually, a series of load tests is con- ducted, with the load gradually increased to the specified maximal load and beyond. This step enables a more thorough study of system performance under full load. The computer-produced tables and graphs, based on the per- formance measurement information, allow the tester to decide what changes are to be introduced into each simulation for each test iteration. For exam- ple, the tester may wish to:

Change the hardware, including the communication system, to allow the software system to fulfill its performance requirements at each load level.

Change the scenario in order to reveal the load contributed by each user or event.

Test an entirely different scenario.

Test new combinations of hardware and scenario components. The tester will continue his iterations till he finds the appropriate hardware

240 Example