Advantages and disadvantages of automated tests
10.3.3 Advantages and disadvantages of automated tests
The decision to employ automated testing tools is difficult to make because of the substantial investments involved in purchasing the tools and in ade-
ement
quately training a team for their effective and efficient implementation. The first part of this section presents a comprehensive qualitative com- parison of automated testing and manual testing, conducted by listing the
ation
advantages and disadvantages of automated testing. A quantitative compar- ison, especially an economic analysis based on empirical data, is sorely needed to support the qualitative comparison. The second part of the section deals with some early quantitative findings that point to the economic advantages of using automated testing tools.
The main advantages of automated tests are: (1) Accuracy and completeness of performance. Computerized testing guar-
antees – to the maximum degree possible – that all tests and test cases are carried out completely and accurately. Manual testing suffers from periods of tester weariness or low concentration, traits that induce inac- curate keying-in of the test case, omissions, and so forth.
(2) Accuracy of results log and summary reports. Automated tests are pro- grammed for accuracy of reporting of errors detected. In contrast, the testers who perform manual tests occasionally do not recognize errors and may overlook others in their logs and summaries.
(3) Comprehensiveness of information. Naturally, once the test data – including test results – are stored in a database, queries and reports about the test and its results are incomparably more available than the same items would be after performance of manual tests. Besides sup- porting testing and correction follow up, the improved error information enhances the input needed for preventive and corrective actions (see Chapter 16).
(4) Few manpower resources required to perform tests. Manual perform- ance of testing, in comparison, is a major consumer of manpower resources.
(5) Shorter duration of testing. The duration of computerized testing is usu- ally far shorter than that of manual testing. In addition, automated tests can be carried out, uninterrupted, 24 hours a day, seven days a week, in (5) Shorter duration of testing. The duration of computerized testing is usu- ally far shorter than that of manual testing. In addition, automated tests can be carried out, uninterrupted, 24 hours a day, seven days a week, in
10.3 Autom
both being rather impractical in most cases.
(6) Performance of complete regression tests. Because of the shortage of time and manpower resources, manual regression tests tend to be con- ducted on only a relatively small portion of the software package. Hence, the advantages of automated testing: the minimal time and man- power resources required make it possible to rerun tests based on
ated tes
previous results. This option substantially reduces the risk of not detect- ing any errors introduced during the previous round of corrections.
(7) Performance of test classes beyond the scope of manual testing.
ting
Computerization enables the tester to perform, for example, load tests, availability tests and efficiency tests for medium- and large-scale sys- tems. These tests are almost impossible to perform manually on systems of greater than small size.
The main disadvantages of automated testing are: (1) High investments required in package purchasing and training. An
organization that decides to implement automated testing must invest in software packages and additional training to qualify its staff to perform those tests. Despite the claims of the software package developers, although the amount of training varies by software package, it is still long and thus expensive.
(2) High package development investment costs. In cases where available automated testing packages do not fully suit the system’s requirements, custom-made packages must be developed.
(3) High manpower requirements for test preparation. The human resources required for preparing an automated test procedure are usual- ly substantially higher than those required for preparing a manual procedure for the same software package.
(4) Considerable testing areas left uncovered. At present, automated software testing packages do not cover the entire variety of development tools and types of applications either available manually or still needed. This forces testers to mix manual and automated testing in their test plans.
The advantages and disadvantages of automated software testing are pre- sented in Frame 10.7.
Quantitative comparison – empirical findings Dustin et al. (1999) report the findings of a study carried out during 1997–1998, initiated by the European Systems and Software Institute (ESSI). Graphical user interface (GUI) software was chosen to be tested. The study comprised 10 comparative experiments, with parallel manual testing and
Frame 10.7 Automated software testing: advantages and