The efficiency of peer reviews
8.3.5 The efficiency of peer reviews
The issue of defect detection efficiency of peer review methods proper and in comparison to other SQA defect detection methods is constantly being debated. Some of the more common metrics applied to estimate the efficien- cy of peer reviews, as suggested in the literature, are:
Peer review detection efficiency (average hours worked per defect detected).
Peer review defect detection density (average number of defects detected per page of the design document).
Internal peer review effectiveness (percentage of defects detected by peer review as a percentage of total defects detected by the developer).
PARTICIPANTS
Inspection Walkthrough
8 Moderator (scribe)
Coordinator (scribe)
R ev iews Coder or
User (presenter)
Author
representative PROCESS Organizational
Organizational preparations
preparations
Overview meeting
Thorough review of Brief overview document
reading
Inspection session(s) Walkthrough session(s)
Inspection session report
Walkthrough
session report Corrections
Inspection summary report
and reworking
Follow-up of corrections and reworking
Figure 8.2: Inspection vs. walkthrough – participants and processes
The literature provides rather meager indications about findings inspection effectiveness. Dobbins (1998) quotes Madachy’s findings from an analysis of the design and code inspections conducted on the Litton project. Madachy’s findings regarding the first two metrics cited above are presented in Table 8.2.
Dobbins (1998) also cites Don O’Neill’s 1992 National Software Quality Experiment, conducted in 27 inspection laboratories operating in the US. This experiment provides some insight into the code inspection process, especially at the preparation stage. A total of 90 925 source code
Table 8.2: The Litton project’s inspection efficiency according to Madachy
Inspection efficiency metrics
8.3 P
Inspection Type of
Total
Defect
detection document
No. of
number of
eer r
and major
(work hours)
page)
major defect)
ev
iew
Code inspections
*276 422 lines of code. Source: After Dobbins (1998)
Total number of defects detected
Number of major defects detected
Total preparation time (minutes)
Accordingly:
Average preparation time per detected defect
12.3 minutes (0.2 hours)
Average preparation time per detected major defect
94.3 minutes (1.57 hours) Considering the different environments, a comparison of the defect densities
detected in the National Software Quality Experiment and those found in the Litton project reveal relatively small differences, as shown below:
National Software Litton Project Quality Experiment
Total defect detection density (defects per KLOC*) 20.3 25.9 Major defect detection density (defects per KLOC*)
168 The internal effectiveness of inspections is discussed by Cusumano (1991, pp. 352–353), who reports the results of a study on the effectiveness of design
8 review, code inspection and testing at Fujitsu (Japan) for the period R
ev
1977–1982. After two decades, the findings are still of interest, even though
iews
no efficiency metrics are provided. A comparison by year of inspection, pre- sented in Table 8.3, shows substantial improvement in software quality associated with an increased share of code inspection and design reviews and
a reduced share of software testing. The software quality is measured here by the number of defects per 1000 lines of maintained code, detected by the users during the first six months of regular software system use.
Though quantitative research results refer only to the inspection method, we can expect to obtain similar results after application of the walkthrough method. This assumption will one day have to be verified empirically for us to be certain.