Primary Attributes

7.2.1 Primary Attributes

We consider four primary attributes: the scale, the goal, the property, and the method. We review these in turn in the following:

The Scale: We consider three possible values for this attribute: •

A unit: This represents a programming unit that implements a data abstraction or

a functional abstraction. As such, it can be a class in an object-oriented language, the implementation of an abstract data type, or a routine. •

A subsystem: This represents a component in an integrated software system. To test such a component in a credible manner, we may have to run the whole system and observe the behavior of this particular component within the system. This situation arises in the context of maintenance, for example, where we may change a component and then test the whole system to check whether the

A SOFTWARE TESTING TAXONOMY

changes are satisfactory, and whether the new subsystem works smoothly within the overall system.

A system: This represents a complete autonomous software system. The Goal: This is perhaps the most important attribute of a test. The values it may take

include the following: • Finding and removing faults: The most common goal of software testing is to observe the behavior of the program on test data, and to diagnose and remove faults whenever the program fails to satisfy its specification.

• Proving the absence of faults/certifying compliance with quality standards: While in practice it is virtually impossible to test a program on all possible com- binations of inputs and configurations, this possibility cannot be excluded in the- ory. Also, we can imagine cases where the set of possible input data is small, or cases where we can design a test data D such that if the program runs successfully on D, then it runs successfully on all the input space S.

• Estimating fault density: Software testing can be used to estimate the fault den- sity of a product, as will be discussed in Chapter 13. This is done by seeding faults, then running a test to compute how many seeded faults have been recov- ered and how many unseeded faults have been uncovered; fault density can then

be estimated by interpolation. • Estimating the frequency of failures: While the previous goals are concerned with faults, this and the next goal are concerned with failures instead; there is

a sound rationale for focusing on failures rather than faults, because it is better to reason about observable/relevant effects than about hypothetical causes. To pursue this goal, we run the software product under conditions that simulate its operational environment, and estimate its failure rate; the estimate is reliable only to the extent that the test environment is a faithful reflection of the operating environment, and that the test data reflects, in its distribution, the usage pattern of the software product. It is only under such conditions that the failure rate observed during testing can be borne out during field usage. We identify three possible instances of this goal, depending on whether we want to estimate the reliability, the safety, or the security of the product: ○ Estimating reliability

○ Estimating safety ○ Estimating security We get one instance or another, depending on the oracle that we use for the test: To

estimate reliability, we use the functional specifications of the software product; to estimate safety, we use the safety-critical requirements of the product; to estimate security, we use the security requirements of the product.

• Ensuring the infrequency of failures: As an alternative to proving the absence of faults, we may want to prove that faults are not causing frequent failures; after all,

a program may have faults and still be reliable, or more generally experience infrequent failure. Also, as an alternative to estimating the frequency of failures, we may simply establish that the frequency of failure is lower than a required

7.2 A CLASSIFICATION SCHEME 129

threshold. Whereas estimating the frequency of failure is a mere analytical proc- ess, that analyzes the product as it is, ensuring the infrequency of failures may involve diagnosing and removing faults until the frequency of failures of the soft- ware product is deemed to be lower than the mandated threshold. As we argued in the last item, in order for the estimate of the failure rate to be reliable, the soft- ware product must be tested in an environment that mimics its operating envi- ronment, and the test data must be distributed according to the usage pattern of the product in the field. Also, this goal admits three instances, depending on what oracle is used in the test: • Certifying reliability

• Certifying security • Certifying safety We get one instance or another, depending on the oracle that we use for the test.

Method: Given a limited amount of resources (time, manpower, and budget), we cannot run the software product on the set S of all possible input data; as a substi- tute, we want to run the software product on a set of test data, say D (a subset of S), that is large enough to help us achieve our goal, yet small enough to minimize costs. The test data generation method is the process that enables us to derive set D according to our goal; the criterion that we use to derive D from S depends on the goal of the test, as follows:

• If the goal of the test is to diagnose and remove faults, then D should cause the maximum number of failures, that is, uncover/sensitize the maximum number of faults.

• If the goal of the test is to prove correctness, then D should be chosen in such a way that if the program runs successfully on D, we can be reasonably confident (or assured) that it runs successfully on all of S.

• If the goal of the test is to estimate the product’s failure rate or to ensure that the product’s failure rate exceeds a mandated threshold, then D must be a represen- tative sample of the usage pattern of the product.

Test data generation methods are usually divided into three broad families: • Structural methods, which generate test data by analyzing the structure of the

software product and targeting the data in such a way as to exercise relevant com- ponents of the product.

• Functional methods, which generate test data by analyzing the specification of the software product or its intended function, and targeting the data in such a way as to exercise all the services or functionalities that are part of the specification or intended function.

• Random, with respect to a usage pattern. This method generates test data in such

a way as to simulate the conditions of usage of the software product in its oper- ating environment.

A SOFTWARE TESTING TAXONOMY

It is possible to map the goal of the test to the test data generation method in a nearly deterministic manner, as shown in the following table:

Method

Functional Random Goal Finding and removing faults

Structural

Proving the absence of faults √ Estimating the frequency of failure

√ Ensuring the infrequency of failure

√ Estimating fault density

Target Attribute: Most typically, one tests a software product to affect or estimate some functional quality of the product, such as correctness, reliability, safety, security, and so on; but as the following list indicates, one may also be interested in testing the product for a broad range of attributes.

• Functionality: Testing a software product for functional properties such as correctness, reliability, safety, security, and so on is the most common form of testing, and is the default option is all our discussions.

• Robustness: Whereas correctness mandates that the software product behaves according to the specification for all inputs in the specified domain, robustness further mandates that the program behaves reasonably (whatever that means: we can all rec- ognize unreasonable program behavior when we see it) outside of the specification domain, that is, on inputs or situations for which the specification made no provisions.

• Design and structure: In integration testing, the focus of the test is on ensuring that the parts of the software system interact with each other as provided by the design; here the attribute we want to test or ensure is the proper realization of the design.

• Performance: We may want to test a software product for the purpose of empirically analyzing its performance under normal usage conditions (e.g., normal workload). • Graceful degradation: In the same way that we distinguish between correctness (functional behavior for normal inputs) and robustness (functional behavior for exceptional or unplanned inputs), we distinguish between performance (opera- tional behavior under normal workloads) and graceful degradation (operational behavior under excessive workloads). To test a software product for graceful degradation, we operate it under excessive workloads and observe whether its performance decreases in a continuous, acceptable manner.

In Section 7.2.2, we review the secondary attributes and discuss how they are affected by the choices made for the primary attributes reviewed in this section.

7.2 A CLASSIFICATION SCHEME 131