TESTING ISSUES

16.2 TESTING ISSUES

To get a sense of the issues that arise in testing product lines, we review the traditional lifecycle of software testing then we explore how this lifecycle can be combined with the PLE process discussed earlier (and illustrated in Fig. 16.1). Given a software product P and a specification R, and given that we want to test whether P is correct with respect to R, we proceed through the following phases:

• Test Data Generation, whereby we inspect program P or specification R or both to generate test data T on which we envision to test P. • Test Oracle Design, whereby we derive a test oracle Ω from specification R, which for any element s of T tells whether the pair (s, P(s)) represents a correct execution of program P on initial state s.

• Test Driver Design, whereby we combine the test data generation criterion with the oracle design to derive a test driver that runs program P on test data T and stores the outcome.

• Test Outcome Analysis, whereby we analyze the outcome of the test and take action according to the goal of the test (fault density estimation, fault removal, reliability estimation, product certification, etc.).

Of course, we can carry out these steps for each application that we develop at the application engineering phase, but this raises the following issues:

• This is extremely inefficient: Indeed, each application is made up of common parts (that stem from commonality analysis) and application-specific parts

352 TESTING PRODUCT LINES

(that stem from the specific variabilities of the application). The common parts have been tested each time an application is tested; and the variable parts have been tested each time an application with the same variability has been tested. Ideally, we would like to focus the testing effort on those aspects of the application that have not received adequate coverage.

• This is incompatible with the spirit of PLE: The whole paradigm of PLE revolves around streamlined reuse of reuse artifacts and processes; it is only fitting that reuse should extend to testing artifacts (such as test data) and processes (such as testing common features only once, rather than repeatedly for each application).

• This alters the Economics of PLE: The economics of PLE is based on the assumption that a great deal of effort is invested in domain engineering in order to support the rapid, efficient, cost-effective production of applications at appli- cation engineering time. If we burden the application engineering phase with the task of testing each application, this may undermine the economic rationale of the product line.

• The application specification is not self-contained: The specification of an application for the purposes of application engineering (written in the AML, for example) is cast in the context of domain engineering and merely specifies the attributes that characterize the application within the scope of the domain; as such, it is not self-contained. To write the specification of the application in a self-contained manner (for the purpose of oracle design), one needs to refer to all the implicit domain requirements that arise in domain engineering.

In light of these issues, it is legitimate to consider shifting the bulk of testing to the domain engineering phase, rather than the application engineering phase. Unfor- tunately, this option raises a host of issues as well, such as the following:

• Absence of Executable Assets: At the end of domain engineering, we do not have any self-contained executable assets to speak of. What we typically have are adaptable software components that are intended to be used as part of complete applications.

• Absence of Verifiable Specifications: Not only do we not have completely defined operational software products we also do not have precise requirements specifica- tions to test applications against. Instead, we have domain models and feature models that capture domain knowledge and represent domain variabilities.

• Combinatorics of Variabilities: Testing a single software product is already hard enough due to the massive size of typical input spaces. PLE compounds that complexity by adding the extra dimensions of variability: If a product line has five dimensions of variability, and each variability may take four possible values

(considered a toy-size example), we have in effect 4 5 = 1024 possible configura- tions to test. If we assume that (some or all of ) the variabilities are optional (i.e., a user may opt out of a variability), then the number of application configurations

can reach 5 5 = 3125.

16.3 TESTING APPROACHES 353

• Feature Interactions: One way to deal with the combinatorial explosion alluded to earlier is to consider the variabilities one at a time, falling back on default options. The trouble with that option is that it fails to uncover problems that arise when variabilities are combined; in particular, it fails to detect feature interac- tions that may arise.

• Failure to Certify the Composition Step: Testing that takes place at the domain engineering phase precedes, by definition, the application engineering phase, hence it fails to detect issues that may arise at the latter phase. In particular, it fails to ensure that variabilities are bound appropriately according to the specification.