Representativeness, external validity, and generalizability

Box 6 Domain, instance, case, population, sample, and replication In this book, we define the domain of the theory represented by the rectangle in the picture below as the universe of all possible instances represented by the symbol x of the object of study to which the theory applies. The domain is a characteristic of the theory. It does not refer to the set of instances that is selected for a study. For a test in a survey, a subset of instances must be selected from the domain. We call such a subset a population represented by an ellipse in the picture below, in which three populations are depicted. Usually a smaller subset of instances is selected from the population for the study. We call such a subset from the population, selected for a study, a sample. A sample from a population must be representative for the population, which can be achieved by using probability sampling techniques. Populations are never “representative” for an entire domain. The significance of a test result for the theory in a survey must always be assessed by means of replications in other, equally unrepresen- tative, populations from the domain. A candidate population is not just any group of instances selected from the domain, but is defined by one or more criteria. This allows a researcher to claim that a proposition has been tested in a named population such as, “the population of European airline companies” rather than in a group of instances selected for the study. For a test in a single or a comparative case study, instances of the object of study must be selected from the domain. We call such instances cases. Cases are never “represen- tative” for a domain. The significance of a test result for the theory in a case study must always be assessed by means of replications in other, equally unrepresentative, instances from the domain. A group of cases is rarely a sample as defined for a population, with the exception of a group of instances selected for a quasi survey see Chapter 7: “Testing a probabilistic relation with a case study”. x x x x x x x x x x x x x x x x xx x x x x x x xx x x x x x x x x x x x xx x x x xxx xx x x x x x xx x x xx x x x x x x x x xx x xx x x x x xxx x x x x x x x x xx x x x x x xxx x x x xx x x x xxx x x x x x x x x x x x x x xx x x x x x x x x x xx x x x x x x xx in the theoretical domain, as well as the degree of similarity between the causal relations in these instances and in the domain. The actual extent of domain representativeness of a group of instances cannot be determined because the distribution of values of the variable charac- teristics of all instances in a theoretical domain cannot be known. Population representativeness is the degree of similarity between the distribution of the values of the variables in a sample and their dis- tribution in the population from which the sample is drawn as well as the degree of similarity between the causal relations in the sample and in the population. The actual degree of population representativeness can be determined in principle because it is possible in principle though usually unfeasible to determine the distribution of values of the variable characteristics of all instances in the population. The degree of population representativeness of a probability sample can be estimated if the distribution of the values of the variables in the instances of the sample is known. External validity is a characteristic of a study outcome. External valid- ity is the extent to which the outcome of a study in one instance or in a group of instances applies or can be generalized to instances other than those in the study. Two important forms of external validity are ecological validity and statistical generalizability. Ecological validity is the extent to which the outcome of a laboratory experiment applies to instances of the object of study in its real life context. Statistical gener- alizability is the likelihood that research results obtained in a sample of a population are also true for the population. Generalizability is a characteristic of a proposition and therefore of a theory. It is the degree of confidence that a proposition is correct and applies to the entire theoretical domain. Generalizability is enhanced if the proposition is supported in a series of replications. Generalizability decreases if the proposition is not supported in a number of such tests. The alleged lack of “generalizability” of the case study is a misunderstand- ing. First, generalizability is not a characteristic of a study but of a proposition. Second, external validity which is a characteristic of a study’s outcome is not an issue in most forms of case study research because usually there is no population to which results are “general- ized” with exception of the quasi survey; see Chapters 4 and 7. Third, cases in case study research are equally unrepresentative of a theoretical domain as populations in survey research. There is, however, a general “lack of generalizability” of propositions in the sense that most propositions are tested only once in one-shot studies. This problem, however, applies in principle to all types of propositions, irrespective of the research strategy by which they are tested. With more replication studies, the generalizability of propositions could be enhanced. Generalization, thus, is an aim rather than a claim. It is something a research community aims to be able to do after a series of replications rather than claiming to be able to do on the basis of an assumed degree of representativeness of the instances in which a test was conducted.

3.2.5 Exploration for theory-oriented research

Before a theory can be tested or built by research, an exploration must be performed. As can be seen in Flowchart 2 we distinguish between an exploration of theory which comes first and a consecutive exploration of prac- tice. If propositions have been found in the exploration of theory, the goal of the exploration of practice is to find support for the relevance of the propositions and to find reasons for prioritizing one or more of them for testing. If no propositions have been found in the explor- ation of theory, the goal of the exploration of practice is to find propo- sitions. We will discuss each of these three types of exploration separately. Hence, there are three types of exploration for theory- oriented research that will be discussed below: ■ exploration of theory; ■ exploration of practice for finding a proposition; and ■ exploration of practice for confirming the relevance of a proposition.

3.2.5.1 Exploration of theory

The aim of exploration of theory in theory-oriented research is, first, to find candidate propositions for testing and, second, to select one or more of these propositions for being tested in the study. Initial exploration will consist of conducting a literature review in order to identify potentially rele- vant texts such as books, review articles, research articles, and theoretical articles and of reading a selection of these sources. These sources must be evaluated and contradictory statements must be inter- preted. A literature review will describe what is considered to be “known” about the object of study and what is not yet known. A critical literature review weighs the evidence for what is considered to be “known”. For instance, assumptions and not yet tested propositions 48 cannot be accepted as “knowledge” and, more importantly, the num- ber and quality of replications of each “proven” proposition must be critically assessed. The literature review will conclude with a list of ■ propositions that need further replication, ■ propositions that have never been tested, and ■ aspects of the object of study about which no proposition has been found. Usually a literature review does not describe the most recent insights in the field because publications lag several years behind actual develop- ments known only to experts who attend conferences and exchange information among them. Such experts usually also know important sources that will not be found in a literature search and they will also have explanations for the presence or absence of certain ideas. In other words, it is necessary not only to conduct “desk research” but also to communicate with insiders. Experts on the theory are usually quite eager to convey their insights to students and to interested colleagues. If this exploration of theory consisting of a literature review as well as communicating with experts has been successful which it usually is, this phase of the research process can be concluded with ■ a description of the current body of knowledge, ■ a list of propositions that have some support but need further testing replication, ■ a list of propositions that have been proposed but not yet tested, ■ a description of aspects of the object of study about which no proposition has yet been formulated, and ■ reasoning about what needs to be done next. The last result is, obviously, the most important. It specifies either a proposition that should be further tested and why this one or a proposition that should be tested for the first time and why this one or an aspect of the object of study about which a new proposition should be built and why.

3.2.5.2 Exploration of practice for finding a proposition

If the conclusion of the exploration of theory is that a new prop- osition should be built, it is usually concluded that theory-building research should be conducted. In our view, however, an exploration of practice should be conducted first before a decision is made to conduct