Representativeness, external validity, and generalizability
Box 6 Domain, instance, case, population, sample, and replication
In this book, we define the domain of the theory represented by the rectangle in the picture below as the universe of all possible instances represented by the symbol x
of the object of study to which the theory applies. The domain is a characteristic of the theory. It does not refer to the set of instances that is selected for a study.
For a test in a survey, a subset of instances must be selected from the domain. We call
such a subset a population represented by an ellipse in the picture below, in which three populations are depicted. Usually a smaller subset of instances is selected from
the population for the study. We call such a subset from the population, selected for a study, a sample. A sample from a population must be representative for the population,
which can be achieved by using probability sampling techniques. Populations are never “representative” for an entire domain. The significance of a test result for the theory in
a survey must always be assessed by means of replications in other, equally unrepresen- tative, populations from the domain. A candidate population is not just any group of
instances selected from the domain, but is defined by one or more criteria. This allows a researcher to claim that a proposition has been tested in a named population such
as, “the population of European airline companies” rather than in a group of instances selected for the study.
For a test in a single or a comparative case study, instances of the object of study must be selected from the domain. We call such instances cases. Cases are never “represen-
tative” for a domain. The significance of a test result for the theory in a case study must always be assessed by means of replications in other, equally unrepresentative, instances
from the domain.
A group of cases is rarely a sample as defined for a population, with the exception of
a group of instances selected for a quasi survey see Chapter 7: “Testing a probabilistic relation with a case study”.
x x x x x x x x x x x
x x x x x xx x x x x x
x xx x x x x x x x x x x x xx x
x x xxx xx x x x x x
xx x x xx x x x x x x
x x xx x
xx x x x x xxx x x x x
x x x x
xx x x x x x xxx x x x xx x
x x xxx x x x x x x x x x x x x x xx x x
x x x x x x x xx x x
x x x x xx
in the theoretical domain, as well as the degree of similarity between the causal relations in these instances and in the domain. The actual
extent of domain representativeness of a group of instances cannot be determined because the distribution of values of the variable charac-
teristics of all instances in a theoretical domain cannot be known.
Population representativeness is the degree of similarity between
the distribution of the values of the variables in a sample and their dis- tribution in the population from which the sample is drawn as well as
the degree of similarity between the causal relations in the sample and in the population. The actual degree of population representativeness
can be determined in principle because it is possible in principle though usually unfeasible to determine the distribution of values of
the variable characteristics of all instances in the population. The degree of population representativeness of a probability sample can be
estimated if the distribution of the values of the variables in the instances of the sample is known.
External validity is a characteristic of a study outcome. External valid-
ity is the extent to which the outcome of a study in one instance or in a group of instances applies or can be generalized to instances other
than those in the study. Two important forms of external validity are ecological validity and statistical generalizability. Ecological validity is
the extent to which the outcome of a laboratory experiment applies to instances of the object of study in its real life context. Statistical gener-
alizability
is the likelihood that research results obtained in a sample of a population are also true for the population.
Generalizability is a characteristic of a proposition and therefore of
a theory. It is the degree of confidence that a proposition is correct and applies to the entire theoretical domain. Generalizability is enhanced
if the proposition is supported in a series of replications. Generalizability decreases if the proposition is not supported in a number of such tests.
The alleged lack of “generalizability” of the case study is a misunderstand- ing. First, generalizability is not a characteristic of a study but of a
proposition. Second, external validity which is a characteristic of a study’s outcome is not an issue in most forms of case study research
because usually there is no population to which results are “general- ized” with exception of the quasi survey; see Chapters 4 and 7. Third,
cases in case study research are equally unrepresentative of a theoretical domain as populations in survey research.
There is, however, a general “lack of generalizability” of propositions in the sense that most propositions are tested only once in one-shot studies.
This problem, however, applies in principle to all types of propositions, irrespective of the research strategy by which they are tested. With
more replication studies, the generalizability of propositions could be enhanced. Generalization, thus, is an aim rather than a claim. It is
something a research community aims to be able to do after a series of replications rather than claiming to be able to do on the basis of an
assumed degree of representativeness of the instances in which a test was conducted.