residing on the sampled plot. The sampled unit is the plot of land, the observation unit would be the individual plants.
Sampling frame: The list of sampling units. For a mailed survey, it may be a
list of addresses of households in a city. For an ecological study, it may be a map of areas downstream from power plants.
In a perfect survey, the target population would be the same as the sampled popu- lation. This type of survey rarely happens. There are always difficulties in obtain-
ing a sampling frame or being able to identify all elements within the target population. A particular aspect of this problem is nonresponse. Even if the re-
searcher was able to obtain a list of all individuals in the target population, there may be a distinct subset of the target population which refuses to fill out the survey
or allow themselves to be observed. Thus, the sampled population becomes a sub- set of the target population. An attempt at characterizing the nonresponders is
very crucial in attempting to use a sample to describe a population. The group of nonresponders may have certain demographics or a particular political leaning
that if not identified could greatly distort the results of the survey. An excellent dis- cussion of this topic can be found in the textbook, Sampling: Design and Analysis
by Sharon L. Lohr 1999, Pacific Grove, CA: Duxbury Press.
The basic design simple random sampling consists of selecting a group of n
units in such a way that each sample of size n has the same chance of being selected. Thus, we can obtain a random sample of eligible voters in a bond-issue poll by drawing
names from the list of registered voters in such a way that each sample of size n has the same probability of selection. The details of simple random sampling are discussed in
Section 4.11. At this point, we merely state that a simple random sample will contain as much information on community preference as any other sample survey design,
provided all voters in the community have similar socioeconomic backgrounds.
Suppose, however, that the community consists of people in two distinct in- come brackets, high and low. Voters in the high-income bracket may have opinions
on the bond issue that are quite different from the opinions of low-income bracket voters. Therefore, to obtain accurate information about the population, we want
to sample voters from each bracket. We can divide the population elements into two groups, or strata, according to income and select a simple random sample
from each group. The resulting sample is called a stratified random sample. See Chapter 5 of Scheaffer et al., 2006. Note that stratification is accomplished by
using knowledge of an auxiliary variable, namely, personal income. By stratifying on high and low values of income, we increase the accuracy of our estimator.
Ratio estimation is a second method for using the information contained in an aux- iliary variable. Ratio estimators not only use measurements on the response of
interest but they also incorporate measurements on an auxiliary variable. Ratio estimation can also be used with stratified random sampling.
Although individual preferences are desired in the survey, a more economi- cal procedure, especially in urban areas, may be to sample specific families, apart-
ment buildings, or city blocks rather than individual voters. Individual preferences can then be obtained from each eligible voter within the unit sampled. This tech-
nique is called cluster sampling. Although we divide the population into groups for both cluster sampling and stratified random sampling, the techniques differ. In
stratified random sampling, we take a simple random sample within each group, whereas in cluster sampling, we take a simple random sample of groups and then
sample all items within the selected groups clusters. See Chapters 8 and 9 of Scheaffer et al., 2006, for details.
simple random sampling sampling frame
stratified random sample
ratio estimation
cluster sampling
Sometimes, the names of persons in the population of interest are available in a list, such as a registration list, or on file cards stored in a drawer. For this situ-
ation, an economical technique is to draw the sample by selecting one name near the beginning of the list and then selecting every tenth or fifteenth name thereafter.
If the sampling is conducted in this manner, we obtain a systematic sample. As you might expect, systematic sampling offers a convenient means of obtaining sample
information; unfortunately, we do not necessarily obtain the most information for a specified amount of money. Details are given in Chapter 7 of Scheaffer et al.,
2006.
The following example will illustrate how the goal of the study or the infor- mation available about the elements of the population determine which type of
sampling design to use in a particular study.
EXAMPLE 2.3
Identify the type of sampling design in each of the following situations.
a.
The selection of 200 people to serve as potential jurors in a medical malpractice trial is conducted by assigning a number to each of 140,000
registered voters in the county. A computer software program is used to randomly select 200 numbers from the numbers 1 to 140,000. The people
having these 200 numbers are sent a postcard notifying them of their selection for jury duty.
b.
Suppose you are selecting microchips from a production line for inspec- tion for bent probes. As the chips proceed past the inspection point,
every 100th chip is selected for inspection.
c.
The Internal Revenue Service wants to estimate the amount of personal deductions taxpayers made based on the type of deduction: home office,
state income tax, property taxes, property losses, and charitable contri- butions. The amount claimed in each of these categories varies greatly
depending on the adjusted gross income of the taxpayer. Therefore, a simple random sample would not be an efficient design. The IRS decides
to divide taxpayers into five groups based on their adjusted gross incomes and then takes a simple random sample of taxpayers from each of the
five groups.
d.
The USDA inspects produce for E. coli contamination. As trucks carry- ing produce cross the border, the truck is stopped for inspection. A ran-
dom sample of five containers is selected for inspection from the hundreds of containers on the truck. Every apple in each of the five con-
tainers is then inspected for E. coli.
Solution
a.
A simple random sample is selected using the list of registered voters as the sampling frame.
b.
This is an example of systematic random sampling. This type of inspec- tion should provide a representative sample of chips because there is no
reason to presume that there exists any cyclic variation in the production of the chips. It would be very difficult in this situation to perform simple
random sampling because no sampling frame exists.
c.
This is an example of stratified random sampling with the five levels of personal deductions serving as the strata. Overall the personal deductions
of taxpayers increase with income. This results in the stratified random
systematic sample
sample having a much smaller total sample size than would be required in a simple random sample to achieve the same level of precision in its
estimators.
d.
This is a cluster sampling design with the clusters being the containers and the individual apples being the measurement unit.
The important point to understand is that there are different kinds of surveys that can be used to collect sample data. For the surveys discussed in this text, we
will deal with simple random sampling and methods for summarizing and analyz- ing data collected in such a manner. More complicated surveys lead to even more
complicated problems at the summarization and analysis stages of statistics.
The American Statistical Association http:www.amstat.org publishes a series of brochures on surveys: What Is a Survey? How to Plan a Survey, How to
Collect Survey Data, Judging the Quality of a Survey, How to Conduct Pretesting, What Are Focus Groups? and More about Mail Surveys. These describe many
of the elements crucial to obtaining a valid and useful survey. They list many of the potential sources of errors commonly found in surveys with guidelines on how
to avoid these pitfalls. A discussion of some of the issues raised in these brochures follows.
Problems Associated with Surveys
Even when the sample is selected properly, there may be uncertainty about whether the survey represents the population from which the sample was selected.
Two of the major sources of uncertainty are nonresponse, which occurs when a portion of the individuals sampled cannot or will not participate in the survey, and
measurement problems, which occur when the respondent’s answers to questions do not provide the type of data that the survey was designed to obtain.
Survey nonresponse may result in a biased survey because the sample is not
representative of the population. It is stated in Judging the Quality of a Survey that in surveys of the general population women are more likely to participate than
men; that is, the nonresponse rate for males is higher than for females. Thus, a po- litical poll may be biased if the percentage of women in the population in favor of
a particular issue is larger than the percentage of men in the population supporting the issue. The poll would overestimate the percentage of the population in favor of
the issue because the sample had a larger percentage of women than their percent- age in the population. In all surveys, a careful examination of the nonresponse
group must be conducted to determine whether a particular segment of the popu- lation may be either under- or overrepresented in the sample. Some of the remedies
for nonresponse are
1.
Offering an inducement for participating in the survey
2.
Sending reminders or making follow-up telephone calls to the individuals who did not respond to the first contact
3.
Using statistical techniques to adjust the survey findings to account for the sample profile differing from the population profile
Measurement problems are the result of the respondents not providing the in-
formation that the survey seeks. These problems often are due to the specific word- ing of questions in a survey, the manner in which the respondent answers the survey
survey nonresponse
measurement problems
questions, and the fashion in which an interviewer phrases questions during the in- terview. Examples of specific problems and possible remedies are as follows:
1.
Inability to recall answers to questions: The interviewee is asked how many times he or she visited a particular city park during the past year.
This type of question often results in an underestimate of the average number of times a family visits the park during a year because people
often tend to underestimate the number of occurrences of a common event or an event occurring far from the time of the interview. A possible
remedy is to request respondents to use written records or to consult with other family members before responding.
2.
Leading questions: The fashion in which an opinion question is posed may result in a response that does not truly represent the interviewee’s opinion.
Thus, the survey results may be biased in the direction in which the ques- tion is slanted. For example, a question concerning whether the state
should impose a large fine on a chemical company for environmental violations is phrased as, “Do you support the state fining the chemical
company, which is the major employer of people in our community, con- sidering that this fine may result in their moving to another state?” This
type of question tends to elicit a “no” response and thus produces a dis- torted representation of the community’s opinion on the imposition of the
fine. The remedy is to write questions carefully in an objective fashion.
3.
Unclear wording of questions: An exercise club attempted to determine the number of times a person exercises per week. The question asked of
the respondent was, “How many times in the last week did you exercise?” The word exercise has different meanings to different individuals. The
result of allowing different definitions of important words or phrases in survey questions is to greatly reduce the accuracy of survey results.
Several remedies are possible: The questions should be tested on a variety of individuals prior to conducting the survey to determine
whether there are any confusing or misleading terms in the questions. During the training of the interviewer, all interviewers should have the
“correct” definitions of all key words and be advised to provide these definitions to the respondents.
Many other issues, problems, and remedies are provided in the brochures from the ASA.
The stages in designing, conducting, and analyzing a survey are contained in Figure 2.1, which has been reproduced from an earlier version of What Is a Survey?
in Cryer and Miller 1991, Statistics for Business: Data Analysis and Modeling, Boston, PWS-Kent. This diagram provides a guide for properly conducting a suc-
cessful survey.
Data Collection Techniques
Having chosen a particular sample survey, how does one actually collect the data? The most commonly used methods of data collection in sample surveys are personal
interviews and telephone interviews. These methods, with appropriately trained interviewers and carefully planned callbacks, commonly achieve response rates of
60 to 75 and sometimes even higher. A mailed questionnaire sent to a specific group of interested persons can sometimes achieve good results, but generally the
response rates for this type of data collection are so low that all reported results are suspect. Frequently, objective information can be found from direct observation
rather than from an interview or mailed questionnaire.
Data are frequently obtained by personal interviews. For example, we can
use personal interviews with eligible voters to obtain a sample of public sentiment toward a community bond issue. The procedure usually requires the interviewer
to ask prepared questions and to record the respondent’s answers. The primary advantage of these interviews is that people will usually respond when confronted
in person. In addition, the interviewer can note specific reactions and eliminate misunderstandings about the questions asked. The major limitations of the per-
sonal interview aside from the cost involved concern the interviewers. If they are not thoroughly trained, they may deviate from the required protocol, thus in-
troducing a bias into the sample data. Any movement, facial expression, or state- ment by the interviewer can affect the response obtained. For example, a leading
question such as “Are you also in favor of the bond issue?” may tend to elicit a positive response. Finally, errors in recording the responses can lead to erroneous
results.
Information can also be obtained from persons in the sample through
telephone interviews. With the competition among telephone service providers, an interviewer can place any number of calls to specified areas of the country
relatively inexpensively. Surveys conducted through telephone interviews are frequently less expensive than personal interviews, owing to the elimination of
travel expenses. The investigator can also monitor the interviews to be certain that the specified interview procedure is being followed.
A major problem with telephone surveys is that it is difficult to find a list or directory that closely corresponds to the population. Telephone directories have
many numbers that do not belong to households, and many households have unlisted numbers. A technique that avoids the problem of unlisted numbers is random-digit
dialing. In this method, a telephone exchange number the first three digits of a seven-digit number is selected, and then the last four digits are dialed randomly
until a fixed number of households of a specified type are reached. This technique produces samples from the target population but most random digit-dialing samples
include only landline numbers. Thus, the increasing number of households with cell phones only are excluded. Also, many people screen calls before answering a call.
These two problems are creating potentially large biases in telephone surveys.
Telephone interviews generally must be kept shorter than personal inter- views because responders tend to get impatient more easily when talking over the
telephone. With appropriately designed questionnaires and trained interviewers, telephone interviews can be as successful as personal interviews.
personal interviews
Original study idea
Report preparation
Interviewer hiring
Interviewer training
Questionnaire preparation
Pretest Revision of
operational plan
Questionnaire revision
Data collection
Data processing
Data analysis
Final report outline
Preliminary operational
plan Interviewer
hiring Final sample
design Listing
work Sample
selection
FIGURE 2.1
Stages of a survey
telephone interviews
Another useful method of data collection is the self-administered question- naire, to be completed by the respondent. These questionnaires usually are mailed
to the individuals included in the sample, although other distribution methods can be used. The questionnaire must be carefully constructed if it is to encourage par-
ticipation by the respondents.
The self-administered questionnaire does not require interviewers, and thus its use results in savings in the survey cost. This savings in cost is usually bought at
the expense of a lower response rate. Nonresponse can be a problem in any form of data collection, but since we have the least contact with respondents in a mailed
questionnaire, we frequently have the lowest rate of response. The low response rate can introduce a bias into the sample because the people who answer question-
naires may not be representative of the population of interest. To eliminate some of the bias, investigators frequently contact the nonrespondents through follow-up
letters, telephone interviews, or personal interviews.
The fourth method for collecting data is direct observation. If we were in-
terested in estimating the number of trucks that use a particular road during the 4 – 6
P
.
M
. rush hours, we could assign a person to count the number of trucks passing a specified point during this period, or electronic counting equipment
could be used. The disadvantage in using an observer is the possibility of error in observation.
Direct observation is used in many surveys that do not involve measurements on people. The USDA measures certain variables on crops in sections of fields in
order to produce estimates of crop yields. Wildlife biologists may count animals, animal tracks, eggs, or nests to estimate the size of animal populations.
A closely related notion to direct observation is that of getting data from objective sources not affected by the respondents themselves. For example, health
information can sometimes be obtained from hospital records, and income infor- mation from employer’s records especially for state and federal government work-
ers. This approach may take more time but can yield large rewards in important surveys.
2.4 Experimental Studies
An experimental study may be conducted in many different ways. In some studies, the researcher is interested in collecting information from an undisturbed natural
process or setting. An example would be a study of the differences in reading scores of second-grade students in public, religious, and private schools. In other studies,
the scientist is working within a highly controlled laboratory, a completely artificial setting for the study. For example, the study of the effect of humidity and tempera-
ture on the length of the life cycles of ticks would be conducted in a laboratory since it would be impossible to control the humidity or temperature in the tick’s natural
environment. This control of the factors under study allows the entomologist to obtain results that can then be more easily attributed to differences in the levels
of the temperature and humidity, since nearly all other conditions remain constant throughout the experiment. In a natural setting, many other factors are varying
and they may also result in changes in the life cycles of the ticks. However, the greater the control in these artificial settings, the less likely the experiment is por-
traying the true state of nature. A careful balance between control of conditions and depiction of a reality must be maintained in order for the experiments to be
useful. In this section and the next one, we will present some standard designs of
self-administered questionnaire
direct observation
experiments. In experimental studies, the researcher controls the crucial factors by one of two methods.
Method 1: The subjects in the experiment are randomly assigned to
the treatments. For example, ten rats are randomly assigned to each of the four dose levels of an experimental drug under investigation.
Method 2: Subjects are randomly selected from different populations
of interest. For example, 50 male and 50 female dogs are randomly selected from animal shelters in large and small cities and tested for
the presence of heart worms.
In Method 1, the researcher randomly selects experimental units from a homoge- neous population of experimental units and then has complete control over the as-
signment of the units to the various treatments. In Method 2, the researcher has control over the random sampling from the treatment populations but not over the
assignment of the experimental units to the treatments.
In experimental studies, it is crucial that the scientist follows a systematic plan established prior to running the experiment. The plan includes how all ran-
domization is conducted, either the assignment of experimental units to treat- ments or the selection of units from the treatment populations. There may be
extraneous factors present that may affect the experimental units. These factors may be present as subtle differences in the experimental units or slight differences
in the surrounding environment during the conducting of the experiment. The ran- domization process ensures that, on the average, any large differences observed
in the responses of the experimental units in different treatment groups can be attributed to the differences in the groups and not to factors that were not con-
trolled during the experiment. The plan should also include many other aspects on how to conduct the experiment. A list of some of the items that should be included
in such a plan are listed here:
1.
The research objectives of the experiment
2.
The selection of the factors that will be varied the treatments
3.
The identification of extraneous factors that may be present in the ex- perimental units or in the environment of the experimental setting the
blocking factors
4.
The characteristics to be measured on the experimental units response variable
5.
The method of randomization, either randomly selecting from treatment populations or the random assignment of experimental units to treatments
6.
The procedures to be used in recording the responses from the experi- mental units
7.
The selection of the number of experimental units assigned to each treatment may require designating the level of significance and power of
tests or the precision and reliability of confidence intervals
8.
A complete listing of available resources and materials
Terminology
A designed experiment is an investigation in which a specified framework is pro- vided in order to observe, measure, and evaluate groups with respect to a desig-
nated response. The researcher controls the elements of the framework during the
designed experiment
