Observational Studies Lyman Ott Michael Longnecker

control the researcher has over the physical setting in which the observations are made. In an experimental study, the researcher attempts to maintain control over all variables that may have an effect on the response variables. Observational studies may be dichotomized into either a comparative study or descriptive study. In a comparative study, two or more methods of achieving a result are compared for effectiveness. For example, three types of healthcare de- livery methods are compared based on cost effectiveness. Alternatively, several groups are compared based on some common attribute. For example, the starting income of engineers are contrasted from a sample of new graduates from private and public universities. In a descriptive study, the major purpose is to characterize a population or process based on certain attributes in that population or process— for example, studying the health status of children under the age of 5 years old in families without health insurance or assessing the number of overcharges by companies hired under federal military contracts. Observational studies in the form of polls, surveys, and epidemiological studies, for example, are used in many different settings to address questions posed by researchers. Surveys are used to measure the changing opinion of the nation with respect to issues such as gun control, interest rates, taxes, the mini- mum wage, Medicare, and the national debt. Similarly, we are informed on a daily basis through newspapers, magazines, television, radio, and the Internet of the re- sults of public opinion polls concerning other relevant and sometimes irrelevant political, social, educational, financial, and health issues. In an observational study, the factors treatments of interest are not manip- ulated while making measurements or observations. The researcher in an environ- mental impact study is attempting to establish the current state of a natural setting from which subsequent changes may be compared. Surveys are often used by natu- ral scientists as well. In order to determine the proper catch limits of commercial and recreational fishermen in the Gulf of Mexico, the states along the Gulf of Mexico must sample the Gulf to determine the current fish density. There are many biases and sampling problems that must be addressed in order for the survey to be a reliable indicator of the current state of the sampled population. A problem that may occur in observational studies is assigning cause- and-effect relationships to spurious associations between factors. For example, in many epidemiological studies we study various environmental, social, and ethnic factors and their relationship with the incidence of certain diseases. A public health question of considerable interest is the relationship between heart disease and the amount of fat in one’s diet. It would be unethical to randomly assign vol- unteers to one of several high-fat diets and then monitor the people over time to observe whether or not heart disease develops. Without being able to manipulate the factor of interest fat content of the diet, the scientist must use an observational study to address the issue. This could be done by comparing the diets of a sample of people with heart disease with the diets of a sample of people without heart disease. Great care would have to be taken to record other relevant factors such as family history of heart disease, smok- ing habits, exercise routine, age, and gender for each person, along with other physical characteristics. Models could then be developed so that differences be- tween the two groups could be adjusted to eliminate all factors except fat content of the diet. Even with these adjustments, it would be difficult to assign a cause-and- effect relationship between high fat content of a diet and the development of heart disease. In fact, if the dietary fat content for the heart disease group tended to be higher than that for the group free of heart disease after adjusting for relevant comparative study descriptive study cause-and-effect relationships factors, the study results would be reported as an association between high dietary fat content and heart disease, not a causal relationship. Stated differently, in observational studies we are sampling from populations where the factors or treatments are already present and we compare samples with respect to the factors treatments of interest to the researcher. In contrast, in the controlled environment of an experimental study, we are able to randomly assign the people as objects under study to the factors or treatments and then observe the response of interest. For our heart disease example, the distinction is shown here: Observational study: We sample from the heart disease population and heart disease–free population and compare the fat content of the diets for the two groups. Experimental study: Ignoring ethical issues, we would assign volunteers to one of several diets with different levels of dietary fat the treatments and compare the different treatments with respect to the response of in- terest incidence of heart disease after a period of time. Observational studies are of three basic types: ● A sample survey is a study that provides information about a population at a particular point in time current information. ● A prospective study is a study that observes a population in the present using a sample survey and proceeds to follow the subjects in the sample forward in time in order to record the occurrence of specific outcomes. ● A retrospective study is a study that observes a population in the present using a sample survey and also collects information about the subjects in the sample regarding the occurrence of specific outcomes that have already taken place. In the health sciences, a sample survey would be referred to as a cross-sectional or prevalence study. All individuals in the survey would be asked about their current disease status and any past exposures to the disease. A prospective study would identify a group of disease-free subjects and then follow them over a period of time until some of the individuals develop the disease. The development or nondevel- opment of the disease would then be related to other variables measured on the subjects at the beginning of the study, often referred to as exposure variables. A retrospective study identifies two groups of subjects: cases—subjects with the disease—and controls—subjects without the disease. The researcher then attempts to correlate the subjects prior health habits to their current health status. Although prospective and retrospective studies are both observational stud- ies, there are some distinct differences. ● Retrospective studies are generally cheaper and can be completed more rapidly than prospective studies. ● Retrospective studies have problems due to inaccuracies in data due to recall errors. ● Retrospective studies have no control over variables that may affect dis- ease occurrence. ● In prospective studies subjects can keep careful records of their daily activities ● In prospective studies subjects can be instructed to avoid certain activities that may bias the study association causal sample survey prospective study retrospective study ● Although prospective studies reduce some of the problems of retrospec- tive studies, they are still observational studies and hence the potential in- fluences of confounding variables may not be completely controlled. It is possible to somewhat reduce the influence of the confounding variables by restricting the study to matched subgroups of subjects. Both prospective and retrospective studies are often comparative in nature. Two specific types of such studies are cohort studies and case-control studies. In a co- hort study, a group of subjects is followed forward in time to observe the differ- ences in characteristics of subjects who develop a disease with those who do not. Similarly, we could observe which subjects commit crimes while also recording in- formation about their educational and social backgrounds. In case-control studies, two groups of subjects are identified, one with the disease and one without the dis- ease. Next, information is gathered about the subjects from their past concerning risk factors that are associated with the disease. Distinctions are then drawn about the two groups based on these characteristics. EXAMPLE 2.1 A study was conducted to determine if women taking oral contraceptives had a greater propensity to develop heart disease. A group of 5,000 women currently using oral contraceptives and another group of 5,000 women not using oral contra- ceptives were selected for the study. At the beginning of the study, all 10,000 women were given physicals and were found to have healthy hearts. The women’s health was then tracked for a 3-year period. At the end of the study, 15 of the 5,000 users had developed a heart disease, whereas only 3 of the nonusers had any evidence of heart disease. What type of design was this observational study? Solution This study is an example of a prospective observational study. All women were free of heart disease at the beginning of the study and their exposure oral contraceptive use measured at that time. The women were then under ob- servation for 3 years, with the onset of heart disease recorded if it occurred during the observation period. A comparison of the frequency of occurrence of the disease is made between the two groups of women, users and nonusers of oral contraceptives. EXAMPLE 2.2 A study was designed to determine if people who use public transportation to travel to work are more politically active than people who use their own vehicle to travel to work. A sample of 100 people in a large urban city was selected from each group and then all 200 individuals were interviewed concerning their political activities over the past 2 years. Out of the 100 people who used public transportation, 18 reported that they had actively assisted a candidate in the past 2 years, whereas only 9 of the 100 persons who used their own vehicles stated they had participated in a political campaign. What type of design was this study? Solution This study is an example of a retrospective observational study. The in- dividuals in both groups were interviewed about their past experiences with the po- litical process. A comparison of the degree of participation of the individuals was made across the two groups. cohort studies case-control studies In Example 2.2, many of the problems with using observational studies are present. There are many factors that may affect whether or not an individual decides to par- ticipate in a political campaign. Some of these factors may be confounded with rid- ership on public transportation—for example, awareness of the environmental impact of vehicular exhaust on air pollution, income level, and education level. These factors need to be taken into account when designing an observational study. The most widely used observational study is the survey. Information from surveys impact nearly every facet of our daily lives. Government agencies use sur- veys to make decisions about the economy and many social programs. News agen- cies often use opinion polls as a basis of news reports. Ratings of television shows, which come from surveys, determine which shows will be continued for the next television season. Who conducts surveys? The various news organizations all use public opin- ion polls: Such surveys include the New York TimesCBS News, Washington Post ABC News, Wall Street JournalNBC News, Harris, GallupNewsweek, and CNN Time polls. However, the vast majority of surveys are conducted for a spe- cific industrial, governmental, administrative, political, or scientific purpose. For example, auto manufacturers use surveys to find out how satisfied customers are with their cars. Frequently we are asked to complete a survey as part of the war- ranty registration process following the purchase of a new product. Many impor- tant studies involving health issues are determined using surveys—for example, amount of fat in a diet, exposure to secondhand smoke, condom use and the prevention of AIDS, and the prevalence of adolescent depression. The U.S. Bureau of the Census is required by the U.S. Constitution to enu- merate the population every 10 years. With the growing involvement of the gov- ernment in the lives of its citizens, the Census Bureau has expanded its role beyond just counting the population. An attempt is made to send a census questionnaire in the mail to every household in the United States. Since the 1940 census, in ad- dition to the complete count information, further information has been obtained from representative samples of the population. In the 2000 census, variable sam- pling rates were employed. For most of the country, approximately five of six households were asked to answer the 14 questions on the short version of the form. The remaining households responded to a longer version of the form con- taining an additional 45 questions. Many agencies and individuals use the resulting information for many purposes. The federal government uses it to determine allo- cations of funds to states and cities. Businesses use it to forecast sales, to manage personnel, and to establish future site locations. Urban and regional planners use it to plan land use, transportation networks, and energy consumption. Social sci- entists use it to study economic conditions, racial balance, and other aspects of the quality of life. The U.S. Bureau of Labor Statistics BLS routinely conducts more than 20 surveys. Some of the best known and most widely used are the surveys that estab- lish the consumer price index CPI. The CPI is a measure of price change for a fixed market basket of goods and services over time. It is a measure of inflation and serves as an economic indicator for government policies. Businesses tie wage rates and pension plans to the CPI. Federal health and welfare programs, as well as many state and local programs, tie their bases of eligibility to the CPI. Escalator clauses in rents and mortgages are based on the CPI. This one index, determined on the basis of sample surveys, plays a fundamental role in our society. Many other surveys from the BLS are crucial to society. The monthly Current Population Survey establishes basic information on the labor force, employment, and unemployment. The consumer expenditure surveys collect data on family expenditures for goods and services used in day-to-day living. The Establishment Survey collects information on employment hours and earnings for nonagricul- tural business establishments. The survey on occupational outlook provides infor- mation on future employment opportunities for a variety of occupations, projecting to approximately 10 years ahead. Other activities of the BLS are addressed in the BLS Handbook of Methods web version: www.bls.govopubhom. Opinion polls are constantly in the news, and the names of Gallup and Harris have become well known to everyone. These polls, or sample surveys, reflect the atti- tudes and opinions of citizens on everything from politics and religion to sports and entertainment. The Nielsen ratings determine the success or failure of TV shows. How do you figure out the ratings? Nielsen Media Research NMR continu- ally measures television viewing with a number of different samples all across the United States. The first step is to develop representative samples. This must be done with a scientifically drawn random selection process. No volunteers can be ac- cepted or the statistical accuracy of the sample would be in jeopardy. Nationally, there are 5,000 television households in which electronic meters called People Meters are attached to every TV set, VCR, cable converter box, satellite dish, or other video equipment in the home. The meters continually record all set tunings. In addition, NMR asks each member of the household to let them know when they are watching by pressing a pre-assigned button on the People Meter. By matching this button activity to the demographic information agegender NMR collected at the time the meters were installed, NMR can match the set tuning—what is being watched—with who is watching. All these data are transmitted to NMR’s comput- ers, where they are processed and released to customers each day. In addition to this national service, NMR has a slightly different metering system in 55 local mar- kets. In each of those markets, NMR gathers just the set-tuning information each day from more than 20,000 additional homes. NMR then processes the data and re- leases what are called “household ratings” daily. In this case, the ratings report what channel or program is being watched, but they do not have the “who” part of the picture. To gather that local demographic information, NMR periodically at least four times per year ask another group of people to participate in diary surveys. For these estimates, NMR contacts approximately 1 million homes each year and ask them to keep track of television viewing for 1 week, recording their TV-viewing activity in a diary. This is done for all 210 television markets in the United States in November, February, May, and July and is generally referred to as the “sweeps.” For more information on the Nielsen ratings, go the NMR website www. nielsenmedia.com and click on the “What TV Ratings Really Mean” button. Businesses conduct sample surveys for their internal operations in addition to using government surveys for crucial management decisions. Auditors estimate account balances and check on compliance with operating rules by sampling accounts. Quality control of manufacturing processes relies heavily on sampling techniques. Another area of business activity that depends on detailed sampling activities is marketing. Decisions on which products to market, where to market them, and how to advertise them are often made on the basis of sample survey data. The data may come from surveys conducted by the firm that manufactures the product or may be purchased from survey firms that specialize in marketing data.

2.3 Sampling Designs for Surveys

A crucial element in any survey is the manner in which the sample is selected from the population. If the individuals included in the survey are selected based on con- venience alone, there may be biases in the sample survey, which would prevent the survey from accurately reflecting the population as a whole. For example, a mar- keting graduate student developed a new approach to advertising and, to evaluate this new approach, selected the students in a large undergraduate business course to assess whether the new approach is an improvement over standard advertise- ments. Would the opinions of this class of students be representative of the general population of people to which the new approach to advertising would be applied? The income levels, ethnicity, education levels, and many other socioeconomic char- acteristics of the students may differ greatly from the population of interest. Fur- thermore, the students may be coerced into participating in the study by their instructor and hence may not give the most candid answers to questions on a sur- vey. Thus, the manner in which a sample is selected is of utmost importance to the credibility and applicability of the study’s results. In order to precisely describe the components that are necessary for a sample to be effective, the following definitions are required. Target population: The complete collection of objects whose description is the major goal of the study. Designating the target population is a crucial but often difficult part of the first step in an observational or experimen- tal study. For example, in a survey to decide if a new storm-water drainage tax should be implemented, should the target population be all persons over the age of 18 in the county, all registered voters, or all per- sons paying property taxes? The selection of the target population may have a profound effect on the results of the study. Sample: A subset of the target population. Sampled population: The complete collection of objects that have the potential of being selected in the sample; the population from which the sample is actually selected. In many studies, the sampled population and the target population are very different. This may lead to very erroneous conclusions based on the information collected in the sample. For example, in a telephone survey of people who are on the property tax list the target population, a subset of this population may not answer their telephone if the caller is unknown, as viewed through caller ID. Thus, the sampled population may be quite different from the target population with respect to some important characteristics such as income and opinion on certain issues. Observation unit: The object upon which data are collected. In studies in- volving human populations, the observation unit is a specific individual in the sampled population. In ecological studies, the observation unit may be a sample of water from a stream or an individual plant on a plot of land. Sampling unit: The object that is actually sampled. We may want to sample the person who pays the property tax but may only have a list of tele- phone numbers. Thus, the households in the sampled population serve as the sampled units, and the observation units are the individuals residing in the sampled household. In an entomology study, we may sample 1-acre plots of land and then count the number of insects on individual plants observation unit sampling unit target population sample sampled population residing on the sampled plot. The sampled unit is the plot of land, the observation unit would be the individual plants. Sampling frame: The list of sampling units. For a mailed survey, it may be a list of addresses of households in a city. For an ecological study, it may be a map of areas downstream from power plants. In a perfect survey, the target population would be the same as the sampled popu- lation. This type of survey rarely happens. There are always difficulties in obtain- ing a sampling frame or being able to identify all elements within the target population. A particular aspect of this problem is nonresponse. Even if the re- searcher was able to obtain a list of all individuals in the target population, there may be a distinct subset of the target population which refuses to fill out the survey or allow themselves to be observed. Thus, the sampled population becomes a sub- set of the target population. An attempt at characterizing the nonresponders is very crucial in attempting to use a sample to describe a population. The group of nonresponders may have certain demographics or a particular political leaning that if not identified could greatly distort the results of the survey. An excellent dis- cussion of this topic can be found in the textbook, Sampling: Design and Analysis by Sharon L. Lohr 1999, Pacific Grove, CA: Duxbury Press. The basic design simple random sampling consists of selecting a group of n units in such a way that each sample of size n has the same chance of being selected. Thus, we can obtain a random sample of eligible voters in a bond-issue poll by drawing names from the list of registered voters in such a way that each sample of size n has the same probability of selection. The details of simple random sampling are discussed in Section 4.11. At this point, we merely state that a simple random sample will contain as much information on community preference as any other sample survey design, provided all voters in the community have similar socioeconomic backgrounds. Suppose, however, that the community consists of people in two distinct in- come brackets, high and low. Voters in the high-income bracket may have opinions on the bond issue that are quite different from the opinions of low-income bracket voters. Therefore, to obtain accurate information about the population, we want to sample voters from each bracket. We can divide the population elements into two groups, or strata, according to income and select a simple random sample from each group. The resulting sample is called a stratified random sample. See Chapter 5 of Scheaffer et al., 2006. Note that stratification is accomplished by using knowledge of an auxiliary variable, namely, personal income. By stratifying on high and low values of income, we increase the accuracy of our estimator. Ratio estimation is a second method for using the information contained in an aux- iliary variable. Ratio estimators not only use measurements on the response of interest but they also incorporate measurements on an auxiliary variable. Ratio estimation can also be used with stratified random sampling. Although individual preferences are desired in the survey, a more economi- cal procedure, especially in urban areas, may be to sample specific families, apart- ment buildings, or city blocks rather than individual voters. Individual preferences can then be obtained from each eligible voter within the unit sampled. This tech- nique is called cluster sampling. Although we divide the population into groups for both cluster sampling and stratified random sampling, the techniques differ. In stratified random sampling, we take a simple random sample within each group, whereas in cluster sampling, we take a simple random sample of groups and then sample all items within the selected groups clusters. See Chapters 8 and 9 of Scheaffer et al., 2006, for details. simple random sampling sampling frame stratified random sample ratio estimation cluster sampling