Useful Concepts

8.4 Useful Concepts

  1. Experiments vs. Observational studies:

  Scientific studies are commonly classified into observational and experi- mental depending on whether the independent variable is observational or experimental. An independent variable is Experimental if its value is as-

  signed by the experimenter to the subjects. 1 For example if experimenters

  assign different subjects different dosages of a drug then the drug’s dose is an experimental variable. The crucial point here is that the experimenter, not the subjects, decides which treatment each subject will receive.

  1 Note that being “experimental” is a property of the independent variable, not a property of the dependent variable. If you want to know whether a study is experimental you need to focus on the

  independent variable; do not worry about the dependent variable.

  88 CHAPTER 8. INTRO TO EXPERIMENTAL DESIGN

  If an independent variable is not experimental then it is called observa- tional. For example, age does not qualify as an experimental variable since age cannot be assigned to subjects. All experimenters can do is to classify subjects according to their age.

  The goal of experimental independent variables is to study cause and effect relationships (e.g., to establish whether a drug is or is not a beneficial to treat

  a particular sickness). Observational variables on the other hand can only

  be used to study relationships, without establishing causes and effects. For example, early observational studies about the relationship between smok- ing and cancer showed that smokers tended to have a higher cancer rate than non-smokers. The dependent variable was whether the subject had cancer. The independent variable was whether the person smoked. Note in this case the independent variable is observational, since it would be unethical for ex- perimenters to force a group of subject to smoke. The experimenters could only classify subjects into smokers or non-smokers. Unfortunately, the re- sults of observational studies like this were inconclusive since they could be due to an intervening variable that is systematically different between the two groups. For example highly stressed subjects could have a propensity to smoke and a higher cancer risk. According to this view, stress, and not smoking per se, could cause cancer. It could also be the case that subjects with cancer have a higher propensity to smoke just because cancer stresses you out. According to this view it is cancer that causes smoking!

  A classic example of observational studies involves the analysis of differ- ences between men and women on some dependent variable of interest. Note that since these these studies are observational. We can investigate whether men and women are different on some dependent variable (e.g., memory, IQ, or income) but we cannot assess what the causes are for the obtained differences. These causes could be due to trivial variables, such as the fact that on average men tend to be heavier, or they could be due to com- plex socio-political variables (e.g., the underlying cause for the observed differences could be that the educational system treats men and women dif- ferently causing).

  2. Exploratory vs. Confirmatory Studies: The distinction between these two different types of research is somewhat subtle, yet important. Confir- matory studies tend to have only a small and well defined set of treatments and behaviors under study. Clear a-priori hypotheses are made about what

  8.5. EXERCISES

  to expect in the data and only those a-priori hypothesis are tested. For ex- ample, a confirmatory study may investigate whether taking an aspirin a day reduces the chances of heart attack. In exploratory studies the experimenters may have a general idea about what to expect in the data but they are willing to analyze the data in a very wide variety of ways with the hope of finding potentially interesting results. Exploratory studies are extremely important in science and they are the source in many cases of new unexpected results. However, the results in exploratory studies tend to be inconclusive basically because by analyzing the data in many different ways it is always possible to find some sort of pattern that may just be due to chance. You can see this effect in sports analysis when all sorts of different statistics are used to explain after the fact why a team has lost or won. In general, results of exploratory analysis are treated with caution and attempts are made to replicate these results with well defined confirmatory experiments.

  3. Random Assignment vs. Random Selection: This is a very important

  distinction that novices sometimes confuse. Random selection refers to the process of selecting a sample from a population of subjects. In many experiments this selection does not need to be random. For example, in psychophysical experiments may times the experimenter himself and his colleagues are the subjects of the study, and thus they are not “randomly selected”. However these non-randomly selected subjects may still be ran- domly assigned to the different treatment conditions. Random selection has an effect on the external validity of the experiment, our power to generalize to a larger population. Random assignment has an effect on the internal va- lidity of the experiment, our power to infer that the independent variable is the one causing the observed results.