708 C OHEN ,S ANBORN , AND S HIFFRIN
708 C OHEN ,S ANBORN , AND S HIFFRIN
acceptance decisions could well be significantly lower ease of use, it is not unreasonable to expect the eventual than the decision reached with the better method by itself. production of packaged programs placing the best state-of- This was seen, for example, in the case of the power law the-art methods within reach of the average researcher. versus exponential, with few data per individual, where individual analysis was essentially at chance. Applying a
AUTHOR NOTE
joint agreement heuristic in such a situation is equivalent to randomly selecting one half of the group analysis cases.
All the authors contributed equally to this article. A.N.S. was sup-
It is clear that such a procedure can only reduce the effi- ported by a National Defense Science and Engineering Graduate Fellow-
ship and an NSF Graduate Research Fellowship. R.M.S. was supported
cacy of group analysis. Nonetheless, the joint acceptance by NIMH Grants 1 R01 MH12717 and 1 R01 MH63993. The authors heuristic may be the best strategy available in practice.
thank Woojae Kim, Jay Myung, Michael Ross, Eric-Jan Wagenmak-
We tested this joint acceptance heuristic on the simulated ers, Trisha Van Zandt, Michael Lee, and Daniel Navarro for their help in the preparation of the manuscript and Dominic Massaro, John Paul data discussed previously with uninformed parameter se- Minda, David Rubin, and John Wixted for the use of their data. Cor- lection. The results are given in Figure 6 for the four model respondence concerning this article should be addressed to A. L. Cohen, comparisons. The x- and y-axes of each panel are the differ- Department of Psychology, University of Massachusetts, Amherst, MA ent levels of individuals per experiment and trials per con- 01003 (e-mail: [email protected]). dition, respectively, discussed above. Simulations with one individual per experiment are not included in the figures, be-
REFERENCES
cause average and individual analyses always agree. For each Akaike, H. (1973). Information theory as an extension of the maximum model comparison, the upper left panel gives the proportion
likelihood principle. In B. N. Petrov & F. Csaki (Eds.), Second Inter-
of simulations for which the average and individual analy-
national Symposium on Information Theory (pp. 267-281). Budapest:
ses agreed. The upper right panel indicates the proportion
Akademiai Kiado.
of simulations for which the incorrect model was selected Anderson, N. H. (1981). Foundations of information integration theory. New York: Academic Press. when average and individual analyses agreed. The lower left Anderson, R. B., & Tweney, R. D. (1997). Artifactual power curves in and lower right panels give the proportion of simulations in
forgetting. Memory & Cognition, 25, 724-730.
which average and individual data, respectively, were used Ashby, F. G., & Gott, R. E. (1988). Decision rules in the perception and for which the incorrect model was selected when the average
categorization of multidimensional stimuli. Journal of Experimental
and individual analyses disagreed. Light and dark shadings Psychology: Learning, Memory, & Cognition, 14, 33-53.
Ashby, F. G., & Maddox, W. T. (1993). Relations between prototype,
indicate high and low proportions, respectively.
exemplar, and decision bound models of categorization. Journal of
First, note that, in general, agreement between group
Mathematical Psychology, 37, 372-400.
and individual analyses was high. The only exception was Ashby, F. G., Maddox, W. T., & Lee, W. W. (1994). On the dangers of averaging across subjects when using multidimensional scaling or the for the exponential and power simulations, especially for
similarity-choice model. Psychological Science, 5, 144-151.
a large number of subjects with few trials. Second, when Barron, A. R., Rissanen, J., & Yu, B. (1998). The MDL principle in average and individual analyses agreed, model selection
modeling and coding. IEEE Transactions on Information Theory, 44,
was nearly perfect for most model pairs. The accuracy
2743-2760.
for the exponential and power simulations was less than Berger, J. O., & Pericchi, L. R. (1996). The intrinsic Bayes factor for model selection and prediction. Journal of the American Statistical that for the other model pairs but was still good. Third,
Association, 91, 109-122.
the simulations do not suggest a clear choice for which Bower, G. H. (1961). Application of a model to paired-associate learn- analyses to favor when average and individual analyses
ing. Psychometrika, 26, 255-280.
disagree. For the exponential and power simulations, av- Browne, M. W. (2000). Cross-validation methods. Journal of Math- ematical Psychology, 44, 108-132. erage analysis clearly performs better, and interestingly, Burnham, K. P., & Anderson, D. R. (1998). Model selection and for few subjects and trials, group analysis is even better
inference: A practical information-theoretic approach. New York:
when it disagrees with individual analysis. For the FLMP
Springer.
and LIM, individual analysis is preferred when there is Burnham, K. P., & Anderson, D. R. (2002). Model selection and disagreement. There is no obvious recommendation for
multimodel inference: A practical information-theoretic approach
the GCM and prototype simulations. In our simulations, (2nd ed.). New York: Springer.
Busemeyer, J. R., & Wang, Y.-M. (2000). Model comparisons and
the joint acceptance heuristic tends to outperform even the
model selections based on generalization criterion methodology. Jour-
best single method of analysis.
nal of Mathematical Psychology, 44, 171-189.
The field of model selection and data analysis method- Cover, T. M., & Thomas, J. A. (1991). Elements of information theory. New York: Wiley. ology continues to evolve, of course, and although there Estes, W. K. (1956). The problem of inference from curves based on may be no single best approach, the methods continue to
group data. Psychological Bulletin, 53, 134-140.
improve. Researchers have, for example, already begun to Estes, W. K., & Maddox, W. T. (2005). Risks of drawing inferences explore hierarchical analyses. Hierarchical analyses hold
about cognitive processes from model fits to individual versus average
the potential of combining the best elements of both in-
performance. Psychonomic Bulletin & Review, 12, 403-408.
Grünwald, P. [D.] dividual and group methods (see, e.g., Navarro, Griffiths, (2000). Model selection based on minimum descrip-
tion length. Journal of Mathematical Psychology, 44, 133-152.
Steyvers, & Lee, 2006). The technical methods of model Grünwald, P. D. (2005). Minimum description length tutorial. In P. D. selection also continue to evolve in other ways. We can,
Grünwald, I. J. Myung, & M. A. Pitt (Eds.), Advances in minimum
for example, expect to see methods combining elements of
description length: Theory and applications (pp. 23-80). Cambridge,
BMS and NML (e.g., Grünwald, 2007). We end by noting MA: MIT Press.
Grünwald, P. D. (2007). The minimum description length principle.
that although advanced research in model selection tech-
Cambridge, MA: MIT Press.
niques tends not to focus on the issues of accessibility and Hastie, T., Tibshirani, R., & Friedman, J. (2001). The elements of sta-