Abstract
In psychology, measurement instruments are constructed from scales, which are obtained on the grounds of exploratory and confirmatory factor analysis. Looking at the literature, one can find various recommendations regarding how these techniques should be used during the scale construction process. Some authors suggest to use exploratory factor analysis on the entire data set while others advice to perform an internal cross-validation by randomly splitting the data set in two and then either perform exploratory factor analysis on both parts or exploratory factor analysis on the first part and confirmatory factor analysis on the other. In spite of all these divergent recommendations, there is no consensus on which method yields the best result. In this paper, we analyze this issue in light of the prediction versus accommodation debate and argue that the answer to this question depends on one’s conception of the criteria that should be used to achieve the goals of the scientific enterprise.
Access this article
Rent this article via DeepDyve
Similar content being viewed by others
Notes
This paper is primarily concerned with the issue of model selection during scale construction in psychology. With that being said, our analysis should not be considered as restricted only to that field. Our analysis can be extended to other disciplines that use factor analytic methods for the development of measurement instruments (e.g. sociology, education), although some idiosyncratic differences might occur.
Some items might be reversed score items.
See Hood (2013) for an analysis of the issue of realism with respect to measurement models.
In such a measurement model, the items are not used to predict the latent factors. As we will see, a factorial structure is a space where the items are the points.
Note that Penny’s and Rosie’s partitions of D can be different.
Hitchcock and Sober’s analysis takes place within the context of an instrumentalist perspective and, as such, they consider that science should aim at predictive accuracy (see Hitchcock and Sober 2004, pp. 2–3). It should be noted, however, that our analysis is independent of the realism/instrumentalism debate. As we will see, our analysis only relies on the assumption that empirical adequacy is, among others, one goal of the scientific enterprise, and that predictive accuracy is, among others, an indicator of the overall empirical adequacy of a theory.
See also Gardner (1982) for a discussion of novelty.
Conceptually, part of the unexplained variance might be due to something different than error.
Even though the hypothesis of continuity is often violated, it has been shown that, under the assumption of normality, violation of the assumption of continuity is likely negligible (cf. Byrne 2012).
There are other extraction techniques that can be used in cases where the assumption of normality is violated (cf. Flora et al. 2012). Consequently, despite their importance, these two conditions are not necessary for the use of factor analysis during scale construction.
It has been argued by Michell (1997) that, given the violation of the assumption of continuity, measurement of psychological attributes is not possible (see also Michell 2003, 2004). His argument revolves around the assumption that only continuous quantities can be measured and, since the hypothesis that psychological attributes are continuous is not tested, it follows that one is not justified to believe that it is possible to measure psychological attributes. Borsboom and Mellenbergh (2004) answered this objection and showed that this hypothesis is actually tested, though not in isolation.
It should be mentioned that some authors argued that an algorithm that searches for pure measurement models should be used rather than factor analysis techniques to obtain a measurement model (e.g. Silva et al. 2006; Kummerfeld et al. 2014; see also Murray-Watters and Glymour 2015). The analysis of these alternative techniques is beyond the scope of this paper. We will concentrate on factor analysis given its place with regard to scale construction in the psychology literature.
Once a scale has been properly validated, researchers can use different measurement models and study the relationships between the latent factors through structural equation modeling. We will leave the analysis of model selection with respect to structural models for future research.
To some extent, trying to replicate results from previous studies can be understood as a prediction that the results can be generalized.
Thus understood, replication amounts to the reproduction of a result using the same technique.
This would not be the case if one were to use principal component analysis instead of EFA.
References
Abdi, H., & Williams, L. J. (2010). Principal component analysis. Wiley Interdisciplinary Reviews: Computational Statistics, 2(4), 433–459.
Akaike, H. (1987). Factor analysis and AIC. Psychometrika, 52(3), 317–332.
Aron, A., Coups, E. J., & Aron, E. N. (2013). Statistics for psychology (6th ed.). London: Pearson.
Asendorpf, J. B., Conner, M., De Fruyt, F., De Houwer, J., Denissen, J. J. A., Fiedler, K., et al. (2013). Recommendations for increasing replicability in psychology. European Journal of Personality, 27(2), 108–119.
Autzen, B. (2016). Significance testing, p-values and the principle of total evidence. European Journal for Philosophy of Science, 6(2), 281–295.
Bentler, P. M. (2007). On tests and indices for evaluating structural models. Personality and Individual Differences, 42(5), 825–829.
Borsboom, D. (2008). Latent variable theory. Measurement, 6(1–2), 25–53.
Borsboom, D., & Mellenbergh, G. J. (2004). Why psychometrics is not pathological: A comment on Michell. Theory & Psychology, 14(1), 105–120.
Borsboom, D., Mellenbergh, G. J., & van Heerden, J. (2003). The theoretical status of latent variables. Psychological Review, 110(2), 203–219.
Brandt, M. J., IJzerman, H., Dijksterhuis, A., Farach, F. J., Geller, J., Giner-Sorolla, R., et al. (2014). The replication recipe: What makes for a convincing replication? Journal of Experimental Social Psychology, 50, 217–224.
Brown, T. A. (2015). Confirmatory factor analysis for applied research (2nd ed.). New York: Guilford Publications.
Browne, M. W., & Cudeck, R. (1992). Alternative ways of assessing model fit. Sociological Methods & Research, 21(2), 230–258.
Byrne, B. M. (2012). Structural equation modeling with Mplus. New York: Routledge.
Comrey, A. L. (1988). Factor-analytic methods of scale development in personality and clinical psychology. Journal of Consulting and Clinical Psychology, 56(5), 754–761.
Costello, A. B., & Osborne, J. W. (2005). Best practices in exploratory factor analysis: Four recommendations for getting the most from your analysis. Practical Assessment, Research & Evaluation, 10(7), 1–9.
Cudeck, R., & Browne, M. W. (1983). Cross-validation of covariance structures. Multivariate Behavioral Research, 18(2), 147–167.
Edwards, J. R., & Bagozzi, R. P. (2000). On the nature and direction of relationships between constructs and measures. Psychological Methods, 5(2), 155–174.
Fabrigar, L. R., Wegener, D. T., MacCallum, R. C., & Strahan, E. J. (1999). Evaluating the use of exploratory factor analysis in psychological research. Psychological Methods, 4(3), 272–299.
Field, A. (2009). Discovering statistics using SPSS (3rd ed.). London: Sage.
Finch, J. F., & West, S. G. (1997). The investigation of personality structure: Statistical models. Journal of Research in Personality, 31(4), 439–485.
Flora, D. B., LaBrish, C., & Chalmers, R. P. (2012). Old and new ideas for data screening and assumption testing for exploratory and confirmatory factor analysis. Frontiers in Psychology, 3(55), 1–21.
Forster, M. R. (2002). Predictive accuracy as an achievable goal of science. Philosophy of Science, 69(3), S124–S134.
Forster, M., & Sober, E. (1994). How to tell when simpler, more unified, or less ad hoc theories will provide more accurate predictions. British Journal for the Philosophy of Science, 45(1), 1–35.
Francis, G. (2012). Publication bias and the failure of replication in experimental psychology. Psychonomic Bulletin & Review, 19(6), 975–991.
Gardner, M. R. (1982). Predicting novel facts. The British Journal for the Philosophy of Science, 33(1), 1–15.
Glymour, C. (2001). The mind’s arrows: Bayes nets and graphical causal models in psychology. Cambridge: MIT Press.
Hinkin, T. R. (1998). A brief tutorial on the development of measures for use in survey questionnaires. Organizational Research Methods, 1(1), 104–121.
Hitchcock, C., & Sober, E. (2004). Preduction versus accommodation and the risk of overfitting. British Journal for the Philosophy of Science, 55(1), 1–34.
Hood, S. B. (2013). Psychological measurement and methodological realism. Erkenntnis, 78(4), 739–761.
Horn, J. L. (1965). A rationale and test for the number of factors in factor analysis. Psychometrika, 30(2), 179–185.
Howell, D. C. (2010). Statistical methods for psychology (7th ed.). Wadsworth: Cengage Learning.
Hu, L.-T., & Bentler, P. M. (1998). Fit indices in covariance structure modeling: Sensitivity to underparameterized model misspecification. Psychological Methods, 3(4), 424–453.
Huck, S. W. (2012). Reading statistics and research (6th ed.). London: Pearson.
Hurley, A. E., Scandura, T. A., Schriesheim, C. A., Brannick, M. T., Seers, A., Vandenberg, R. J., et al. (1997). Exploratory and confirmatory factor analysis: Guidelines, issues, and alternatives. Journal of Organizational Behavior, 18(6), 667–683.
Johnson, K. (2016). Realism and uncertainty of unobservable common causes in factor analysis. Noûs, 50(2), 329–355.
Jolliffe, I. T. (2002). Principal component analysis (2nd ed.). Berlin: Springer.
Jolliffe, I. T., & Morgan, B. J. T. (1992). Principal component analysis and exploratory factor analysis. Statistical Methods in Medical Research, 1(1), 69–95.
Kline, R. B. (2005). Structural equation modeling. New York: The Guilford Press.
Kuhn, T. S. (1983). Rationality and theory choice. The Journal of Philosophy, 80(10), 563–570.
Kummerfeld, E., Ramsey, J., Yang, R., Spirtes, P., & Scheines, R. (2014). Causal clustering for 2-factor measurement models. In T. Calders, F. Esposito, E. Hüllermeier, & R. Meo (Eds.), Machine learning and knowledge discovery in databases, volume 8725 of lecture notes in computer science (pp. 34–49). Berlin: Springer.
MacCallum, R. C., Roznowski, M., & Necowitz, L. B. (1992). Model modifications in covariance structure analysis: The problem of capitalization on chance. Psychological Bulletin, 111(3), 490–504.
Maher, P. (1988). Prediction, accommodation and the logic of discovery. PSA: Proceedings of the Biennial Meeting of the Philosophy of Science Association, 1, 273–285.
Makel, M. C., Plucker, J. A., & Hegarty, B. (2012). Replications in psychology research. Perspectives on Psychological Science, 7(6), 537–542.
Maslach, C., & Jackson, S. E. (1981). The measurement of experienced burnout. Journal of Occupational Behavior, 2(2), 99–113.
Mellenbergh, G. J. (1994). Generalized linear item response theory. Psychological Bulletin, 115(2), 300–307.
Michell, J. (1997). Quantitative science and the definition of measurement in psychology. British Journal of Psychology, 88(3), 355–383.
Michell, J. (2003). The quantitative imperative positivism, naive realism and the place of qualitative methods in psychology. Theory & Psychology, 13(1), 5–31.
Michell, J. (2004). The place of qualitative research in psychology. Qualitative Research in Psychology, 1(4), 307–319.
Mulaik, S. A. (1991). Factor analysis, information-transforming instruments, and objectivity: A reply and discussion. The British Journal for the Philosophy of Science, 42(1), 87–100.
Murray-Watters, A., & Glymour, C. (2015). What is going on inside the arrows? Discovering the hidden springs in causal models. Philosophy of Science, 82(4), 556–586.
Musgrave, A. (1974). Logical versus historical theories of confirmation. British Journal for the Philosophy of Science, 25(1), 1–23.
Myrvold, W. C., & Harper, W. L. (2002). Model selection, simplicity, and scientific inference. Philosophy of Science, 69(S3), S135–S149.
Norton, J. D. (2015). Replicability of experiment. Theoria, 30(2), 229–248.
O’Connor, B. (2000). SPSS and SAS programs for determining the number of components using parallel analysis and Velicer’s MAP test. Behavioral Research Methods, Intruments and Computers, 32(3), 396–402.
Park, H. S., Dailey, R., & Lemus, D. (2002). The use of exploratory factor analysis and principal component analysis in communication research. Human Communication Research, 28(4), 562–577.
Preacher, K. J., Zhang, G., Kim, C., & Mels, G. (2013). Choosing the optimal number of factors in exploratory factor analysis: A model selection perspective. Multivariate Behavioral Research, 48(1), 28–56.
Schreiber, J. B., Nora, A., Stage, F. K., Barlow, E. A., & King, J. (2006). Reporting structural equation modeling and confirmatory factor analysis results: A review. The Journal of Educational Research, 99(6), 323–338.
Schurz, G. (2014). Bayesian pseudo-confirmation, use-novelty, and genuine confirmation. Studies in History and Philosophy of Science, 45(1), 87–96.
Silva, R., Scheines, R., Glymour, C., & Spirtes, P. (2006). Learning the structure of linear latent variable models. The Journal of Machine Learning Research, 7, 191–246.
Sober, E. (2004). Likelihood, model selection, and the Duhem-Quine problem. Journal of Philosophy, 101(5), 221–241.
Spirtes, P., Glymour, C., & Scheines, R. (1991). From probability to causality. Philosophical Studies, 64(1), 1–36.
Spirtes, P., Glymour, C., & Scheines, R. (2000). Causation, Prediction, and Search. Cambridge: MIT Press.
Stevens, S. S. (1946). On the theory of scales of measurement. Science, 103(2684), 677–680.
Stone, M. (1974). Cross-validatory choice and assessment of statistical predictions. Journal of the Royal Statistical Society B, 36(2), 111–147.
Stone, M. (1977). An asymptotic equivalence of choice of model by cross-validation and Akaike’s criterion. Journal of the Royal Statistical Society B, 39(1), 44–47.
Suppes, P. (2007). Statistical concepts in philosophy of science. Synthese, 154(3), 485–496.
Tabachnick, B. G., & Fidell, L. S. (2013). Using multivariate statistics (6th ed.). London: Pearson.
Thompson, B. (1994). The pivotal role of replication in psychological research: Empirically evaluating the replicability of sample results. Journal of Personality, 62(2), 157–176.
Thompson, B. (2004). Exploratory and confirmatory factor analysis: Understanding concepts and applications. Washington, DC: American Psychological Association.
Thurstone, L. L. (1947). Multiple factor analysis. Chicago: University of Chicago Press.
Thurstone, L. L. (1954). An analytical method for simple structure. Psychometrika, 19(3), 173–182.
Ullman, J. B. (2013). Structural equation modeling. In B. G. Tabachnick & L. S. Fidell (Eds.), Using multivariate statistics (6th ed.). London: Pearson.
van Fraassen, B. C. (1980). The scientific image. New York: Oxford University Press.
Velicer, W. F., & Jackson, D. N. (1990a). Component analysis versus common factor analysis: Some further observations. Multivariate Behavioral Research, 25(1), 97–114.
Velicer, W. F., & Jackson, D. N. (1990b). Component analysis versus common factor analysis: Some issues in selecting an appropriate procedure. Multivariate Behavioral Research, 25(1), 1–28.
White, R. (2003). The epistemic advantage of prediction over accommodation. Mind, 112(448), 653–683.
Worrall, J. (2002). New evidence for old. In P. Gärdenfors, J. Wolenski, & K. Kijana-Placek (Eds.), In the scope of logic, methodology and philosophy of science (pp. 191–209). Dordrecht: Kluwer Academic Publishers.
Acknowledgements
I would like to thank Stephan Hartmann for valuable comments and suggestions made on a previous draft of this paper. I am also grateful to anonymous referees, whose comments and suggestions helped to improve this article, and to Sarah-Geneviève Trépanier, for enlightening discussions on the subject. This research was financially supported by the Social Sciences and Humanities Research Council of Canada.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Peterson, C. Accommodation, prediction and replication: model selection in scale construction. Synthese 196, 4329–4350 (2019). https://doi.org/10.1007/s11229-017-1660-0
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11229-017-1660-0