Skip to main content
SpringerLink
Log in

Adjusting Person Fit Index for Skewness in Cognitive Diagnosis Modeling

  • Published:
Journal of Classification Aims and scope Submit manuscript

Abstract

Because the validity of diagnostic information generated by cognitive diagnosis models (CDMs) depends on the appropriateness of the estimated attribute profiles, it is imperative to ensure the accurate measurement of students’ test performance by conducting person fit (PF) evaluation to avoid flawed remediation measures. The standardized log-likelihood statistic lZ has been extended to the CDM framework. However, its null distribution is found to be negatively skewed. To address this issue, this study applies different methods of adjusting the skewness of lZ that have been proposed in the item response theory context, namely, χ2-approximation, Cornish-Fisher expansion, and Edgeworth expansion to bring its null distribution closer to the standard normal distribution. The skewness-corrected PF statistics are investigated by calculating their type I error and detection rates using a simulation study. Fraction-subtraction data are also used to illustrate the application of these PF statistics.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Bedrick, E. J. (1997). Approximating the conditional distribution of person fit indexes for checking the Rasch model. Psychometrika, 62, 191–199.

    Article  Google Scholar 

  • Conjin, J. M. (2013). Detecting and explaining person misfit in non-cognitive measurement. Ridderkerk: Ridderprint.

    Google Scholar 

  • Cui, Y., & Leighton, J. O. (2009). The hierarchy consistency index: evaluating person fit for cognitive diagnostic assessment. Journal of Educational Measurement, 46, 429–449.

    Article  Google Scholar 

  • Cui, Y., & Li, J. (2015). Evaluating person fit for cognitive diagnostic assessment. Applied Psychological Measurement, 39, 223–238.

    Article  Google Scholar 

  • Cui, Y., & Roberts, M. R. (2013). Validating student score inferences with person-fit statistic and verbal reports: a person-fit study for cognitive diagnostic assessment. Educational Measurement: Issues and Practice, 32, 34–42.

    Article  Google Scholar 

  • Cui, Y., Gierl, M. J., & Chang, H. H. (2012). Estimating classification consistency and accuracy for cognitive diagnostic assessment. Journal of Educational Measurement, 49, 19–38.

    Article  Google Scholar 

  • de la Torre, J. (2009). DINA model and parameter estimation: a didactic. Journal of Educational and Behavioral Statistics, 34, 115–130.

    Article  Google Scholar 

  • de la Torre, J. (2011). The generalized DINA model framework. Psychometrika, 76, 179–199.

    Article  MathSciNet  Google Scholar 

  • de la Torre, J., & Deng, W. (2008). Improving person fit assessment by correcting the ability estimate and its reference distribution. Journal of Educational Measurement, 45, 159–177.

    Article  Google Scholar 

  • de la Torre, J., & Minchen, N. (2014). Cognitively diagnostic assessments and the cognitive diagnosis model framework. Psicologa Educativa, 20, 89–97.

    Article  Google Scholar 

  • Drasgow, F., Levine, M. V., & Williams, E. A. (1985). Appropriateness measurement with polychotomous item response models and standardized indices. British Journal of Mathematical and Statistical Psychology, 38, 67–86.

    Article  Google Scholar 

  • Glas, C. A. W., & Dagohoy, A. V. T. (2007). A person fit test for IRT models for polytomous items. Psychometrika, 72, 159–180.

    Article  MathSciNet  Google Scholar 

  • Haertel, E. (1989). Using restricted latent class models to map the skill structure of achievement items. Journal of Educational Measurement, 26, 301–321.

    Article  Google Scholar 

  • Hartz, S. M. (2002). A Bayesian framework for the unified model for assessing cognitive abilities: blending theory with practicality, unpublished doctoral dissertation, University of Illinois at Urbana–Champaign, Urbana–Champaign, IL.

  • Henson, R. A., Templin, J. L., & Willse, J. T. (2009). Defining a family of cognitive diagnosis models using log-linear models with latent variables. Psychometrika, 74, 191–210.

    Article  MathSciNet  Google Scholar 

  • Hulin, C. L., Drasgow, F., & Parsons, C. K. (1983). Item response theory: application to psychological measurement. Homewood: Dow Jones-Irwin.

    Google Scholar 

  • Junker, B. W., & Sijtsma, K. (2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25, 258–272.

    Article  MathSciNet  Google Scholar 

  • Levine, M. V., & Drasgow, F. (1983). Appropriateness measurement: validating studies and variable ability models. In D. J. Weiss (Ed.), New horizons in testing: latent trait test theory and computerized adaptive testing (pp. 109–131). New York: Academic Press.

    Google Scholar 

  • Levine, M. V., & Rubin, D. B. (1979). Measuring the appropriateness of multiple choice test scores. Journal of Educational Statistics, 4, 269–290.

    Article  Google Scholar 

  • Liu, Y., Douglas, J., & Henson, R. (2009). Testing person fit in cognitive diagnosis. Applied Psychological Measurement, 33, 579–598.

    Article  Google Scholar 

  • Ma, W., & de la Torre, J (2017). GDINA: the generalized DINA model framework. R package version 1.4.2, Available at https://CRAN.R-project.org/package=GDINA.

  • Ma, W., Iaconangelo, C., & de la Torre, J. (2016). Model similarity, model selection, and attribute classification. Applied Psychological Measurement, 40, 200–217.

    Article  Google Scholar 

  • Mccullagh, E. (1986). The conditional distribution of goodness-of-fit statistics for discrete data. Journal of the American Statistical Association, 81, 104–107.

    Article  Google Scholar 

  • Meijer, R., & Nering, M. L. (1997). Trait level estimation for nonfitting response vectors. Applied Psychological Measurement, 21, 321–336.

    Article  Google Scholar 

  • Molenaar, I. W., & Hoijtink, H. (1990). The many null distributions of person fit indices. Psychometrika, 55, 75–106.

    Article  MathSciNet  Google Scholar 

  • Nering, M. L. (1995). The distribution of person fit using true and estimated person parameters. Applied Psychological Measurement, 19, 121–129.

    Article  Google Scholar 

  • Nering, M. L. (1997). The distribution of indexes of person-fit within the computerized adaptive testing environment. Applied Psychological Measurement, 21, 115–127.

    Article  Google Scholar 

  • Reise, S. P. (1995). Scoring method and the detection of person misfit in a personality assessment context. Applied Psychological Measurement, 19, 213–229.

    Article  Google Scholar 

  • Rupp, A. A., & Templin, J. (2008). Unique characteristics of diagnostic classification models: a comprehensive review of the current state-of-the-art. Measurement: Interdisciplinary Research and Perspectives, 6, 219–262.

    Google Scholar 

  • Sinharay, S. (2015). Assessing person fit using lZ* and the posterior predictive model checking method for dichotomous item response theory models. International Journal of Quantitative Research in Education, 2, 265–284.

    Article  Google Scholar 

  • Sinharay, S. (2016). Assessment of person fit using resampling-based approaches. Journal of Educational Measurement, 53, 63–85.

    Article  Google Scholar 

  • Snijders, T. (2001). Asymptotic distribution of person-fit statistics with estimated person parameter. Psychometrika, 66, 331–342.

    Article  MathSciNet  Google Scholar 

  • Tatsuoka, K. K. (1983). Rule space: an approach for dealing with misconceptions based on item response theory. Journal of Educational Measurement, 20, 345–354.

    Article  Google Scholar 

  • Tatsuoka, K. K. (1984). Caution indices based on item response theory. Psychometrika, 49, 95–110.

    Article  Google Scholar 

  • Tatsuoka, K. K. (1990). Toward an integration of item-response theory and cognitive error diagnosis. In N. Frederiksen, R. Glaser, A. Lesgold, & M. Safto (Eds.), Monitoring skills and knowledge acquisition (pp. 453–488). Hillsdale: Erlbaum.

    Google Scholar 

  • Templin, J. L., & Henson, R. A. (2006). Measurement of psychological disorders using cognitive diagnosis models. Psychological Methods, 11, 287–305.

    Article  Google Scholar 

  • Von Davier, M. (2008). A general diagnostic model applied to language testing data. British Journal of Mathematical and Statistical Psychology, 61, 287–307.

    Article  MathSciNet  Google Scholar 

  • Von Davier, M., & Molenaar, I. (2003). A person-fit index for polytomous rasch models, latent class models, and their mixture generalizations. Psychometrika, 68, 213–228.

    Article  MathSciNet  Google Scholar 

  • Wright, B. D. (1977). Solving measurement problems with the Rasch model. Journal of Educational Measurement, 14, 97–115.

    Article  Google Scholar 

Download references

Funding

This research was funded by the Philippine Commission on Higher Education, Philippine Social Science Council, and University of the Philippines-Diliman. Moreover, this research was carried out in part using the CoARE Facility of the DOST-Advance Science and Technology Institute and the Computing and Archiving Research Environment (CoARE) Project.

Author information

Authors and Affiliations

  1. School of Statistics, University of the Philippines-Diliman, 1101, Quezon City, Philippines

    Kevin Carl P. Santos

  2. The University of Hong Kong, , Pok Fu Lam, Hong Kong

    Jimmy de la Torre

  3. National Board of Medical Examiners, Philadelphia, PA, USA

    Matthias von Davier

Authors
  1. Kevin Carl P. Santos
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jimmy de la Torre
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Matthias von Davier
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kevin Carl P. Santos.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Santos, K.C.P., de la Torre, J. & von Davier, M. Adjusting Person Fit Index for Skewness in Cognitive Diagnosis Modeling. J Classif 37, 399–420 (2020). https://doi.org/10.1007/s00357-019-09325-5

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00357-019-09325-5

Keywords

Search

Navigation