Skip to main content

Advertisement

Log in

How to Build a Complete Q-Matrix for a Cognitively Diagnostic Test

  • Published:
Journal of Classification Aims and scope Submit manuscript

Abstract

The Q-matrix of a cognitively diagnostic test is said to be complete if it guarantees the identifiability of all possible proficiency classes among examinees. An incomplete Q-matrix causes examinees to be assigned to proficiency classes to which they do not belong. Completeness of the Q-matrix is therefore a key requirement of any cognitively diagnostic test. The importance of the completeness property of the Q-matrix of a test as a fundamental condition to guarantee a reliable estimate of an examinee’s attribute profile has only recently been realized by researchers. In fact, inspection of extant assessments based on the cognitive diagnosis framework often revealed that, in hindsight, the Q-matrices used with these tests were not complete. Thus, the availability of rules for building a complete Q-matrix at the early stages of test development is perhaps at least as desirable as rules for identifying the completeness of a given Q-matrix. This article presents procedures for constructing Q-matrices that are complete. The famous Fraction-Subtraction test problems by K. K. Tatsuoka (1984) are used throughout for illustration.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • CHEN, J., DE LA TORRE, J., and ZHANG, Z. (2013), “Relative and Absolute Fit Evaluation in Cognitive Diagnosis Modeling”, Journal of Educational Measurement, 50, 123–140.

    Article  Google Scholar 

  • CHIU, C.-Y. (2013), “Statistical Refinement of the Q-matrix in Cognitive Diagnosis”, Applied Psychological Measurement, 37, 598–618.

    Article  Google Scholar 

  • CHIU, C.-Y., and DOUGLAS, J.A. (2013), “A Nonparametric Approach to Cognitive Diagnosis by Proximity to Ideal Response Profiles”, Journal of Classification, 30, 225–250.

    Article  MathSciNet  MATH  Google Scholar 

  • CHIU, C.-Y., and KO¨ HN, H.-F. (2015), “Consistency of Cluster Analysis for Cognitive Diagnosis: The DINO Model and the DINA Model Revisited”, Applied Psychological Measurement, 39, 465-479.

    Article  Google Scholar 

  • CHIU, C.-Y., DOUGLAS, J.A., and LI, X. (2009), “Cluster Analysis for Cognitive Diagnosis: Theory and Applications”, Psychometrika, 74, 633–665.

    Article  MathSciNet  MATH  Google Scholar 

  • DE LA TORRE, J. (2008), “An Empirically Based Method of Q-matrix Validation for the DINA Model: Development and Applications”, Journal of Educational Measurement, 45, 343–362.

    Article  Google Scholar 

  • DE LA TORRE, J. (2009), “DINA Model and Parameter Estimation: A Didactic”, Journal of Educational and Behavioral Statistics, 34, 115–130.

    Article  Google Scholar 

  • DE LA TORRE, J. (2011), “The Generalized DINA Model Framework”, Psychometrika, 76, 179–199.

    Article  MathSciNet  MATH  Google Scholar 

  • DE LA TORRE, J. (2014), Personal Communication.

  • DE LA TORRE, J., and DOUGLAS, J.A. (2004), “Higher-Order Latent Trait Models for Cognitive Diagnosis”, Psychometrika, 69, 333–353.

    Article  MathSciNet  MATH  Google Scholar 

  • DE LA TORRE, J., and DOUGLAS, J.A. (2008), “Model Evaluation and Multiple Strategies in Cognitive Diagnosis: An Analysis of Fraction Subtraction Data”, Psychometrika, 73, 595–624.

    Article  MathSciNet  MATH  Google Scholar 

  • DE LA TORRE, J., and LEE, Y.-S. (2013), “Evaluating the Wald Test for Item-Level Comparison of Saturated and Reduced Models in Cognitive Diagnosis”, Journal of Educational Measurement, 50, 355–373.

    Article  Google Scholar 

  • DECARLO, L.T. (2011), “On the Analysis of Fraction Subtraction Data: The DINA Model, Classification, Latent Class Sizes, and the Q-matrix”, Applied Psychological Measurement, 35, 8–26.

    Article  Google Scholar 

  • DECARLO, L.T. (2012), “Recognizing Uncertainty in the Q-Matrix via a Bayesian Extension of the DINA Model”, Applied Psychological Measurement, 36, 447–468.

    Article  Google Scholar 

  • DIBELLO, L.V., ROUSSOS, L.A., and STOUT, W.F. (2007), “Review of Cognitively Diagnostic Assessment and a Summary of Psychometric Models”, in Handbook of Statistics: Vol. 26. Psychometrics, eds. C.R. Rao and S. Sinharay, Amsterdam: Elsevier, pp. 979–1030.

  • FU, J., and LI, Y. (2007), “An Integrative Review of Cognitively Diagnostic Psychometric Models”, paper presented at the Annual Meeting of the National Council on Measurement in Education, Chicago, IL.

  • HABERMAN, S.J., and VON DAVIER, M. (2007), “Some Notes on Models for Cognitively Based Skill Diagnosis”, in Handbook of Statistics: Vol. 26. Psychometrics, eds. C.R. Rao and S. Sinharay, Amsterdam: Elsevier, pp. 1031–1038.

  • HAERTEL, E.H. (1989), “Using Restricted Latent Class Models to Map the Skill Structure of Achievement Items”, Journal of Educational Measurement, 26, 333–352.

    Article  Google Scholar 

  • HARTZ, S.M. (2002), “A Bayesian Framework for the Unified Model for Assessing Cognitive Abilities: Blending Theory with Practicality”, Doctoral Dissertation, available from ProQuest Dissertations and Theses database, UMI No. 3044108.

  • HARTZ, S.M., and ROUSSOS, L.A. (2008), “The Fusion Model for Skill Diagnosis: Blending Theory with Practicality”, Research Report No. RR-08-71, Princeton, NJ: Educational Testing Service.

  • HENSON, R.A., TEMPLIN, J.L., and WILLSE, J.T. (2009), “Defining a Family of Cognitive Diagnosis Models Using Log-Linear Models with Latent Variables”, Psychometrika, 74, 191–210.

    Article  MathSciNet  MATH  Google Scholar 

  • JUNKER, B.W., and SIJTSMA, K. (2001), “Cognitive Assessment Models with Few Assumptions, and Connections with Nonparametric Item Response Theory”, Applied Psychological Measurement, 25, 258–272.

    Article  MathSciNet  Google Scholar 

  • KÖ HN, H.-F., and CHIU, C.-Y. (2017), “A Procedure for Assessing the Completeness of the Q-Matrices of Cognitively Diagnostic Tests”, Psychometrika, 82, 112–132.

    Article  MathSciNet  MATH  Google Scholar 

  • KÖ HN,H.-F., CHIU, C.-Y., and BRUSCO,M.J. (2015), “Heuristic Cognitive Diagnosis when the Q-Matrix is Unknown”, British Journal of Mathematical and Statistical Psychology, 68, 268–291.

    Article  Google Scholar 

  • LEIGHTON, J., and GIERL, M. (2007), Cognitive Diagnostic Assessment for Education: Theory and Applications, Cambridge, UK: Cambridge University Press.

    Book  Google Scholar 

  • LEON, S.J. (1998), Linear Algebra with Applications (5th ed.), Upper Saddle River, NJ: Prentice Hall.

  • LIU, J., XU, G., and YING, Z. (2012), “Data-Driven Learning of Q-Matrix”, Applied Psychological Measurement, 36, 548–564.

    Article  Google Scholar 

  • LIU, J., XU, G., and YING, Z. (2013), “Theory of the Self-Learning Q-Matrix”, Bernoulli, 19, 1790–1817.

    Article  MathSciNet  MATH  Google Scholar 

  • MACREADY, G.B., and DAYTON, C.M. (1977), “The Use of Probabilistic Models in the Assessment of Mastery”, Journal of Educational Statistics, 33, 379–416.

    Google Scholar 

  • MADISON, M.J., and BRADSHAW, L.P. (2015), “The Effects of Q-Matrix Design on Classification Accuracy in the Log-Linear Cognitive Diagnosis Model”, Educational and Psychological Measurement, 75, 491–511.

    Article  Google Scholar 

  • MADISON, M.J., BRADSHAW, L.P., and HOLLINGSWORTH, B. (2014), Q*Power–User Manual (Beta Version, 2014), retrieved from http://www.lainebradshaw.com/qpower/. MARIS, E. (1999), “Estimating Multiple Classification Latent Class Models”, Psychometrika, 64, 187–212.

  • MISLEVY, R.J. (1996), “Test Theory Reconceived”, Journal of Educational Measurement, 33, 379-416.

    Article  Google Scholar 

  • RUPP,A.A., and TEMPLIN, J.L. (2008), “Unique Characteristics of Diagnostic Classification Models: A Comprehensive Review of the Current State-of-the-Art”, Measurement, 6, 219-262.

    Google Scholar 

  • RUPP, A.A., TEMPLIN, J.L., and HENSON, R.A. (2010), Diagnostic Measurement. Theory, Methods, and Applications, New York: Guilford.

  • TATSUOKA, K.K. (1984), “Analysis of Errors in Fraction Addition and Subtraction Problems”, Report NIE-G-81-0002, Computer-based Education Research Library, Urbana, IL: University of Illinois, retrieved from Education Research Information Center [ERIC] website http://www.eric.ed.gov (ED257665). (The fraction addition and subtraction data are also available in the R package CDM by Robitzsch, A., Kiefer, T., George, A. C., and Uenlue, A.(2017), CDM: Cognitive Diagnosis Modeling. R Package Version 5.9-27, downloadable from the Comprehensive R Archive Network [CRAN] website https://cran.r-project.org/web/packages/CDM/.)

  • TATSUOKA, K.K. (1985), “A Probabilistic Model for Diagnosing Misconception in the Pattern Classification Approach”, Journal of Educational and Behavioral Statistics, 12, 55–73.

    Article  Google Scholar 

  • TATSUOKA, K.K. (1990), “Toward an Integration of Item-Response Theory and Cognitive Error Diagnosis”, in Diagnostic Monitoring of Skill and Knowledge Acquisition, eds. N. Frederiksen, R. Glaser, A. Lesgold, and M. Shafto, Hillsdale, NJ: Erlbaum, pp. 453-488.

  • TATSUOKA, K.K. (2009), Cognitive Assessment. An Introduction to the Rule SpaceMethod”, New York: Routledge/Taylor and Francis. TEMPLIN, J., and BRADSHAW, L. (2014), “Hierarchical Diagnostic Classification Models: A Family of Models for Estimating and Testing Attribute Hierarchies”, Psychometrika, 79, 317-339.

  • TEMPLIN, J.L., and HENSON, RA. (2006), “Measurement of Psychological Disorders Using Cognitive Diagnosis Models”, Psychological Methods, 11, 287–305.

    Article  Google Scholar 

  • VON DAVIER, M. (2005), “A General Diagnostic Model Applied to Language Testing Data”, Research Report No. RR-05-16, Princeton, NJ: Educational Testing Service.

  • VON DAVIER, M. (2008), “A General Diagnostic Model Applied to Language Testing Data”, British Journal of Mathematical and Statistical Psychology, 61, 287–301.

    Article  MathSciNet  Google Scholar 

  • VON DAVIER, M. (2014), “The DINA Model as a Constrained General Diagnostic Model: Two Variants of a Model Equivalency”, British Journal of Mathematical and Statistical Psychology, 67, 49–71.

    Article  MathSciNet  Google Scholar 

  • XU, G., and ZHANG, S. (2016), “Identifiability of Diagnostic Classification Models”, Psychometrika, 81(3), 625–649.

    Article  MathSciNet  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Hans-Friedrich Köhn.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Köhn, HF., Chiu, CY. How to Build a Complete Q-Matrix for a Cognitively Diagnostic Test. J Classif 35, 273–299 (2018). https://doi.org/10.1007/s00357-018-9255-0

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00357-018-9255-0

Keywords

Navigation