Abstract
The Q-matrix of a cognitively diagnostic test is said to be complete if it guarantees the identifiability of all possible proficiency classes among examinees. An incomplete Q-matrix causes examinees to be assigned to proficiency classes to which they do not belong. Completeness of the Q-matrix is therefore a key requirement of any cognitively diagnostic test. The importance of the completeness property of the Q-matrix of a test as a fundamental condition to guarantee a reliable estimate of an examinee’s attribute profile has only recently been realized by researchers. In fact, inspection of extant assessments based on the cognitive diagnosis framework often revealed that, in hindsight, the Q-matrices used with these tests were not complete. Thus, the availability of rules for building a complete Q-matrix at the early stages of test development is perhaps at least as desirable as rules for identifying the completeness of a given Q-matrix. This article presents procedures for constructing Q-matrices that are complete. The famous Fraction-Subtraction test problems by K. K. Tatsuoka (1984) are used throughout for illustration.
Similar content being viewed by others
References
CHEN, J., DE LA TORRE, J., and ZHANG, Z. (2013), “Relative and Absolute Fit Evaluation in Cognitive Diagnosis Modeling”, Journal of Educational Measurement, 50, 123–140.
CHIU, C.-Y. (2013), “Statistical Refinement of the Q-matrix in Cognitive Diagnosis”, Applied Psychological Measurement, 37, 598–618.
CHIU, C.-Y., and DOUGLAS, J.A. (2013), “A Nonparametric Approach to Cognitive Diagnosis by Proximity to Ideal Response Profiles”, Journal of Classification, 30, 225–250.
CHIU, C.-Y., and KO¨ HN, H.-F. (2015), “Consistency of Cluster Analysis for Cognitive Diagnosis: The DINO Model and the DINA Model Revisited”, Applied Psychological Measurement, 39, 465-479.
CHIU, C.-Y., DOUGLAS, J.A., and LI, X. (2009), “Cluster Analysis for Cognitive Diagnosis: Theory and Applications”, Psychometrika, 74, 633–665.
DE LA TORRE, J. (2008), “An Empirically Based Method of Q-matrix Validation for the DINA Model: Development and Applications”, Journal of Educational Measurement, 45, 343–362.
DE LA TORRE, J. (2009), “DINA Model and Parameter Estimation: A Didactic”, Journal of Educational and Behavioral Statistics, 34, 115–130.
DE LA TORRE, J. (2011), “The Generalized DINA Model Framework”, Psychometrika, 76, 179–199.
DE LA TORRE, J. (2014), Personal Communication.
DE LA TORRE, J., and DOUGLAS, J.A. (2004), “Higher-Order Latent Trait Models for Cognitive Diagnosis”, Psychometrika, 69, 333–353.
DE LA TORRE, J., and DOUGLAS, J.A. (2008), “Model Evaluation and Multiple Strategies in Cognitive Diagnosis: An Analysis of Fraction Subtraction Data”, Psychometrika, 73, 595–624.
DE LA TORRE, J., and LEE, Y.-S. (2013), “Evaluating the Wald Test for Item-Level Comparison of Saturated and Reduced Models in Cognitive Diagnosis”, Journal of Educational Measurement, 50, 355–373.
DECARLO, L.T. (2011), “On the Analysis of Fraction Subtraction Data: The DINA Model, Classification, Latent Class Sizes, and the Q-matrix”, Applied Psychological Measurement, 35, 8–26.
DECARLO, L.T. (2012), “Recognizing Uncertainty in the Q-Matrix via a Bayesian Extension of the DINA Model”, Applied Psychological Measurement, 36, 447–468.
DIBELLO, L.V., ROUSSOS, L.A., and STOUT, W.F. (2007), “Review of Cognitively Diagnostic Assessment and a Summary of Psychometric Models”, in Handbook of Statistics: Vol. 26. Psychometrics, eds. C.R. Rao and S. Sinharay, Amsterdam: Elsevier, pp. 979–1030.
FU, J., and LI, Y. (2007), “An Integrative Review of Cognitively Diagnostic Psychometric Models”, paper presented at the Annual Meeting of the National Council on Measurement in Education, Chicago, IL.
HABERMAN, S.J., and VON DAVIER, M. (2007), “Some Notes on Models for Cognitively Based Skill Diagnosis”, in Handbook of Statistics: Vol. 26. Psychometrics, eds. C.R. Rao and S. Sinharay, Amsterdam: Elsevier, pp. 1031–1038.
HAERTEL, E.H. (1989), “Using Restricted Latent Class Models to Map the Skill Structure of Achievement Items”, Journal of Educational Measurement, 26, 333–352.
HARTZ, S.M. (2002), “A Bayesian Framework for the Unified Model for Assessing Cognitive Abilities: Blending Theory with Practicality”, Doctoral Dissertation, available from ProQuest Dissertations and Theses database, UMI No. 3044108.
HARTZ, S.M., and ROUSSOS, L.A. (2008), “The Fusion Model for Skill Diagnosis: Blending Theory with Practicality”, Research Report No. RR-08-71, Princeton, NJ: Educational Testing Service.
HENSON, R.A., TEMPLIN, J.L., and WILLSE, J.T. (2009), “Defining a Family of Cognitive Diagnosis Models Using Log-Linear Models with Latent Variables”, Psychometrika, 74, 191–210.
JUNKER, B.W., and SIJTSMA, K. (2001), “Cognitive Assessment Models with Few Assumptions, and Connections with Nonparametric Item Response Theory”, Applied Psychological Measurement, 25, 258–272.
KÖ HN, H.-F., and CHIU, C.-Y. (2017), “A Procedure for Assessing the Completeness of the Q-Matrices of Cognitively Diagnostic Tests”, Psychometrika, 82, 112–132.
KÖ HN,H.-F., CHIU, C.-Y., and BRUSCO,M.J. (2015), “Heuristic Cognitive Diagnosis when the Q-Matrix is Unknown”, British Journal of Mathematical and Statistical Psychology, 68, 268–291.
LEIGHTON, J., and GIERL, M. (2007), Cognitive Diagnostic Assessment for Education: Theory and Applications, Cambridge, UK: Cambridge University Press.
LEON, S.J. (1998), Linear Algebra with Applications (5th ed.), Upper Saddle River, NJ: Prentice Hall.
LIU, J., XU, G., and YING, Z. (2012), “Data-Driven Learning of Q-Matrix”, Applied Psychological Measurement, 36, 548–564.
LIU, J., XU, G., and YING, Z. (2013), “Theory of the Self-Learning Q-Matrix”, Bernoulli, 19, 1790–1817.
MACREADY, G.B., and DAYTON, C.M. (1977), “The Use of Probabilistic Models in the Assessment of Mastery”, Journal of Educational Statistics, 33, 379–416.
MADISON, M.J., and BRADSHAW, L.P. (2015), “The Effects of Q-Matrix Design on Classification Accuracy in the Log-Linear Cognitive Diagnosis Model”, Educational and Psychological Measurement, 75, 491–511.
MADISON, M.J., BRADSHAW, L.P., and HOLLINGSWORTH, B. (2014), Q*Power–User Manual (Beta Version, 2014), retrieved from http://www.lainebradshaw.com/qpower/. MARIS, E. (1999), “Estimating Multiple Classification Latent Class Models”, Psychometrika, 64, 187–212.
MISLEVY, R.J. (1996), “Test Theory Reconceived”, Journal of Educational Measurement, 33, 379-416.
RUPP,A.A., and TEMPLIN, J.L. (2008), “Unique Characteristics of Diagnostic Classification Models: A Comprehensive Review of the Current State-of-the-Art”, Measurement, 6, 219-262.
RUPP, A.A., TEMPLIN, J.L., and HENSON, R.A. (2010), Diagnostic Measurement. Theory, Methods, and Applications, New York: Guilford.
TATSUOKA, K.K. (1984), “Analysis of Errors in Fraction Addition and Subtraction Problems”, Report NIE-G-81-0002, Computer-based Education Research Library, Urbana, IL: University of Illinois, retrieved from Education Research Information Center [ERIC] website http://www.eric.ed.gov (ED257665). (The fraction addition and subtraction data are also available in the R package CDM by Robitzsch, A., Kiefer, T., George, A. C., and Uenlue, A.(2017), CDM: Cognitive Diagnosis Modeling. R Package Version 5.9-27, downloadable from the Comprehensive R Archive Network [CRAN] website https://cran.r-project.org/web/packages/CDM/.)
TATSUOKA, K.K. (1985), “A Probabilistic Model for Diagnosing Misconception in the Pattern Classification Approach”, Journal of Educational and Behavioral Statistics, 12, 55–73.
TATSUOKA, K.K. (1990), “Toward an Integration of Item-Response Theory and Cognitive Error Diagnosis”, in Diagnostic Monitoring of Skill and Knowledge Acquisition, eds. N. Frederiksen, R. Glaser, A. Lesgold, and M. Shafto, Hillsdale, NJ: Erlbaum, pp. 453-488.
TATSUOKA, K.K. (2009), Cognitive Assessment. An Introduction to the Rule SpaceMethod”, New York: Routledge/Taylor and Francis. TEMPLIN, J., and BRADSHAW, L. (2014), “Hierarchical Diagnostic Classification Models: A Family of Models for Estimating and Testing Attribute Hierarchies”, Psychometrika, 79, 317-339.
TEMPLIN, J.L., and HENSON, RA. (2006), “Measurement of Psychological Disorders Using Cognitive Diagnosis Models”, Psychological Methods, 11, 287–305.
VON DAVIER, M. (2005), “A General Diagnostic Model Applied to Language Testing Data”, Research Report No. RR-05-16, Princeton, NJ: Educational Testing Service.
VON DAVIER, M. (2008), “A General Diagnostic Model Applied to Language Testing Data”, British Journal of Mathematical and Statistical Psychology, 61, 287–301.
VON DAVIER, M. (2014), “The DINA Model as a Constrained General Diagnostic Model: Two Variants of a Model Equivalency”, British Journal of Mathematical and Statistical Psychology, 67, 49–71.
XU, G., and ZHANG, S. (2016), “Identifiability of Diagnostic Classification Models”, Psychometrika, 81(3), 625–649.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Köhn, HF., Chiu, CY. How to Build a Complete Q-Matrix for a Cognitively Diagnostic Test. J Classif 35, 273–299 (2018). https://doi.org/10.1007/s00357-018-9255-0
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00357-018-9255-0