Skip to main content

Advertisement

Log in

Incorporating Student Covariates in Cognitive Diagnosis Models

Journal of Classification Aims and scope Submit manuscript

Abstract

In educational measurement, cognitive diagnosis models have been developed to allow assessment of specific skills that are needed to perform tasks. Skill knowledge is characterized as present or absent and represented by a vector of binary indicators, or the skill set profile. After determining which skills are needed for each assessment item, a model is specified for the relationship between item responses and skill set profiles. Cognitive diagnosis models are often used for diagnosis, that is, for classifying students into the different skill set profiles. Generally, cognitive diagnosis models do not exploit student covariate information. However, investigating the effects of student covariates, such as gender, SES, or educational interventions, on skill knowledge mastery is important in education research, and covariate information may improve classification of students to skill set profiles. We extend a common cognitive diagnosis model, the DINA model, by modeling the relationship between the latent skill knowledge indicators and covariates. The probability of skill mastery is modeled as a logistic regression model, possibly with a student-level random intercept, giving a higher-order DINA model with a latent regression. Simulations show that parameter recovery is good for these models and that inclusion of covariates can improve skill diagnosis. When applying our methods to data from an online tutor, we obtain reasonable and interpretable parameter estimates that allow more detailed characterization of groups of students who differ in their predicted skill set profiles.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • AYERS, E., and JUNKER, B.W. (2008), “IRT Modeling of Tutor Performance to Predict End-of-year Exam Scores”, Educational and Psychological Measurement, 68, 972–987.

    Article  MathSciNet  Google Scholar 

  • AYERS, E., NUGENT, R., and DEAN, N. (2009), “A Comparison of Student Skill Knowledge Estimates”, in Educational Data Mining 2009: 2nd International Conference on Educational Data Mining, Proceedings, eds. T. Barnes, M. Desmarais, C. Romero, and S.Ventura, Cordoba, Spain, pp.1-10, http://www.educationaldatamining.org/EDM2009/uploads/proceedings/ayers.pdf.

  • BARNES, T.M. (2003), “The Q-matrix Method of Fault-Tolerant Teaching in Knowledge Assessment and Data Mining”, Ph.D. Thesis, Department of Computer Science, North Carolina State University, NC.

  • BOZARD, J. (2010), “Invariance Testing in Diagnostic Classification Models”, Masters Thesis, The University of Georgia, Athens, GA.

  • CHIU, C. (2008), “Cluster Analysis for Cognitive Diagnosis: Theory and Applications”, Ph.D. Thesis, Educational Psychology, University of Illinois, Urbana Champaign, IL.

  • CHIU, C., DOUGLAS, J., and LI, X. (2009), “Cluster Analysis for Cognitive Diagnosis: Theory and Applications”, Psychometrika, 74, 633–665.

    Article  MathSciNet  MATH  Google Scholar 

  • CHO, S-J., and COHEN, A.S. (2010), “A Multilevel Mixture IRT Model with an Application to DIF”, Journal of Educational and Behavioral Statistics, 35, 336–370.

    Article  Google Scholar 

  • CLOGG, C.C., and GOODMAN, L.A. (1984), “Latent Structure Analysis of a Set of Multidimensional Contingency Tables”, Journal of the American Statistical Association, 79, 762–771.

    Article  MathSciNet  MATH  Google Scholar 

  • COHEN, J. (1960), “A Coefficient of Agreement for Nominal Scales”, Educational and Psychological Measurement, 20, 37–46.

    Article  Google Scholar 

  • DAYTON, C.M., andMACREADY, G.B. (1988), “Concomitant Variable Latent ClassModels”, Journal American Statistical Association, 83, 173–178.

    Article  MathSciNet  Google Scholar 

  • DE LA TORRE, J., and DOUGLAS, J. (2004), “Higher-order Latent Trait Models for Cognitive Diagnosis”, Psychometrika, 69, 333–353.

    Article  MathSciNet  Google Scholar 

  • DE LA TORRE, J. (2009), “DINA Model and Parameter Estimation: A Didactic”, Journal of Educational and Behavioral Statistics, 34, 115–130.

    Article  Google Scholar 

  • DE LA TORRE, J., and CHIU, C.Y. (2009), “A Generalized Index of Item Discrimination for Cognitive Diagnosis Models”, paper presented at the International Meeting of the Psychometric Society, Cambridge, England.

  • EMBRETSON, S.E. (1984), “A General Latent Trait Model for Response Processes”, Psychometrika, 49, 175–186.

    Article  Google Scholar 

  • FORMANN, A.K. (1992), “Linear Logistic Latent Class Analysis for Polytomous Data”, Journal of the American Statistical Association, 87, 476–486.

    Article  Google Scholar 

  • HAERTEL, E. H. (1989), “Using Restricted Latent ClassModels to Map the Skill Structure of Achievement Items”, Journal of Educational Measurement, 26, 333–352.

    Article  Google Scholar 

  • HARTZ, S. (2002), “A Bayesian Framework for the Unified Model for Assessing Cognitive Abilities: Blending Theory with Practicality”, Ph.D Thesis, University of Illinois, Urbana-Champaign, IL.

  • HEFFERNAN, N.T., KOEDINGER, K.R., and JUNKER, B.W. (2001), “Using Web-Based Cognitive Assessment Systems for Predicting Student Performance on State Exams”, research proposal to the Institute of Educational Statistics, US Department of Education; Department of Computer Science at Worcester Polytechnic Institute, Worcester County, MA.

  • HUBERT, L., and ARABIE, P. (1985), “Comparing Partitions”, Journal of Classification, 2, 193–218.

    Article  Google Scholar 

  • JÖRESKOG, K.G. (1971), “Simultaneous Factor Analysis in Several Populations”, Psychometrika, 36, 409–426.

    Article  MATH  Google Scholar 

  • JÖRESKOG, K.G., and GOLDBERGER, A.S. (1975), “Estimation of a Model with Multiple Indicators and Multiple Causes of a Single Latent Variable”, Journal of the American Statistical Association, 70, 631–639.

    MathSciNet  MATH  Google Scholar 

  • JUNKER, B.W., and SIJTSMA, K. (2001), “Cognitive Assessment Models with Few Assumptions, and Connections with Nonparametric Item Response Theory”, Applied Psychological Measurement, 25, 258–272.

    Article  MathSciNet  Google Scholar 

  • LARSEN, K., PETERSEN, J. H., BUDTZ-JØRGENSEN, E., and ENDAHL, L. (2000), “Interpreting Parameters in the Logistic Regression Model with Random Effects”, Biometrics, 56, 909–914.

    Article  MATH  Google Scholar 

  • LI, F., and COHEN, A.S. (2006), “A Higher-Order DINA Rasch Model For Detection of Differential Item Functioning”, paper presented at the annual meeting of the Pacific Rim Objective Measurement Symposium, Hong Kong, People’s Republic of China.

  • MACREADY, G.B., and DAYTON, C.M. (1977), “The Use of Probabilistic Models in the Assessment of Mastery”, Journal of Educational Statistics, 2, 99–120.

    Article  Google Scholar 

  • MAGIDSON, J., and VERMUNT, J.K. (2001), “Latent Class Factor and Cluster Models, Bi-plots and Related Graphical Displays”, Sociological Methodology, 31, 223–264.

    Article  Google Scholar 

  • MARIS, E. (1999), “Estimating Multiple Classification Latent Class Models”, Psychometrika, 64, 187–212.

    Article  MathSciNet  Google Scholar 

  • MEREDITH, W. (1993), “Measurement Invariance, Factor Analysis and Factorial Invariance”, Psychometrika, 58, 525–543.

    Article  MathSciNet  MATH  Google Scholar 

  • MISLEVY, R.J. (1985), “Estimation of Latent Group Effects”, Journal of the American Statistical Association, 80, 993–997.

    Article  MathSciNet  Google Scholar 

  • MISLEVY, R.J. (1987), “Exploiting Auxiliary Information about Examinees in the Estimation of Item Parameters”, Applied Psychological Measurement, 11, 81–91.

    Article  Google Scholar 

  • MISLEVY, R.J., JOHNSON, E.G., and MURAKI, E. (1992), “Scaling Procedures in NAEP”, Journal of Educational Statistics, 17, 131–154.

    Article  Google Scholar 

  • MILLSAP, R. E., and EVERSON, H. T. (1993), “Methodology Review: Statistical Approaches for Assessing Measurement Bias”, Applied Psychological Measurement, 17, 297–334.

    Article  Google Scholar 

  • MUTHÉN, B., and LEHMAN, J. (1985), “Multiple Group IRT Modeling: Applications to Item Bias Analysis”, Journal of Educational Statistics, 10, 133–142.

    Article  Google Scholar 

  • MUTHÉN, L. K., and MUTHÉN, B. O. (2010), Mplus User’s Guide (Sixth Ed.), Los Angeles, CA: Muthén & Muthén.

    Google Scholar 

  • R DEVELOPMENT CORE TEAM. (2004), “R: A Language and Environment for Statistical Computing”, R Foundation for Statistical Computing, Vienna, Austria, ISBN 3-900051-07-0, http://www.R-project.org.

  • RUBIN, D.B. (1987), Multiple Imputation for Nonresponse in Surveys, New York: Wiley.

    Book  Google Scholar 

  • RUPP, A., and TEMPLIN, J. (2007), “The Effects of Q-Matrix Misspecification on Parameter Estimates and Classification Accuracy in the DINA Model”, Educational and Psychological Measurement, 68 (1), 78–96.

    Article  MathSciNet  Google Scholar 

  • RUPP, A., TEMPLIN, J., and HENSON, R. (2010), Diagnostic Measurement: Theory, Methods, and Applications, New York: The Guildford Press.

    Google Scholar 

  • SELF, J. (1993), “Model-Based Cognitive Diagnosis”, User Modeling and User-Adapted Interaction, 3, 89–106.

    Article  Google Scholar 

  • SMIT, A., KELDERMAN, H., and VAN DER FLIER, H. (1999), “Collateral Information and Mixed Rasch Models”, Methods of Psychological Research Online, 4, 19–32.

    Google Scholar 

  • SMIT, A., KELDERMAN, H., and VAN DER FLIER, H. (2000), “The Mixed Birnbaum Model: Estimation using Collateral Information”, Methods of Psychological Research Online, 5, 31–43.

    Google Scholar 

  • SPIEGELHALTER, D.J., THOMAS, A., and BEST, N.G. (2003), WinBUGS: Bayesian Inference Using Gibbs Sampling, Manual Version 1.4, Cambridge: Medical Research Council Biostatistics Unit.

    Google Scholar 

  • STATACORP. (2009), Stata Statistical Software: Release 11, College Station, TX: Stata-Corp LP.

    Google Scholar 

  • TATSUOKA, K.K. (1983), “Rule Space: An Approach for Dealing with Misconceptions Based on Item Response Theory”, Journal of Educational Measurement, 20, 345–354.

    Article  Google Scholar 

  • TEMPLIN, J. (2004), “Generalized Linear Mixed Proficiency Models”, Ph.D. Thesis, University of Illinois, Urbana-Champaign, IL.

  • TEMPLIN, J., HENSON, R., and DOUGLAS, J. (2007), “General Theory and Estimation of Cognitive Diagnosis Models: Using Mplus to DeriveModel Estimates”, Manuscript under Review.

  • TEMPLIN, J.L., HENSON, R.A., TEMPLIN, S.E., and ROUSSOS, L. (2008), “Robustness of Hierarchical Modeling of Skill Association in Cognitive Diagnosis Models, Applied Psychological Measurement, 32, 559–574.

    Article  MathSciNet  Google Scholar 

  • THISSEN, D., STEINBERG, L., and WAINER, H. (1988), “Use of Item Response Theory in the Study of Group Differences in Trace Lines”, in Test Validity, eds. H. Wainer and H. Braun, Hillsdale, NJ: Erlbaum, pp. 147–169.

    Google Scholar 

  • VERMUNT, J.K. (2003), “Multilevel Latent Class Models”, Sociological Methodology, 33, 213–239.

    Article  Google Scholar 

  • VON DAVIER, M. (2010), “Hierarchical Mixtures of Diagnostic Models”, Psychological Test and Assessment Modeling, 52, 8–28.

    Google Scholar 

  • XU, X., and VON DAVIER, M. (2008), “Fitting the Structural Diagnostic Model to NAEP Data”, Research Report RR-08-27, Princeton, NJ: Educational Testing Service.

  • WEDEL, M. (2002), “Concomitant Variables in Finite Mixture Models”, Statistica Neerlandica, 56, 362–375.

    Article  MathSciNet  MATH  Google Scholar 

  • ZHANG, W. (2006), “Detecting Differential Item Functioning Using the DINA Model”, Ph.D. Thesis, The University of North Carolina at Greensboro, Greensboro, NC.

  • ZWINDERMAN, A.H. (1991), “A Generalized RaschModel for Manifest Predictors”, Psychometrika, 56, 589–600.

    Article  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Elizabeth Ayers.

Additional information

The research reported here was supported by the Institute of Education Sciences, U.S. Department of Education, through grant R305A070254 to University of California, Berkeley. The opinions expressed are those of the authors and do not represent the view of the Institute or the U.S. Department of Education.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Ayers, E., Rabe-Hesketh, S. & Nugent, R. Incorporating Student Covariates in Cognitive Diagnosis Models. J Classif 30, 195–224 (2013). https://doi.org/10.1007/s00357-013-9130-y

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00357-013-9130-y

Keywords

Navigation