Skip to main content
SpringerLink
Log in

Effects of Item Calibration Errors on Computerized Adaptive Testing under Cognitive Diagnosis Models

  • Published:
Journal of Classification Aims and scope Submit manuscript

Abstract

In a cognitive diagnostic computerized adaptive testing (CD-CAT) exam, an item pool that consists of items with calibrated item parameters is used for item selection and attribute estimation. The parameter estimates for the items in the item pool are often treated as if they were the true population parameters, and therefore, the calibration errors are ignored. The purpose of this study was to investigate the effects of calibration errors on the attribute classification accuracy, the measurement precision of attribute mastery classification, and the test information under the log-linear cognitive diagnosis model (LCDM) framework. The deterministic input, noisy “and” gate (DINA) model and the compensatory re-parameterized unified model (C-RUM) were used in fixed-length CD-CAT simulations. The results showed that high levels of calibration errors were associated with low classification accuracy, low test information, and misleading estimation of measurement precision. The effects of calibration errors decreased as the test length increased, and the DINA model appeared to be more vulnerable in the presence of calibration errors. The C-RUM was less influenced by calibration errors because of its additive characteristics in the LCDM framework. The same conclusions applied when item exposure control was incorporated and when different item selection methods were used. Finally, the use of a larger calibration sample size to calibrate the item pool was found to reduce the magnitudes of error variances and increase the attribute classification accuracy.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • CHEN, J., DE LA TORRE, J., and ZHANG, Z. (2013), “Relative and Absolute Fit Evaluation in Cognitive Diagnosis Modeling”, Journal of Educational Measurement, 50, 123–140.

    Article  Google Scholar 

  • CHEN, P., XIN, T., WANG, C., and CHANG, H. (2012), “On-Line Calibration Methods in Cognitive Diagnostic Computerized Adaptive Testing”, Psychometrika, 77, 201–222.

    Article  MathSciNet  Google Scholar 

  • CHEN, S.-Y., LEI, P., and LIAO, W. (2008), “Controlling Item Exposure and Test Overlap on the Fly in Computerized Adaptive Testing”, British Journal of Mathematical and Statistical Psychology, 61, 471–492.

    Article  MathSciNet  Google Scholar 

  • CHEN, Y., LIU, J., and YING, Z. (2015), “Online Item Calibration for Q-Matrix in CD-CAT”, Applied Psychological Measurement, 39, 5–15.

    Article  Google Scholar 

  • CHENG, Y. (2009), “When Cognitive Diagnosis Meets Computerized Adaptive Testing: CD-CAT”, Psychometrika, 74, 619–632.

    Article  MathSciNet  Google Scholar 

  • CHENG, Y., PATTON, J., and SHAO, C. (2015), “A-Stratified Computerized Adaptive Testing in the Presence of Calibration Error”, Educational and Psychological Measurement, 75, 260–283.

    Article  Google Scholar 

  • CHOI, H.-J., TEMPLIN, J., COHEN, A., and ATWOOD, C. (2010), “The Impact of Model Misspecification on Estimation Accuracy in Diagnostic Classification Models”, paper presented at the annual meeting of the National Council on Measurement in Education, Denver, CO.

  • DE LA TORRE, J. (2011), “The Generalized DINA Model Framework”, Psychometrika, 76, 179–199.

    Article  MathSciNet  Google Scholar 

  • DE LA TORRE, J., and CHIU, C.-Y. (2016), “A General Method of Empirical Q-Matrix Validation”, Psychometrika, 81, 253–273.

    Article  MathSciNet  Google Scholar 

  • DOEBLER, A. (2012), “The Problem of Bias in Person Parameter Estimation in Adaptive Testing”, Applied Psychological Measurement 36, 255–270.

    Article  Google Scholar 

  • HENSON, R., and DOUGLAS, F. (2005), “Test Constriction for Cognitive Diagnosis”, Applied Psychological Measurement 29, 262–277.

    Article  MathSciNet  Google Scholar 

  • HENSON, R., TEMPLIN, J., and WILLSE, J. (2009), “Defining a Family of Cognitive Diagnosis Models Using Log Linear Models with Latent Variables”, Psychometrika, 74, 191–210.

    Article  MathSciNet  Google Scholar 

  • HSU, C.-L., WANG, W.-C., and CHEN, S.-Y. (2013), “Variable-Length Computerized Adaptive Testing Based on Cognitive Diagnosis Models”, Applied Psychological Measurement, 37, 563–582.

    Article  Google Scholar 

  • HSU, C.-L., and WANG, W.-C. (2015), “Variable-Length Computerized Adaptive Testing Using the Higher Order DINA Model”, Journal of Educational Measurement, 52, 125–143.

    Article  Google Scholar 

  • KAPLAN, M., DE LA TORRE, J., and BARRADA, J.R. (2015), “New Item Selection Methods for Cognitive Diagnosis Computerized Adaptive Testing”, Applied Psychological Measurement, 39, 167–188.

    Article  Google Scholar 

  • KUNINA-HABENICHT, O., RUPP, A.A., and WILHELM, O. (2012), “The Impact of Model Misspecification on Parameter Estimation and Item-fit Assessment in Log-linear Diagnostic Classification Models”, Journal of Educational Measurement, 49, 59–81.

    Article  Google Scholar 

  • LEIGHTON J.P., GIERL M.J., and HUNKA S.M. (2004), “The Attribute Hierarchy Method for Cognitive Assessment: A Variation on Tatsuoka’s Rule-Space Approach”, Journal of Educational Measurement, 41, 205–237.

    Article  Google Scholar 

  • LIU, H., YOU, X., WANG, W., DING, S., and CHANG, H.-H. (2013), “The Development of Computerized Adaptive Testing with Cognitive Diagnosis for an English Achievement Test in China”, Journal of Classification, 30, 152–172.

    Article  MathSciNet  Google Scholar 

  • MAO, X., and XIN, T. (2013), “The Application of the Monte Carlo Approach to Cognitive Diagnostic Computerized Adaptive Testing with Content Constraints”, Applied Psychological Measurement, 37, 482–496.

    Article  Google Scholar 

  • MCGLOHEN, M., and CHANG, H.-H. (2008), “Combining Computer Adaptive Testing Technology with Cognitive Diagnostic Assessment”, Behavior Research Methods, 40, 808–821.

    Article  Google Scholar 

  • OLEA, J., BARRADA, J.R., ABAD, F.J., PONSODA, V., and CUEVAS, L. (2012), “Computerized Adaptive Testing: The Capitalization on Chance Problem”, The Spanish Journal of Psychology, 15, 424–441.

    Article  Google Scholar 

  • PATTON, J., CHENG, Y., YUAN, K.-H, and DIAO, Q. (2013), “The Influence of Calibration Error on Variable-length Computerized Adaptive Testing”, Applied Psychological Measurement, 37, 24–40.

    Article  Google Scholar 

  • RUPP, A.A., TEMPLIN, J.L., and HENSON, R.A. (2010), Diagnostic Measurement: Theory, Methods, and Applications, New York: Guilford Press.

    Google Scholar 

  • ŞAHIN, A., and WEISS, D. (2015), “Effects of Calibration Sample Size and Item Bank Size on Ability Estimation in Computerized Adaptive Testing”, Educational Sciences: Theory and Practice, 15, 1585–1595.

    Google Scholar 

  • STOCKING, M.L. (1994), “Three Practical Issues for Modern Adaptive Testing Item Pools”, Research Report RR-94-5, Princeton NJ: Educational Testing Service.

  • SYMPSON, J.B., and HETTER, R.D. (1985), “Controlling Item-Exposure Rates in Computerized Adaptive Testing”, in Proceedings of the 27th Annual Meeting of the Military Testing Association, San Diego CA: Navy Personnel Research and Development Center, pp. 973–977.

  • TATSUOKA, K.K. (1990), “Toward an Integration of Item-Response Theory and Cognitive Error Diagnosis”, in Monitoring Skills and Knowledge Acquisition, eds. N. Frederiksen, R. Glaser, A. Lesgold, and M. Safto, Hillsdale, NJ: Erlbaum, pp. 453–488.

    Google Scholar 

  • VAN DER LINDEN, W.J., and GLAS, C.A.W. (2000), “Capitalization on Item Calibration Error in Adaptive Testing”, Applied Measurement in Education, 12, 35–53.

    Article  Google Scholar 

  • VAN DER LINDEN, W.J., and GLAS, C.A.W. (Eds.) (2010), Elements of Adaptive Testing, New York: Springer.

    MATH  Google Scholar 

  • WANG, C. (2013), “Mutual Information Item Selection Method in Cognitive Diagnostic Computerized Adaptive Testing with Short Test Length”, Educational and Psychological Measurement, 73, 1017–1035.

    Article  Google Scholar 

  • XU, X., CHANG, H., and DOUGLAS, J. (2003), “A Simulation Study to Compare CAT Strategies for Cognitive Diagnosis”, paper presented at the annual meeting of the American Educational Research Association, Chicago IL.

Download references

Author information

Authors and Affiliations

  1. Department of Psychology and Counseling, University of Taipei, No.1, Ai-Guo West Road, Taipei, 10048, Taiwan

    Hung-Yu Huang

Authors
  1. Hung-Yu Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hung-Yu Huang.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Huang, HY. Effects of Item Calibration Errors on Computerized Adaptive Testing under Cognitive Diagnosis Models. J Classif 35, 437–465 (2018). https://doi.org/10.1007/s00357-018-9265-y

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00357-018-9265-y

Keywords

Search

Navigation