Skip to main content
SpringerLink
Log in

Scalable continual quality control of formative assessment items in an educational digital library: an empirical study

  • Published:
International Journal on Digital Libraries Aims and scope Submit manuscript

Abstract

An essential component of any library of online learning objects is assessment items, for example, homework, quizzes, and self-study questions. As opposed to exams, these items are formative in nature, as they help the learner to assess his or her own progress through the material. When it comes to quality control of these items, their formative nature poses additional challenges. e.g., there is no particular time interval in which learners interact with these items, learners come to these items with very different levels of preparation and seriousness, guessing generates noise in the data, and the numbers of items and learners can be several orders of magnitude larger than in summative settings. This empirical study aims to find a highly scalable mechanism for continual quality control of this class of digital content with a minimalist amount of additional metadata and transactional data, while taking into account also characteristics of the learners. In a subsequent evaluation of the model on a limited set of transactions, we find that taking into account the learner characteristic of ability improves the quality of item metadata, and in a comparison to Item Response Theory (IRT), we find that the developed model in fact performs slightly better in terms of predicting the outcome of formative assessment transactions, while never matching the performance of IRT on predicting the outcome of summative assessment.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

References

  1. Kortemeyer, G.: Ten years later: why open educational resources have not noticeably affected higher education, and why we should care. EDUCAUSE Review Online, 02/26 (2013)

  2. Gonçalves, M.A., Moreira, B.L., Fox, E.A., Watson, L.T.: What is a good digital library? A quality model for digital libraries. Inf. Process. Manag. 43, 1416 (2007)

    Article  Google Scholar 

  3. Fuhr, N., Hansen, P., Mabe, M., Micsik, A., Soølvberg, I.: Digital libraries: a generic classification and evaluation scheme. In: Constantopoulos, P., Sølvberg, I. (eds.) Proceedings of ECDL ’01. Darmstadt, Germany (2001). 187

    Google Scholar 

  4. Xie, H.I.: Users evaluation of digital libraries (DLs): their uses, their criteria, and their assessment. Inf. Process. Manag. 44, 1346 (2008)

    Article  Google Scholar 

  5. Borgman, C.L., Gilliland-Swetland, A.J., Leazm, G.H., Maym, R., Gwynn, D., Gazan, R., Mautone, P.: Evaluating digital libraries for teaching and learning in undergraduate education: a case study of the Alexandria Digital Earth Prototype (ADEPT). Libr. Trends 49, 221 (2000)

    Google Scholar 

  6. McMartin, F., Iverson, E., Wolf, A., Morrill, J., Morgan, G., Manduca, C.: The use of online digital resources and educational digital libraries in higher education. Int. J. Digit. Libr. 9, 65 (2008)

    Article  Google Scholar 

  7. Bransford, John D., Brown, Ann L., Cocking, Rodney R. (eds.): How People Learn: Brain, Mind, Experience, and School, Expanded edn. The National Academies Press, Washington, DC (2000)

    Google Scholar 

  8. Kortemeyer, G., Hall, M., Parker, J., Minaei-Bidgoli, B., Albertelli II, G., Bauer, W., Kashy, E.: Effective feedback to the instructor from online homework. J. Asynchronous Learn. Netw. 9, 19–28 (2005)

  9. Novak, G.M., Patterson, E.T., Gavrin, A.D., Christian, W.: Just-In-Time-Teaching: Blending Active Learning with Web Technology. Prentice Hall, Upper Saddle River (1999)

    Google Scholar 

  10. Ricci, F., Rokach, L., Shapira, B., Kantor, P.B. (eds.): Recommender Systems Handbook. Springer, Berlin (2011)

    MATH  Google Scholar 

  11. Traub, R.E.: Classical test theory in historical perspective. Educ. Meas.: Issues Pract. 16(4), 8 (1997)

    Article  Google Scholar 

  12. Birnbaum, Allan: Some latent trait models and their use in inferring an examinees ability. In: Lord, F.M., Novick, M.R. (eds.) Statistical Theories of Mental Test Scores, pp. 374–472. Addison-Wesley, Reading, Mass (1968)

    Google Scholar 

  13. Lee, Young-Jin, Palazzo, David J., Warnakulasooriya, Rasil, Pritchard, David E.: Measuring student learning with item response theory. Phys. Rev. ST Phys. Educ. Res 4, 010102 (2008). doi:10.1103/PhysRevSTPER.4.010102

    Article  Google Scholar 

  14. Bergner, Y., Dröschler, S., Kortemeyer, G., Rayyan, S., Seaton, D., Pritchard, D.: Model-based collaborative filtering analysis of student response data: machine-learning item response theory. In: Proceedings of the 5th international conference on educational data mining, pp. 95–102 (2012)

  15. Kortemeyer, G.: Extending item response theory to online homework. Phys. Rev. ST Phys. Educ. Res 10, 010118 (2014). doi:10.1103/PhysRevSTPER.10.010118

    Article  Google Scholar 

  16. Brennan, R.L.: Generalizability Theory (Statistics for Social and Behavioral Sciences). Springer-Verlag, New York (2001)

    Google Scholar 

  17. Chiu, C.W.T.: Scoring Performance Assessments Based on Judgements: Generalizability Theory (Evaluation in Education and Human Services). Springer, New York (2001)

    Book  Google Scholar 

  18. Lord, F.M.: Maximum likelihood and Bayesian parameter estimation in item response theory. J. Educ. Meas. 23, 157162 (1986)

    Article  Google Scholar 

  19. Wingersky, N.S.: LOGIST: a program for computing maximum likelihood procedures for logistic test models. In: Hambleton, R.K. (ed.) Applications of Item Response Theory. Educational Research Institute of British Columbia, Vancouver (1983)

    Google Scholar 

  20. Mislevy, R.J., Bock, R.D.: BILOG—Maximum Likelihood Item Analysis and Test Scoring: LOGISTIC model. International Educational Services, Chicago (1981)

    Google Scholar 

  21. Swaminathan, B., Giford, J.A.: Estimation in the three-parameter latent traitmodel. In: Weis, J. (ed.) New Horizons in Testing. Academic Press, NewYork (1983)

    Google Scholar 

  22. Johnson, M.: Marginal maximum likelihood estimation of item response models in R. J. Stat. Softw. 20, 1 (2007)

    Article  Google Scholar 

  23. Palazzo, David J., Lee, Young-Jin, Warnakulasooriya, Rasil, Pritchard, David E.: Patterns, correlates, and reduction of homework copying. Phys. Rev. ST Phys. Educ. Res 6, 010104 (2010). [12 pages]

    Article  Google Scholar 

  24. Kortemeyer, G.: An empirical study of the effect of granting multiple tries for online homework. Am. J. Phys. (submitted) (preprint doi: arXiv:1407.2276)

  25. Kortemeyer, Gerd, Kashy, Edwin, Benenson, Walter, Bauer, Wolfgang: Experiences using the open-source learning content management and assessment system LON-CAPA in introductory physics courses. Am. J. Phys. 76, 438–444 (2008)

    Article  Google Scholar 

  26. Sumner, T., Dawe, M.: Looking at digital library usability from a reuse perspective. In: Proceedings of JCDL 01, Roanoke, VA, p. 416 (2001)

  27. Klebl, M., Krämer, B.: Distributed Repositories for Educational Content—Part 1: Information Management for Educational Content. eleed, vol. 7 (2010) (urn:nbn:de:0009-5-27716)

  28. Klebl, M., Krämer, B., Zobel, A., Hupfer, M., Lukaschik, C.: Distributed Repositories for Educational Content—Part 2: Technology. eleed, vol. 7 (2010) (urn:nbn:de:0009-5-27748)

  29. Kortemeyer, G., Dröschler, S., Pritchard, D.E.: Harvesting latent and usage-based metadata in a course management system to enrich the underlying educational digital library. Int. J. Digit. Libr. (2013). doi:10.1007/s00799-013-0107-6

  30. Muraki, E.: A generalized partial credit model: application of an EM algorithm. Appl. Psychol. Meas. 16, 159 (1992)

    Article  Google Scholar 

  31. Rizopoulos, D.: LTM: an R package for latent variable modelling and item response theory analyses. J. Stat. Softw. 17, 1 (2006)

    Article  Google Scholar 

  32. R Development Core Team.: R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna (2008) ISBN 3-900051-07-0

  33. Moore, G.E.: Cramming more components onto integrated circuits. Electron. Mag. 4, 114–117 (1965)

  34. Li, K., Uvah, J., Amin, R.: Predicting students performance in elements of statistics. US–China Educ. Rev. A 10, 875–884 (2012)

    Google Scholar 

  35. Nordstrom, B.H.: Predicting Performance in Freshman Chemistry. In: Proceedings of National Meeting of the American Chemical Society, Boston (1990)

  36. Patsias, K., Rahimi, M., Sheng, Y., Rahimi, S.: Parallel computing with a Bayesian item response model. Am. J. Comput. Math. 2, 65 (2002)

    Article  Google Scholar 

  37. Press, W.H., Flannery, B.P., Teukolsky, S.A., Vetterling, W.T.: Numerical recipes in Pascal. Cambridge University Press, New York, Port Chester, Melbourne, Sydney (1989). or any other work on numerical methods

  38. Brier, G.W.: Verification of forecasts expressed in terms of probability. Mon. Wea. Rev. 78, 1–3. doi:10.1175/1520-0493(1950)078<0001:VOFEIT>2.0.CO;2

  39. Kortemeyer, G.: Gender differences in the use of an online homework system in an introductory physics course. Phys. Rev. ST-Phys. Educ. Res. 5, 010107 (2009)

  40. IMS Learning Tool Interoperability. http://www.imsglobal.org/lti/. Accessed Nov 2014

Download references

Author information

Authors and Affiliations

  1. Michigan State University, East Lansing, USA

    Gerd Kortemeyer

Authors
  1. Gerd Kortemeyer
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Gerd Kortemeyer.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kortemeyer, G. Scalable continual quality control of formative assessment items in an educational digital library: an empirical study. Int J Digit Libr 17, 143–155 (2016). https://doi.org/10.1007/s00799-015-0145-3

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00799-015-0145-3

Keywords

Search

Navigation