Skip to main content
SpringerLink
Log in

Evaluation Methods for an AI-Supported Learning Management System: Quantifying and Qualifying Added Values for Teaching and Learning

  • Conference paper
  • First Online:
Adaptive Instructional Systems. Design and Evaluation (HCII 2021)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 12792))

Included in the following conference series:

Abstract

Artificial intelligence offers great opportunities for the future, including for teaching and learning. Applications such as personalized recommendations and learning paths based on learning analytics [i.e. 1], the integration of serious games in intelligent tutoring systems [2], intelligent agents in the form of chatbots [3], and other emerging applications promise great benefits for individualized digital learning. However, what value do these applications really add and how can these benefits be measured?

With this article, we would like to give a brief overview of AI-supported functionalities for learning management system as well as their possible benefits for future learning environments. Furthermore, we outline methods for a comprehensive evaluation that meets the users’ needs and concretizes the actual benefit of an AI-supported LMS.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 99.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 129.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Krauss, C., Salzmann, A., Merceron, A.: Branched learning paths for the recommendation of personalized sequences of course items. In: Schiffner, D. (ed.) Proceedings of DeLFI Workshops 2018 co-located with 16th e-Learning Conference of the German Computer Society (DeLFI 2018) (2018)

    Google Scholar 

  2. Beyyoudh, M., Idrissi, M.K., Bennani, S.: A new approach of integrating serious games in intelligent tutoring systems. In: Serrhini, M., Silva, C., Aljahdali, S. (eds.) EMENA-ISTL 2019. LAIS, vol. 7, pp. 85–91. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-36778-7_10

    Chapter  Google Scholar 

  3. Nenkov, N., Dimitrov, G., Dyachenko, Y., et al.: Artificial intelligence technologies for personnel learning management systems. In: 2016 IEEE 8th International Conference on Intelligent Systems (IS), pp. 189–195. IEEE (2016)

    Google Scholar 

  4. Yu, S., Niemi, H., Mason, J. (eds.): Shaping Future Schools with Digital Technology. PRRE, Springer, Singapore (2019). https://doi.org/10.1007/978-981-13-9439-3

    Book  Google Scholar 

  5. Schulmeister, R.: Lernplattformen für das virtuelle Lernen: Evaluation und Didaktik (2003)

    Google Scholar 

  6. Fardinpour, A., Pedram, M.M., Burkle, M.: Intelligent learning management systems. Int. J. Distance Educ. Technol. 12, 19–31 (2014). https://doi.org/10.4018/ijdet.2014100102

    Article  Google Scholar 

  7. Mavrikis, M., Holmes, W.: Intelligent learning environments: design, usage and analytics for future schools. In: Yu, S., Niemi, H., Mason, J. (eds.) Shaping Future Schools with Digital Technology. PRRE, pp. 57–73. Springer, Singapore (2019). https://doi.org/10.1007/978-981-13-9439-3_4

    Chapter  Google Scholar 

  8. Köhnen, B.: Learning Experience Plattformen (2019). https://www.haufe-akademie.de/digitales-lernen/magazin/learning-experience-plattformen?chorid=04330006&em_src=kw&em_cmp=google%2Fdl%2Fhlx%2F82119%2F04330006&wnr=04330006&gclid=EAIaIQobChMI0umYtLbk7gIVSuDtCh3KfgsTEAAYASABEgLVS_D_BwE. Accessed 12 Feb 2021

  9. Kuo, B.-C., Hu, X.: Intelligent learning environments. Educ. Psychol. 39, 1195–1198 (2019). https://doi.org/10.1080/01443410.2019.1669334

    Article  Google Scholar 

  10. HR Technologist: Emerging Trends for AI in Learning Management Systems (2019). https://www.hrtechnologist.com/articles/learning-development/emerging-trends-for-ai-in-the-learning-management-system/. Accessed 12 Feb 2019

  11. Softengi AI-powered LMS (Learning Management System). https://softengi.com/blog/ai-blog/ai-powered-lms-learning-management-system/. Accessed 12 Feb 2021

  12. Verbert, K., Drachsler, H., Manouselis, N., et al.: Dataset-driven research for improving recommender systems for learning. In: Long, P., Siemens, G., Conole, G., et al. (eds.) Proceedings of the 1st International Conference on Learning Analytics and Knowledge - LAK 2011, pp. 44–53. ACM Press, New York (2011)

    Google Scholar 

  13. Krauss, C.: Time-dependent recommender systems for the prediction of appropriate learning objects. Technische Universität Berlin (2018)

    Google Scholar 

  14. BigData MadeSimple: Is artificial intelligence (AI) the next step of smart learning? (2019). https://bigdata-madesimple.com/is-artificial-intelligence-ai-the-next-step-of-smart-learning/. Accessed 12 Feb 2021

  15. zeomag: How will AI and Machine Learning revolutionize e-learning? (2018). https://www.zeolearn.com/magazine/how-will-ai-and-machine-learning-revolutionize-e-learning. Accessed 12 Feb 2021

  16. docebo: Enterprise E-Learning Trends 2020 A New Era of Learning (2019). https://uhlberg-advisory.de/wp-content/uploads/2019/11/Docebo-Enterprise-E-Learning-Trends-2020.pdf. Accessed 12 Feb 2021

  17. GYRUS: What are the must-haves in an AI-based LMS (2020). https://www.gyrus.com/What-are-the-must-haves-in-an-AI-based-LMS. Accessed 12 Feb 2021

  18. Kondo, N., Okubo, M., Hatanaka, T.: Early detection of at-risk students using machine learning based on LMS log data. In: 2017 6th IIAI International Congress Juli 2017, pp. 198–201 (2017)

    Google Scholar 

  19. eLearning Industry: eLearning Trends To Watch Out For In 2021 (2020). https://elearningindustry.com/elearning-technology-and-content-trends-2021. Accessed 12 Feb 2021

  20. forma LMS: E-learning Trends 2021 (2021). https://www.formalms.org/articles/22-elearning-trends-technology/229-e-learning-trends-2021.html. Accessed 12 Feb 2021

  21. eThink: Artificial Intelligence in the Workplace: The Future of the L&D Landscape (2020). https://ethinkeducation.com/blog/artificial-intelligence-workplace-future-ld-landscape/. Accessed 12 Feb 2021

  22. ideal: AI For Recruiting: A Definitive Guide For HR Professionals (2017). https://ideal.com/ai-recruiter-skillset/. Accessed 12 Feb 2021

  23. tituslearning: 5 ways artificial intelligence can be used in eLearning (2020). https://www.tituslearning.com/artificial-intelligence-elearning. Accessed 12 Feb 2021

  24. Sanders, J.R.: The Program Evaluation Standards: How to Assess Evaluations of Educational Programs, 2 edn. SAGE, Thousand Oaks (1998). 6 [printing]

    Google Scholar 

  25. Fitzpatrick, J.L., Sanders, J.R., Worthen, B.R.: Program Evaluation: Alternative Approaches and Practical Guidelines, 4th edn. Pearson Education, Upper Saddle River, Montreal (2011)

    Google Scholar 

  26. Manouselis, N., Drachsler, H., Vuorikari, R., Hummel, H., Koper, R.: Recommender systems in technology enhanced learning. In: Ricci, F., Rokach, L., Shapira, B., Kantor, P.B. (eds.) Recommender Systems Handbook, pp. 387–415. Springer, Boston, MA (2011). https://doi.org/10.1007/978-0-387-85820-3_12

    Chapter  Google Scholar 

  27. Stockmann, R.: Was ist eine gute Evaluation? Einführung zu Funktionen und Methoden von Evaluationsverfahren: (CEval-Arbeitspapier, 9). Universität des Saarlandes, Fak. 05 Empirische, Saarbrücken (2004)

    Google Scholar 

  28. Böttcher, W., Caspari, A., Hense, J., et al.: Standards für Evaluation, Erste Revision 2016, Mainz (2017)

    Google Scholar 

  29. Brahm, T., Jenert, T.: Technologieeinsatz von der Bedarfsanalyse bis zur Evaluation. In: Ebner, M., Schön, S. (eds.) Lehrbuch für Lernen und Lehren mit Technologien: [L3T], [Stand:] Mai 2011, pp. 127–134. Books on Demand, Norderstedt (2011)

    Google Scholar 

  30. Wang, F., Hannafin, M.J.: Design-based research and technology-enhanced learning environments. ETR&D 53, 5–23 (2005). https://doi.org/10.1007/BF02504682

    Article  Google Scholar 

  31. Krauss, C., Merceron, A., Arbanowski, S.: The timeliness deviation: a novel approach to evaluate educational recommender systems for closed-courses. In: Proceedings of the 9th International Conference on Learning Analytics & Knowledge (LAK19), pp. 195–204. ACM, New York (2019). https://doi.org/10.1145/3303772.3303774

  32. Hernández-Orallo, J.: Evaluation in artificial intelligence: from task-oriented to ability-oriented measurement. Artif. Intell. Rev. 48(3), 397–447 (2016). https://doi.org/10.1007/s10462-016-9505-7

    Article  Google Scholar 

  33. Anastas, J.W.: Quality in qualitative evaluation: issues and possible answers. Res. Soc. Work Pract. 14, 57–65 (2004). https://doi.org/10.1177/1049731503257870

    Article  Google Scholar 

  34. Aspers, P., Corte, U.: What is qualitative in qualitative research. Qual. Sociol. 42(2), 139–160 (2019). https://doi.org/10.1007/s11133-019-9413-7

    Article  Google Scholar 

  35. Patton, M.Q.: Qualitative Research & Evaluation Methods: Integrating Theory and Practice, 4th edn. SAGE, Los Angeles (2015)

    Google Scholar 

  36. Patton, M.Q.: Qualitative Evaluation Checklist (2003)

    Google Scholar 

  37. Şahin, M., Yurdugül, H.: Learners’ needs in online learning environments and third generation learning management systems (LMS 3.0). Tech. Know. Learn. (2020). https://doi.org/10.1007/s10758-020-09479-x

  38. Ali, L., Hatala, M., Gašević, D., et al.: A qualitative evaluation of evolution of a learning analytics tool. Comput. Educ. 58, 470–489 (2012). https://doi.org/10.1016/j.compedu.2011.08.030

    Article  Google Scholar 

  39. Drachsler, H., Bogers, T., Vuorikari, R., et al.: Issues and considerations regarding sharable data sets for recommender systems in technology enhanced learning. Procedia Comput. Sci. 1, 2849–2858 (2010). https://doi.org/10.1016/j.procs.2010.08.010

    Article  Google Scholar 

  40. Erdt, M., Fernandez, A., Rensing, C.: Evaluating recommender systems for technology enhanced learning: a quantitative survey. IEEE Trans. Learn. Technol. 8, 326–344 (2015). https://doi.org/10.1109/TLT.2015.2438867

    Article  Google Scholar 

  41. Said, A., Bellogín, A.: Comparative recommender system evaluation. In: Kobsa, A., Zhou, M., Ester, M., et al. (eds.) Proceedings of the 8th ACM Conference on Recommender systems - RecSys 2014, pp. 129–136. ACM Press, New York (2014)

    Google Scholar 

  42. Weibelzahl, S.: Evaluation of adaptive systems. In: Bauer, M., Gmytrasiewicz, P.J., Vassileva, J. (eds.) UM 2001. LNCS (LNAI), vol. 2109, pp. 292–294. Springer, Heidelberg (2001). https://doi.org/10.1007/3-540-44566-8_49

    Chapter  Google Scholar 

  43. Campos, P.G., Díez, F., Cantador, I.: Time-aware recommender systems: a comprehensive survey and analysis of existing evaluation protocols. User Model. User-Adap. Inter. 24(1–2), 67–119 (2013). https://doi.org/10.1007/s11257-012-9136-x

    Article  Google Scholar 

  44. Herlocker, J.L., Konstan, J.A., Terveen, L.G., et al.: Evaluating collaborative filtering recommender systems. ACM Trans. Inf. Syst. 22, 5–53 (2004). https://doi.org/10.1145/963770.963772

    Article  Google Scholar 

  45. Eraslan Yalcin, M., Kutlu, B.: Examination of students’ acceptance of and intention to use learning management systems using extended TAM. Br. J. Educ. Technol. 50, 2414–2432 (2019). https://doi.org/10.1111/bjet.12798

    Article  Google Scholar 

  46. Sahid, D.S.S., Santosa, P.I., Ferdiana, R., et al.: Evaluation and measurement of Learning Management System based on user experience. In: Seminar, I.A.E. (ed.) Proceedings of the 2016 6th International Annual Engineering Seminar, Eastparc Hotel, Yogyakarta, Indonesia, 1–3 August 2016. IEEE, Piscataway, NJ (2016)

    Google Scholar 

  47. Takashi Nakamura, W., Harada Teixeira de Oliveira, E., Conte, T.: Usability and user experience evaluation of learning management systems - a systematic mapping study. In: Proceedings of the 19th International Conference on Enterprise Information Systems. SCITEPRESS - Science and Technology Publications, pp. 97–108 (2017)

    Google Scholar 

  48. ISO: ISO 9241-11:2018 (2021). https://www.iso.org/standard/63500.html. Accessed 11 Feb 2021

  49. Santoso, H.B., Schrepp, M., Utomo, A.Y., et al.: Measuring user experience of the student-centered e-learning environment. J. Educ. Online 13, 58–79 (2016)

    Google Scholar 

  50. Laugwitz, B., Held, T., Schrepp, M.: Construction and evaluation of a user experience questionnaire. In: Holzinger, A. (ed.) USAB 2008. LNCS, vol. 5298, pp. 63–76. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-89350-9_6

    Chapter  Google Scholar 

  51. Hart, S.G., Staveland, L.E.: Development of NASA-TLX (Task Load Index): results of empirical and theoretical research. In: Hancock, P.A., Meshkati, N. (eds.) Human Mental Workload, vol. 52. North-Holland; Sole distributors for the U.S.A. and Canada, pp. 139–183. Elsevier Science Pub. Co, Amsterdam, New York (1988)

    Google Scholar 

  52. Bruder, A., Schwarz, J.: Evaluation of diagnostic rules for real-time assessment of mental workload within a dynamic adaptation framework. In: Sottilare, R.A., Schwarz, J. (eds.) HCII 2019. LNCS, vol. 11597, pp. 391–404. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-22341-0_31

    Chapter  Google Scholar 

  53. Schwarz, J., Fuchs, S.: Multidimensional real-time assessment of user state and performance to trigger dynamic system adaptation. In: Schmorrow, D.D., Fidopiastis, C.M. (eds.) AC 2017. LNCS (LNAI), vol. 10284, pp. 383–398. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-58628-1_30

    Chapter  Google Scholar 

  54. Schwarz, J.C.: Multifaktorielle Echtzeitdiagnose des Nutzerzustands in adaptiver Mensch-Maschine-Interaktion: Dissertation zur Erlangung des akademischen Grades Doktor der Philosophie (Dr. phil.) (2019)

    Google Scholar 

  55. Witte, T., Haase, H., Schwarz, J.: Measuring cognitive load for adaptive instructional systems by using a pressure sensitive computer mouse. In: Sottilare, R., Schwarz, J. (eds.) Adaptive Instructional Systems (2021)

    Google Scholar 

  56. Krauss, C., Merceron, A., An, T.-S., Zwicklbauer, M., Steglich, S., Arbanowski, S.: Teaching advanced web technologies with a mobile learning companion application. In: Proceedings of The 16th ACM World Conference on Mobile and Contextual Learning (mLearn 2017), Larnaca, Cyprus, 30 October–1 November 2017. ACM (2017). https://doi.org/10.1145/3136907.3136937

  57. European Union ESCO format. https://ec.europa.eu/esco/portal/home. Accessed 12 Feb 2021

  58. MS Global LOM. http://www.imsglobal.org/metadata/index.html. Accessed 12 Feb 2021

  59. IMS Global QTI. http://www.imsglobal.org/question/index.html. Accessed 12 Feb 2021

  60. IMS Global CALIPER. http://www.imsglobal.org/activity/caliper. Accessed 12 Feb 2021

  61. ADL xAPI. https://xapi.com/. Accessed 12 Feb 2021

  62. Krauss, C., Hauswirth, M.: Interoperable education infrastructures: a middleware that brings together adaptive, social and virtual learning technologies. In: The European Research Consortium for Informatics and Mathematics (ed) ERCIM NEWS: Special Theme: Educational Technology, pp. 9–10 (2020). ISSN 0926-4981. https://ercim-news.ercim.eu/images/stories/EN120/EN120-web.pdf. Accessed 12 Feb 2021

Download references

Author information

Authors and Affiliations

  1. Fraunhofer FKIE, Zanderstraße 5, 53225, Bonn, Germany

    Lisa Rerhaye & Daniela Altun

  2. Fraunhofer FOKUS, Kaiserin-Augusta-Allee 31, 10589, Berlin, Germany

    Christopher Krauss & Christoph Müller

Authors
  1. Lisa Rerhaye
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Daniela Altun
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Christopher Krauss
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Christoph Müller
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lisa Rerhaye .

Editor information

Editors and Affiliations

  1. Soar Technology, Inc., Orlando, FL, USA

    Robert A. Sottilare

  2. Fraunhofer FKIE, Wachtberg, Germany

    Jessica Schwarz

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Rerhaye, L., Altun, D., Krauss, C., Müller, C. (2021). Evaluation Methods for an AI-Supported Learning Management System: Quantifying and Qualifying Added Values for Teaching and Learning. In: Sottilare, R.A., Schwarz, J. (eds) Adaptive Instructional Systems. Design and Evaluation. HCII 2021. Lecture Notes in Computer Science(), vol 12792. Springer, Cham. https://doi.org/10.1007/978-3-030-77857-6_28

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-77857-6_28

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-77856-9

  • Online ISBN: 978-3-030-77857-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation