Skip to main content
SpringerLink
Log in

Student engagement detection in online environment using computer vision and multi-dimensional feature fusion

  • Regular Paper
  • Published:
Multimedia Systems Aims and scope Submit manuscript

Abstract

In the post-COVID-19 era, online learning has changed from an emergency teaching method to a new, normalized one. However, online learning is often plagued by low participation and high dropout compared to offline learning. A critical way to address these issues is the accurate detection of student engagement, which will help teachers promptly assess learners’ status. Image data are one of the most straightforward ways to reflect student engagement levels. However, traditional engagement detection methods with images either rely on manual analysis or interfere with student behavior, which leads to a need for more objectivity in final engagement levels. This paper proposes a system that utilizes images obtained from individual webcams in online classrooms. Based on the techniques of multi-dimensional feature fusion and multimodal analysis, this system can rapidly detect and output students’ classroom engagement levels which provides real-time support for teachers to adjust their teaching methods during the teaching process, aiming to enhance students’ engagement in online courses. In the feature extraction module, VGG16 is utilized to recognize students’ facial expressions, ResNet-101 is designed to estimate head pose in each image, and Mediapipe is applied to estimate facial landmarks that reflect eye–mouth behavior. Subsequently, a BP neural network is constructed to fuse these multi-dimensional features and output the engagement level in each image in the data fusion module. The present method is evaluated on the wacv2016 data set and achieves an accuracy of 62.03%, outperforming the single-dimensional method. It is also applied in online courses to demonstrate its validity in the actual scenario further. Pearson Correlation between engagement levels calculated by our multi-dimensional fusion method and NSSE–China survey scores filled out by students is 0.714. It indicates that the method can enable real-time monitoring of students’ classroom engagement with similar results to traditional questionnaires with little human resources and time.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Data availability

The datasets generated and analyzed during the current study are available from the corresponding authors upon reasonable request.

References

  1. Webb, M.E., et al.: “Machine learning for human learners: Opportunities, issues, tensions and threats.” Educ. Technol. Res. Dev. 69, 2109–2130 (2021)

    Google Scholar 

  2. Blankenberger, B., Williams, A.M.: COVID and the impact on higher education: The essential role of integrity and accountability. Adm. Theory Praxis 42(3), 404–423 (2020)

    Google Scholar 

  3. Yates, A., et al.: High school students’ experience of online learning during Covid-19 the influence of technology and pedagogy”. Technol. Pedag. Educ. 30(1), 59–73 (2021)

    Google Scholar 

  4. Muangprathub, J., Boonjing, V., Chamnongthai, K.: Learning recommendation with formal concept analysis for intelligent tutoring system. Heliyon 6(10), e05227 (2020)

    Google Scholar 

  5. Hofer, S.I., Nistor, N., Scheibenzuber, C.: Online teaching and learning in higher education: Lessons learned in crisis situations. Comput. Hum. Behav. 121, 106789 (2021)

    Google Scholar 

  6. Xue, E., Li, J., Liujie, Xu.: Online education action for defeating COVID-19 in China: An analysis of the system, mechanism and mode. Educ. Philos. Theory 54(6), 799–811 (2022)

    Google Scholar 

  7. Rapanta, C., et al.: “Online university teaching during and after the Covid-19 crisis: Refocusing teacher presence and learning activity.” Postdigital Sci. Educ. 2, 923–945 (2020)

    Google Scholar 

  8. Rawashdeh, Al., Zuhir, A., et al.: Advantages and disadvantages of using e-learning in university education: Analyzing students’ perspectives. Electron. J. E-learning 19(3), 107–117 (2021)

    Google Scholar 

  9. Pelletier, K., et al.: EDUCAUSE Horizon report teaching and learning edition. EDUC22, (2022)

  10. Hensley, L.C., Iaconelli, R., Wolters, C.A.: “This weird time we’re in”: How a sudden change to remote education impacted college students’ self-regulated learning. J. Res. Technol. Educ. 54(sup1), S203–S218 (2022)

    Google Scholar 

  11. Bozkurt, A., Sharma, R.C.: Education in normal, new normal, and next normal: Observations from the past, insights from the present and projections for the future. Asian J. Distance Educ. 15(2), i–x (2020)

    Google Scholar 

  12. Fredricks, J.A., Blumenfeld, P.C., Paris, A.H.: School engagement: Potential of the concept, state of the evidence. Rev. Educ. Res. 74(1), 59–109 (2004)

    Google Scholar 

  13. Phan, T., McNeil, S.G., Robin, B.R.: Students’ patterns of engagement and course performance in a Massive Open Online Course. Comput. Educ. 95, 36–44 (2016)

    Google Scholar 

  14. Fisher, R., Perényi, A., Birdthistle, N.: The positive relationship between flipped and blended learning and student engagement, performance and satisfaction. Act. Learn. High. Educ. 22(2), 97–113 (2021)

    Google Scholar 

  15. Hussain, M., et al.: “Student engagement predictions in an e-learning system and their impact on student course assessment scores.” Computational Intell. Neurosci. (2018). https://doi.org/10.1155/2018/6347186

    Article  Google Scholar 

  16. Walker, K.A., Koralesky, K.E.: “Student and instructor perceptions of engagement after the rapid online transition of teaching due to COVID-19.” Nat. Sci. Educ. 50(1), e20038 (2021)

    Google Scholar 

  17. Kim, C., et al.: “From motivation to engagement: The role of effort regulation of virtual high school students in mathematics courses.” J. Educ. Technol. Soc. 18(4), 261–272 (2015)

    Google Scholar 

  18. Martin, F., Borup, J.: “Online learner engagement: Conceptual definitions, research themes, and supportive practices.” Educ. Psychol. 57(3), 162–177 (2022)

    Google Scholar 

  19. Wang, Ze., Bergin, C., Bergin, D.A.: “Measuring engagement in fourth to twelfth grade classrooms: The classroom engagement inventory.” Sch. Psychol. Q. 29(4), 517 (2014)

    Google Scholar 

  20. Özhan, ŞÇ., Kocadere, S.A.: “The effects of flow emotional engagement and motivation on success in a gamified online learning environment.” J. Educ. Comput. Res. 57(8), 2006–2031 (2020)

    Google Scholar 

  21. D'Errico, F., Marinella P., Luca C.: “When emotions enhance students’ engagement in e-learning processes.” J. E-Learning Knowl. Soc. 12(4), (2016).

  22. Anthonysamy, L., Koo, A.C., Hew, S.H.: “Self-regulated learning strategies in higher education: Fostering digital literacy for sustainable lifelong learning.” Educ. Inf. Technol. 25, 2393–2414 (2020)

    Google Scholar 

  23. Dupeyrat, C., Mariné, C.: Implicit theories of intelligence, goal orientation, cognitive engagement, and achievement: A test of Dweck’s model with returning to school adults. Contemp. Educ. Psychol. 30(1), 43–59 (2005)

    Google Scholar 

  24. Alqurashi, E.: What do students engage with the most? A comparative study between high and low achieving students within online learning environments. Open Learning J. Open Distance E-Learning 37(3), 219–234 (2022)

    Google Scholar 

  25. Greene, B.A.: “Measuring cognitive engagement with self-report scales: Reflections from over 20 years of research.” Educ. Psychol. 50(1), 14–30 (2015)

    Google Scholar 

  26. O’Brien, H.L., Toms, E.G.: “The development and evaluation of a survey to measure user engagement.” J. Am. Soc. Inform. Sci. Technol. 61(1), 50–69 (2010)

    Google Scholar 

  27. Ober, T.M., et al.: “Linking self-report and process data to performance as measured by different assessment types.” Comp. Educ. 167, 104188 (2021)

    Google Scholar 

  28. Volpe, R.J., et al.: “Observing students in classroom settings: A review of seven coding schemes.” Sch. Psychol. Rev. 34(4), 454–474 (2005)

    Google Scholar 

  29. Wu, Y., et al.: “Digital twin of intelligent small surface defect detection with cyber-manufacturing systems.” ACM Trans. Internet Technol. (2022). https://doi.org/10.1145/3571734

    Article  Google Scholar 

  30. Wu, Y., et al.: “Edge computing driven low-light image dynamic enhancement for object detection.” IEEE Transactions Netw. Sci. Eng. (2022). https://doi.org/10.1109/TNSE.2022.3151502

    Article  Google Scholar 

  31. Ashwin, T.S., Guddeti, R.M.R.: “Automatic detection of students’ affective states in classroom environment using hybrid convolutional neural networks”. Educ. Inf. Technol. 25(2), 1387–1415 (2020)

    Google Scholar 

  32. Sharma, P., et al.: “Student engagement detection using emotion analysis, eye tracking and head movement with machine learning.” Technology and Innovation in Learning, Teaching and Education: Third International Conference, TECH-EDU 2022, Lisbon, Portugal, August 31–September 2, 2022, Revised Selected Papers. Cham: Springer Nature Switzerland, (2023)

  33. Dewan, M., Murshed, M., Lin, F.: Engagement detection in online learning: A review. Smart Learning Environ. 6(1), 1–20 (2019)

    Google Scholar 

  34. Altuwairqi, K., et al.: Student behavior analysis to measure engagement levels in online learning environments”. Sign. Image Video Process. 15(7), 1387–1395 (2021)

    Google Scholar 

  35. Aluja-Banet, T., Sancho, M.-R., Vukic, I.: Measuring motivation from the virtual learning environment in secondary education. J. Computational Sci. 36, 100629 (2019)

    Google Scholar 

  36. Ghergulescu, I., Muntean, C.H.: “ToTCompute: A novel EEG-based TimeOnTask threshold computation mechanism for engagement modelling and monitoring.” Int. J. Artif. Intell. Educ. 26(3), 821–854 (2016)

    Google Scholar 

  37. KerdawyMohamed, El., et al.: “The automatic detection of cognition using eeg and facial expressions.” Sensors 20(12), 3516 (2020)

    Google Scholar 

  38. Bustos-López, M., et al.: “Wearables for engagement detection in learning environments: A review.” Biosensors 12(7), 509 (2022)

    Google Scholar 

  39. Cacioppo, J.T., Tassinary, L.G.: Inferring psychological significance from physiological signals. Am. Psychol. 45(1), 16 (1990)

    Google Scholar 

  40. Uzelac, A., Gligoric, N., Krco, S.: “A comprehensive study of parameters in physical environment that impact students’ focus during lecture using internet of things”. Comput. Hum. Behav. 53, 427–434 (2015)

    Google Scholar 

  41. Whitehill, J., et al.: “The faces of engagement: Automatic recognition of student engagement from facial expressions.” IEEE Trans. Affect. Comput. 5(1), 86–98 (2014)

    Google Scholar 

  42. Yu, H., et al.: “Measuring and integrating facial expressions and head pose as indicators of engagement and affect in tutoring systems.” International Conference on Human-Computer Interaction. Springer, Cham, (2021)

  43. Buono, P., et al.: “Assessing student engagement from facial behavior in on-line learning.” Multimedia Tools and Applications, 1–19 (2022)

  44. Shen, J., et al.: “Assessing learning engagement based on facial expression recognition in MOOC’s scenario.” Multimed. Syst. 28(2), 469–478 (2022)

    MathSciNet  Google Scholar 

  45. Savchenko, A.V., Savchenko, L.V., Makarov, I.: “Classifying emotions and engagement in online learning based on a single facial expression recognition neural network.” IEEE Trans. Affect. Comput. 13(4), 2132–2143 (2022)

    Google Scholar 

  46. Yirui, W., Ma, Y., Wan, S.: “Multi-scale relation reasoning for multi-modal visual question answering.” Sign. Process. Image Commun. (2021). https://doi.org/10.1016/j.image.2021.116319

    Article  Google Scholar 

  47. Zhang, W., et al.: “Transformer-based multimodal information fusion for facial expression analysis.” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. (2022)

  48. Garrison, D.R., Cleveland-Innes, M.: “Facilitating cognitive presence in online learning: Interaction is not enough.” Am J. Distance Educ. 19(3), 133–148 (2005)

    Google Scholar 

  49. Smilek, D., Carriere, J.S.A., Allan Cheyne, J.: “Out of mind, out of sight: Eye blinking as indicator and embodiment of mind wandering.” Psychol. Sci. 21(6), 786–789 (2010)

    Google Scholar 

  50. Al-Madani, Ali Mansour, et al.: “Real-time driver drowsiness detection based on eye movement and yawning using facial landmark.” 2021 International Conference on Computer Communication and Informatics (ICCCI). IEEE, (2021)

  51. Pabba, C., Kumar, P.: “An intelligent system for monitoring students’ engagement in large classroom teaching through facial expression recognition”. Expert. Syst. 39(1), e12839 (2022)

    Google Scholar 

  52. Li, X., et al.: “Multi-window Transformer parallel fusion feature pyramid network for pedestrian orientation detection.” Multime. Syst. (2022). https://doi.org/10.1007/s00530-022-00993-9

    Article  Google Scholar 

  53. Monkaresi, H., et al.: “Automated detection of engagement using video-based estimation of facial expressions and heart rate.” IEEE Trans. Affect. Comput. 8(1), 15–28 (2016)

    Google Scholar 

  54. Zhang, Z., et al.: “Data-driven online learning engagement detection via facial expression and mouse behavior recognition technology.” J. Educ. Comput. Res. 58(1), 63–86 (2020)

    MathSciNet  Google Scholar 

  55. Xiaoming, C., et al.: Research on student engagement recognition method from the perspective of artificial intelligence: Analysis of deep learning experiment based on a multimodal data fusion. J. Dist. Educ. 01, 32–44 (2019)

    Google Scholar 

  56. Lugaresi, C., et al.: “Mediapipe: A framework for building perception pipelines.” arXiv preprint arXiv:1906.08172 (2019)

  57. Cech, J., Tereza S.: “Real-time eye blink detection using facial landmarks.” Cent. Mach. Perception, Dep. Cybern. Fac. Electr. Eng. Czech Tech. Univ. Prague. 1–8 (2016)

  58. Carrier, Pierre-Luc, et al.: “FER-2013 face database.” Universit de Montral 3 (2013)

  59. He, Kaiming, et al.: “Deep residual learning for image recognition.” Proceedings of the IEEE conference on computer vision and pattern recognition. (2016)

  60. Ruiz, N., Chong, E., Rehg, J.M.: Fine-grained head pose estimation without keypoints[C]//Proceedings of the IEEE conference on computer vision and pattern recognition workshops. 2074–2083, 2018

  61. Wang, H., Zhenghua C., Yi Z.: “Hybrid coarse-fine classification for head pose estimation.” arXiv preprint arXiv:1901.06778 (2019)

  62. Wu, Y., Ji, Q.: “Facial landmark detection: A literature survey.” Int. J. Comput. Vision 127, 115–142 (2019)

    Google Scholar 

  63. Kamath, A., Aradhya B., Vineeth B.: “A crowdsourced approach to student engagement recognition in e-learning environments.” 2016 IEEE Winter Conference on Applications of Computer Vision (WACV). IEEE, (2016)

  64. Gupta, A., et al.: “Daisee: Towards user engagement recognition in the wild.” arXiv preprint arXiv:1609.01885 (2016)

  65. Zheng, X., et al.: “Estimation of learners’ engagement using face and body features by transfer learning.” Artificial Intelligence in HCI: Second International Conference, AI-HCI 2021, Held as Part of the 23rd HCI International Conference, HCII 2021, Virtual Event, July 24–29, 2021, Proceedings. Cham: Springer International Publishing, (2021)

  66. Huang, T., et al.: “Fine-grained engagement recognition in online learning environment.” 2019 IEEE 9th international conference on electronics information and emergency communication (ICEIEC). IEEE, (2019)

  67. Kamath, S., et al.: “Engagement analysis of students in online learning environments.” Machine learning and big data analytics (Proceedings of International Conference on Machine Learning and Big Data Analytics (ICMLBDA) 2021). Springer International Publishing, (2022)

  68. Yonghe, Z., et al.: Learning engagement detection based on face dataset in the mixed scene. Mod. Educ. Technol. 31(10), 84–92 (2021)

    Google Scholar 

  69. Park, Y.-M., Lee, G.-M., Yang, H.-S.: “Deep feature based efficient regularised ensemble for engagement recognition.” Electron. Lett. 55(24), 1281–1283 (2019)

    Google Scholar 

  70. Batra, S., et al.: “DMCNet: Diversified model combination network for understanding engagement from video screengrabs.” Syst. Soft Comput. 4, 200039 (2022)

    Google Scholar 

  71. Liao, J., Liang, Y., Pan, J.: “Deep facial spatiotemporal network for engagement prediction in online learning.” Appl. Intell. 51, 6609–6621 (2021)

    Google Scholar 

  72. Tanwar, S., Vinay K., Shailza S.: “Engagement measurement of a learner during e-learning: A deep learning architecture.” 2022 Seventh International Conference on Parallel, Distributed and Grid Computing (PDGC). IEEE, (2022)

  73. Abedi, A., Shehroz S. K.: “Improving state-of-the-art in detecting student engagement with resnet and tcn hybrid network.” 2021 18th Conference on Robots and Vision (CRV). IEEE, (2021)

  74. Monahan, T., Fisher, J.A.: “Benefits of ‘observer effects’: lessons from the field.” Qual. Res. 10(3), 357–376 (2010)

    Google Scholar 

  75. Goldberg, P., et al.: “Attentive or not? Toward a machine learning approach to assessing students’ visible engagement in classroom instruction”. Educ. Psychol. Rev. 33, 27–49 (2021)

    Google Scholar 

  76. Yan, L., et al.: Higher education measurement in the context of globalization—The development of NSSE-China: cultural adaptation, reliability and validity. Fudan Educ. Forum. 05, 12–18 (2009)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Institute of Vocational Education, Tongji University, Shanghai, 200082, China

    Nan Xie, Wei Pang & Beier Lu

  2. Sino-German College of Applied Sciences, Tongji University, Shanghai, 200082, China

    Zhaojie Liu & Zhengxu Li

Authors
  1. Nan Xie
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhaojie Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zhengxu Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Wei Pang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Beier Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

NX proposed the work concept and designed the manuscript. ZL and ZL collected online videos and processed raw data. WP and BL made important revisions to the manuscript. All authors reviewed the manuscript.

Corresponding author

Correspondence to Nan Xie.

Ethics declarations

Conflict of interest

The authors declare no conflict of interest.

Additional information

Communicated by A. Liu.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Xie, N., Liu, Z., Li, Z. et al. Student engagement detection in online environment using computer vision and multi-dimensional feature fusion. Multimedia Systems 29, 3559–3577 (2023). https://doi.org/10.1007/s00530-023-01153-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00530-023-01153-3

Keywords

Search

Navigation