Skip to main content
SpringerLink
Log in

Video-Based Emotion Recognition in the Wild for Online Education Systems

  • Conference paper
  • First Online:
PRICAI 2022: Trends in Artificial Intelligence (PRICAI 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13631))

Included in the following conference series:

Abstract

With the rapid development of the Internet, online learning has become one of the main ways of acquiring knowledge. In order to make teachers understand students’ emotional states and adjust teaching programs on time, a new video-based model called the Wild Facial Spatiotemporal Network (WFSTN) is proposed in this paper for emotion recognition in online learning environments. The model consists of two modules: a pretrained DenseNet121 for extracting facial spatial features, and a Bidirectional Long-Short Term Memory (Bi-LSTM) network with self-attention for generating attentional hidden states. In addition, a dataset of student emotions in online learning environments (DSEOLE) is produced using a self-developed online educational aid system. The method is evaluated on the Acted Facial Expressions in the Wild (AFEW) and DSEOLE datasets, achieving 72.76% and 73.67% accuracy in three-class classification, respectively. The results show that the proposed method outperforms many existing works on emotion recognition for online education.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Panichkriangkrai, C., Silapasuphakornwong, P., Saenphon, T.: Emotion recognition of students during e-learning through online conference meeting. Sci., Eng. Health Stud. 15, 21020010–21020010 (2021)

    Google Scholar 

  2. Abdullah, S.M.S., Abdulazeez, A.M.: Facial expression recognition based on deep learning convolution neural network: a review. J. Soft Comput. Data Min. 2(1), 53–65 (2021)

    Google Scholar 

  3. Kollias, D., Tagaris, A., Stafylopatis, A.: On line emotion detection using retrainable deep neural networks. In: 2016 IEEE Symposium Series on Computational Intelligence (SSCI), pp. 1–8. IEEE (2016)

    Google Scholar 

  4. Enadula, S.M., Enadula, AS., Burri, RD.: Recognition of student emotions in an online education system. In: 2021 Fourth International Conference on Electrical, Computer and Communication Technologies (ICECCT), pp. 1–4 IEEE (2021))

    Google Scholar 

  5. Sabri, N., et al.: Student emotion estimation based on facial application in E-learning during COVID-19 pandemic. Int. J. Adv. Trends Comput. Sci. Eng, 9(1.4) (2020)

    Google Scholar 

  6. Pi, B., Wang, Y.: Facial expression recognition based on traditional machine learning and deep learning. Softw. Guide 19(6), 44–47 (2020)

    Google Scholar 

  7. Lasri, I., Solh, A.R., El Belkacemi, M.: Facial emotion recognition of students using convolutional neural network. In: 2019 third international conference on intelligent computing in data sciences (ICDS), pp. 1–6 IEEE (2019)

    Google Scholar 

  8. Chen, J., et al.: Research on facial expression recognition based on improved deep residual network model. J. Phys. Conf. Ser. 2010(1), 012139 (2021). IOP Publishing

    Google Scholar 

  9. Cai, Y., Zheng, W., Zhang, T., Li, Q., Cui, Z., Ye, J.: Video based emotion recognition using CNN and BRNN. In: Tan, T., Li, X., Chen, X., Zhou, J., Yang, J., Cheng, H. (eds.) CCPR 2016. CCIS, vol. 663, pp. 679–691. Springer, Singapore (2016). https://doi.org/10.1007/978-981-10-3005-5_56

    Chapter  Google Scholar 

  10. Kaya, H., Gürpınar, F., Salah, A.A.: Video-based emotion recognition in the wild using deep transfer learning and score fusion. Image Vis. Comput. 65, 66–75 (2017)

    Article  Google Scholar 

  11. Vaswani, A., et al.: Attention is all you need. Adv. Neural Inf. Process. Sys. 30 (2017)

    Google Scholar 

  12. Zhang, K., Zhang, Z., Li, Z., et al.: Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Process. Lett. 23(10), 1499–1503 (2016)

    Article  Google Scholar 

  13. Huang, G., et al.: Densely connected convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4700–4708 (2017)

    Google Scholar 

  14. Cao, Q., et al.: Vggface2: a dataset for recognising faces across pose and age. In: 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), pp. 67-74. IEEE (2018)

    Google Scholar 

  15. Schuster, M., Paliwal, K.K.: Bidirectional recurrent neural networks. IEEE Trans. Signal Process. 45(11), 2673–2681 (1997)

    Article  Google Scholar 

  16. Deng, J,, et al. Arcface: additive angular margin loss for deep face recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4690–4699 (2019)

    Google Scholar 

  17. Dhall, A., et al.: Collecting large, richly annotated facial-expression databases from movies. IEEE Multim. 19(03), 34–41 (2012)

    Article  Google Scholar 

  18. Ekman, P., Friesen, W.V.: Detecting deception from the body or face. J. Pers. Soc. Psychol. 29(3), 288 (1974)

    Article  Google Scholar 

  19. Jain, A., Sah, H.R., Kothari, A.: Study for emotion recognition of different age groups students during online class. In: 2021 8th International Conference on Computing for Sustainable Global Development (INDIACom), pp. 621–625. IEEE (2021)

    Google Scholar 

  20. Szegedy, C,, et al.: Going deeper with convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–9 (2015)

    Google Scholar 

  21. Mai, G., et al.: Calligraphy font recognition algorithm based on improved DenseNet network. Computer Syst. Appl.. 31(2), 253–259 (2022)

    Google Scholar 

  22. Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)

    Article  Google Scholar 

  23. Wang, F., et al. Normface: L2 hypersphere embedding for face verification. In: Proceedings of the 25th ACM International Conference on Multimedia, pp. 1041–1049 (2017)

    Google Scholar 

  24. Robbins, H., Monro, S.: A stochastic approximation method. Ann. Math. Stat. 22, 400–407 (1951)

    Google Scholar 

  25. Wright, L.: New deep learning optimizer, ranger: synergistic combination of radam+ lookahead for the best of both (2019). Github. https://github.com/lessw2020/Ranger-Deep-Learning-Optimizer

  26. Liu L, Jiang H, He P, et al. On the variance of the adaptive learning rate and beyond[J]. arXiv preprint arXiv:1908.03265, 2019

  27. Zhang, M., et al. Lookahead optimizer: k steps forward, 1 step back. Adv. Neural Inf. Process. Syst. 32 (2019)

    Google Scholar 

  28. Sarrafzadeh, A., et al.: “How do you know that I don’t understand?” A look at the future of intelligent tutoring systems. Comput. Hum. Behav. 24(4), 1342–1363 (2008)

    Article  Google Scholar 

  29. Pekrun, R., et al.: Academic emotions in students’ self-regulated learning and achievement: a program of qualitative and quantitative research. Educ. Psychol. 37(2), 91–105 (2002)

    Article  Google Scholar 

  30. Liao, J., Liang, Y., Pan, J.: Deep facial spatiotemporal network for engagement prediction in online learning. Appl. Intell. 51(10), 6609–6621 (2021)

    Article  Google Scholar 

  31. He,K., et al.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

    Google Scholar 

  32. Barsoum, E., et al.: Training deep networks for facial expression recognition with crowd-sourced label distribution. In: Proceedings of the 18th ACM International Conference on Multimodal Interaction, pp. 279–283 (2016)

    Google Scholar 

  33. Wang, H., et al. CosFace: large margin cosine loss for deep face recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 5265–5274 (2018)

    Google Scholar 

  34. Shabrina, N.H., et al.: Emotion recognition using convolutional neural network in virtual meeting environment. Ultima Comput. Jurnal Sistem Komputer 13(1), 30–38 (2021)

    Article  Google Scholar 

  35. Nithiyasree, K.: Facial emotion recognition of students using deep convolutional neural network. Turk. J. Comput. Math. Educ. 12(10), 1430–1434 (2021)

    Google Scholar 

  36. Li, Q., et al.: Real-time facial emotion recognition using lightweight convolution neural network. J. Phys. Conf. Ser. IOP Publish. 1827(1), 012130 (2021)

    Article  Google Scholar 

  37. Wang, W., et al.: Emotion recognition of students based on facial expressions in online education based on the perspective of computer simulatio. Complexity 2020 (2020)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Software, South China Normal University, Guangzhou, 510630, China

    Genting Mai, Zijian Guo, Yicong She, Hongni Wang & Yan Liang

Authors
  1. Genting Mai
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zijian Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yicong She
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hongni Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yan Liang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yan Liang .

Editor information

Editors and Affiliations

  1. CSIRO Australian e-Health Research Centre, Brisbane, QLD, Australia

    Sankalp Khanna

  2. Shanghai Jiao Tong University, Shanghai, China

    Jian Cao

  3. University of Tasmania, Hobart, TAS, Australia

    Quan Bai

  4. University of Technology Sydney, Sydney, NSW, Australia

    Guandong Xu

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Mai, G., Guo, Z., She, Y., Wang, H., Liang, Y. (2022). Video-Based Emotion Recognition in the Wild for Online Education Systems. In: Khanna, S., Cao, J., Bai, Q., Xu, G. (eds) PRICAI 2022: Trends in Artificial Intelligence. PRICAI 2022. Lecture Notes in Computer Science, vol 13631. Springer, Cham. https://doi.org/10.1007/978-3-031-20868-3_38

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-20868-3_38

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-20867-6

  • Online ISBN: 978-3-031-20868-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation