Moutan Mukhopadhyay
Anand Nayyar
ABSTRACT
Despite the success and the popularity of the online learning system, it still lacks in dynamically adapting suitable pedagogical methods according to the changing emotions and behaviour of the learner, as can be done in the face-to-face mode of learning. This makes the learning process mechanized, which significantly affects the learning outcome. To resolve this, the first and necessary step is to assess the emotion of a learner and identify the change of emotions during a learning session. Usually, images of facial expressions are analysed to assess one's state of mind. However, human emotions are far more complex, and these psychological states may not be reflected only through the basic emotion of a learner (i.e. analysing a single image), but a combination of two or more emotions which may be reflected on the face over a period of time. From a real survey, we derived four complex emotions that are a combination of basic human emotions often experienced by a learner, in concert, during a learning session. To capture these combined emotions correctly, we considered a fixed set of continuous image frames, instead of discrete images. We built a CNN model to classify the basic emotions and then identify the states of mind of the learners. The outcome is verified mathematically as well as surveying the learners. The results show a 65% and 62% accuracy respectively, for emotion classification and state of mind identification.
- S. Pal, P. K. D. Pramanik and P. Choudhury, "A Step Towards Smart Learning: Designing an Interactive Video-Based M-Learning System for Educational Institutes," International Journal of Web-Based Learning and Teaching Technologies, vol. 14, no. 4, pp. 26--48, 2019.Google Scholar
Cross Ref
- S. Pal, P. K. D. Pramanik, T. Majumdar and P. Choudhury, "A semi-automatic metadata extraction model and method for video-based e-learning contents," Education and Information Technologies, vol. 24, no. 6, pp. 3243--3268, 2019.Google ScholarCross Ref
- M. A. A. Dewan, M. Murshed and F. Lin, "Engagement detection in online learning: a review," Smart Learning Environments, vol. VI, no. 1, pp. 1--20, 2019.Google ScholarCross Ref
- M. Dubey and L. Singh, "Automatic Emotion Recognition Using Facial Expression: A Review," International Research Journal of Engineering and Technology, vol. III, no. 02, pp. 488--492, 2016.Google Scholar
- S. L. Happy, A. Dasgupta, P. Patnaik and A. Routray, "Automated Alertness and Emotion Detection for Empathic Feedback during e-Learning," in IEEE Fifth International Conference on Technology for Education, Kharagpur, 2013.Google Scholar
- H. H. Binali, C. Wu and V. Potdar, "A new significant area: Emotion detection in E-learning using opinion mining techniques," in IEEE International Conference on Digital Ecosystems and Technologies, Istanbul, 2009.Google Scholar
- M. Saneiro, O. C. Santos, S. Salmeron-Majadas and J. G. Boticario, "Towards Emotion Detection in Educational Scenarios from Facial Expressions and Body Movements through Multimodal Approaches," The Scientific World Journal, vol. 2014 (Article ID 484873), 2014.Google Scholar
- K. Fujii, P. Marian, D. Clark, Y. Okamoto and J. Rekimoto, "Sync Class: Visualization System for In-Class Student Synchronization," in 9th Augmented Human International Conference, Seoul, 2018.Google ScholarDigital Library
- A. Sun, Y.-J. Li, Y.-M. Huang and Q. Li, "Using facial expression to detect emotion in e-learning system: A deep learning method," in International Symposium on Emerging Technologies for Education (SETE 2017), Cape Town, South Africa, 2017.Google Scholar
- J. Ou, "Classification Algorithms Research on Facial Expression Recognition," in International Conference on Solid State Devices and Materials Science, Hainan, China, 2012.Google Scholar
- L. B. Krithika and G. G. Lakshmi Priya, "Student Emotion Recognition System (SERS) for e-learning Improvement Based on Learner Concentration Metric," Procedia Computer Science, vol. 85, pp. 767--776, 2016.Google ScholarCross Ref
- I. M. Revina and W. R. SamEmmanuel, "A Survey on Human Face Expression Recognition Techniques," Journal of King Saud University - Computer and Information Sciences, 2018.Google ScholarCross Ref
- L. Chenak, C. Zhoua and L. Shenb, "Facial Expression Recognition Based on SVM in E-learning," IERI Procedia, vol. 2, pp. 781--787, 2012.Google ScholarCross Ref
- D. Yanga, A. Alsadoona, P. Prasad, A. K. Singh and A. Elchouemi, "An Emotion Recognition Model Based on Facial Recognition in Virtual Learning Environment," in 6th International Conference on Smart Computing and Communications, ICSCC 2017, Kurukshetra, 2017.Google Scholar
- G. Yang, J. S. Y. Ortoneda and J. Saniie, "Emotion Recognition Using Deep Neural Network with Vectorized Facial Features," in IEEE International Conference on Electro/Information Technology (EIT), Rochester, Michigan, USA, 2018.Google Scholar
- P. R. Dachapally, "Facial Emotion Detection Using Convolutional Neural Networks and Representational Autoencoder Units," arXiv, no. 1706.01509, 2015.Google Scholar
- H.-W. Ng, D. V. Nguyen, V. Vonikakis and S. Winkler, "Deep Learning for Emotion Recognition on Small Datasets Using Transfer Learning," in ACM International Conference on Multimodal Interaction, Seattle, 2015.Google Scholar
- S. Minaee and A. Abdolrashidi, "Deep-Emotion: Facial Expression Recognition Using Attentional Convolutional Network," arXiv, no. 1902.01019, 2019.Google Scholar
- M. M. Taghi Zadeh, M. Imani and B. Majidi, "Fast Facial emotion recognition Using Convolutional Neural Networks and Gabor Filters," in 5th Conference on Knowledge-Based Engineering and Innovation, Iran University of Science and Technology, Tehran, Iran, 2019.Google Scholar
- S. Zhang, X. Pan, Y. Cui, X. Zhao and L. Liu, "Learning Affective Video Features for Facial Expression Recognition via Hybrid Deep Learning," IEEE Access, vol. 7, pp. 32297--32304, 2019.Google ScholarCross Ref
- M. M. Abbasi and A. Beltiukov, "Summarizing Emotions from Text Using Plutchik's Wheel of Emotions," in Scientific Conference on Information Technologies for Intelligent Decision Making Support, Ufa, 2019.Google Scholar
- Microsoft Azure, "Face," Microsoft, 2019. [Online]. Available: https://azure.microsoft.com/en-in/services/cognitive-services/face/. [Accessed 1st December 2019].Google Scholar
- S. Li and W. Deng, "Deep Facial Expression Recognition: A Survey," Computer Vision and Pattern Recognition, pp. 1--25, 2018.Google Scholar
- R. Verma, "fer2013," Kaggle Inc, 26 May 2018. [Online]. Available: https://www.kaggle.com/deadskull7/fer2013. [Accessed 22 November 2019].Google Scholar
Index Terms
- Facial Emotion Detection to Assess Learner's State of Mind in an Online Learning System
Recommendations
Facial expression of emotion and perception of the Uncanny Valley in virtual characters
With technology allowing for increased realism in video games, realistic, human-like characters risk falling into the Uncanny Valley. The Uncanny Valley phenomenon implies that virtual characters approaching full human-likeness will evoke a negative ...
Facial emotion recognition based real-time learner engagement detection system in online learning context using deep learning models
AbstractThe dramatic impact of the COVID-19 pandemic has resulted in the closure of physical classrooms and teaching methods being shifted to the online medium.To make the online learning environment more interactive, just like traditional offline ...
On assisting a visual-facial affect recognition system with keyboard-stroke pattern information
Towards realizing a multimodal affect recognition system, we are considering the advantages of assisting a visual-facial expression recognition system with keyboard-stroke pattern information. Our work is based on the assumption that the visual-facial ...
Comments