Skip to main content
SpringerLink
Log in

Recognizing Cognitive Emotions in E-Learning Environment

  • Conference paper
  • First Online:
Bridges and Mediation in Higher Distance Education (HELMeTO 2020)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1344))

Abstract

In the present work, we describe the development of a Facial Expressions Recognition (FER) system able to recognize cognitive emotions in a distance education context. In this case, many research works show that the recognition of basic emotions is not enough and that recognizing emotions more related to the presence/lack of engagement and flow would be more appropriate. Therefore, we developed a FER system able to classify the following cognitive emotions: enthusiasm, interest, surprise, boredom, perplexity, frustration, and the neutral one. After several experiments, we tested which was the best combination of features and the best algorithm for our classification problem. Results show that the combination of Action Units and gaze and a Multiclass Support Vector Machine achieves the best accuracy on the dataset. Results are encouraging and we plan to integrate the system into an e-learning platform to create a more personalized educational environment.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Damasio, A.R.: Descartes Error: Emotion, Reason and the Human Brain. G.P. Putnam Sons, New York (1994)

    Google Scholar 

  2. Pekrun, R.: The control-value theory of achievement emotions: assumptions, corollaries, and implications for educational research and practice. Educ. Psychol. Rev. 18(4), 315–341 (2006). https://doi.org/10.1007/s10648-006-9029-9

    Article  Google Scholar 

  3. Pekrun, R., Goetz, T., Frenzel, A.C., Barchfeld, P., Perry, R.P.: Measuring emotions in students’ learning and performance: the achievement emotions questionnaire (AEQ). Contemp. Educ. Psychol. 36(1), 36–48 (2011)

    Article  Google Scholar 

  4. Picard, R.W.: Affective Computing. MIT Press, Cambridge (1997)

    Google Scholar 

  5. Del Coco, M., Carcagnì, P., Palestra, G., Leo, M., Distante, C.: Analysis of HOG suitability for facial traits description in FER problems. In: Murino, V., Puppo, E. (eds.) ICIAP 2015. LNCS, vol. 9280, pp. 460–471. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-23234-8_43

    Chapter  Google Scholar 

  6. Khalfallah, J., Slama, B.H.: Facial expression recognition for intelligent tutoring systems in remote laboratories platform. Procedia Comput. Sci. 73, 274–281 (2015)

    Article  Google Scholar 

  7. Shen, L., Wang, M., Shen, R.: Affective e-learning: “using emotional” data to improve learning in pervasive learning environment. J. Educ. Technol. Soc. 12(2), 176 (2009)

    Google Scholar 

  8. Ekman, P.: Basic emotions. In: Dalgleish, T., Power, M.J. (eds.) Handbook of Cognition and Emotion, pp. 45–60. Wiley, Hoboken (1999). https://doi.org/10.1002/0470013494.ch3

  9. O’regan, K.: Emotion and e-learning. J. Asynchronous Learn. Netw. 7(3), 78–92 (2003)

    Google Scholar 

  10. O’Reilly, H., et al.: The EU-emotion stimulus set: a validation study. Behav. Res. Methods 48(2), 567–576 (2016). https://doi.org/10.3758/s13428-015-0601-4

    Article  Google Scholar 

  11. Lyons, M.J., Akamatsu, S., Kamachi, M., Gyoba, J.: Coding facial expressions with gabor wavelets. In: 3rd IEEE International Conference on Automatic Face and Gesture Recognition, pp. 200–205 (1998). http://doi.org/10.1109/AFGR.1998.670949

  12. http://www.geocities.ws/senthilirtt/Senthil%20IRTT%20Face%20Database%20Version%201.1

  13. Fleiss, J.L.: Measuring nominal scale agreement among many raters. Psychol. Bull. 76(5), 378–382 (1971)

    Article  Google Scholar 

  14. Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2005, vol. 1, pp. 886–893. IEEE (2005)

    Google Scholar 

  15. Ekman, P., Friesen, W.: Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press, Palo Alto (1978)

    Google Scholar 

  16. https://github.com/TadasBaltrusaitis/OpenFace

  17. D’Errico, F., Paciello, M., De Carolis, B., Vattani, A., Palestra, G., Anzivino, G.: Cognitive emotions in e-learning processes and their potential relationship with students’ academic adjustment. Int. J. Emot. Educ. 10(1), 89–111 (2018)

    Google Scholar 

  18. De Carolis, B., D’Errico, F., Paciello, M., Palestra, G.: Cognitive emotions recognition in e-learning: exploring the role of age differences and personality traits. In: Gennari, R., Vittorini, P., De la Prieta, F., Di Mascio, T., Temperini, M., Azambuja Silveira, R., Ovalle Carranza, D.A. (eds.) MIS4TEL 2019. AISC, vol. 1007, pp. 97–104. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-23990-9_12

    Chapter  Google Scholar 

  19. Ashwin, T.S., Jose, J., Raghu, G., Reddy, G.R.: An E-learning system with multifacial emotion recognition using supervised machine learning. In: IEEE Seventh International Conference on Technology for Education (2015)

    Google Scholar 

  20. Al-Awni, A.: Mood extraction using facial features to improve learning curves of students in elearning systems. Int. J. Adv. Comput. Sci. Appl. 7(11), 444–453 (2016)

    Google Scholar 

  21. Krithika, L.B., Lakshmi Priyya, G.G.: Student emotion recognition system (SERS) for e-learning. Procedia Comput. Sci. 85, 767–776 (2016)

    Article  Google Scholar 

  22. Magdin, M., Turcani, M., Hudec, L.: Evaluating the Emotional State of a User Using a Webcam. Special Issue Artif. Intell. Underpinn. 4(1), 61–68 (2016)

    Google Scholar 

  23. Roli, F.: Multiple classifier systems. In: Li, S.Z., Jain, A. (eds.) Encyclopedia of Biometrics, p. 1843. Springer, Boston (2009). https://doi.org/10.1007/978-0-387-39940-9

    Chapter  Google Scholar 

  24. Ho, T.K.: Random decision forests. In: Proceedings of 3rd International Conference on Document Analysis and Recognition, Montreal, Quebec, Canada, vol. 1, pp. 278–282 (1995). https://doi.org/10.1109/icdar.1995.598994

  25. Rumelhart, D., Hinton, G., Williams, R.: Learning representations by back-propagating errors. Nature 323, 533–536 (1986). https://doi.org/10.1038/323533a0

    Article  MATH  Google Scholar 

  26. Tabassum, T., Allen, A.A., De, P.: Non-intrusive identification of student attentiveness and finding their correlation with detectable facial emotions. In: Proceedings of the 2020 ACM Southeast Conference (ACM SE 2020), pp. 127–134. Association for Computing Machinery, New York (2020)

    Google Scholar 

  27. D’Mello, S.K., Calvo, R.A.: Beyond the basic emotions: what should affective computing compute? In: Brewster, S., Bødker, S., Mackay, W. (eds.) Extended Abstracts of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI 2013), pp. 2287–2294. ACM, New York (2013)

    Google Scholar 

  28. Loderer, K., Pekrun, R., Lester, J.C.: Beyond cold technology: a systematic review and meta-analysis on emotions in technology-based learning environments. Learn. Instruct. 101162 (2018)

    Google Scholar 

  29. Duffy, M.C., Lajoie, S.P., Pekrun, R., Lachapelle, K.: Emotions in medical education: examining the validity of the medical emotion scale (MES) across authentic medical learning environments. Learn. Instruct. 101150 (2018)

    Google Scholar 

  30. Castelfranchi, C.: Affective Appraisal versus Cognitive Evaluation in Social Emotions and Interactions. In: Paiva, A. (ed.) IWAI 1999. LNCS (LNAI), vol. 1814, pp. 76–106. Springer, Heidelberg (2000). https://doi.org/10.1007/10720296_7

    Chapter  Google Scholar 

  31. Miceli, M., Castelfranchi, C.: Expectancy and Emotion. OUP, Oxford (2014)

    Google Scholar 

  32. Di Mele, L., D’Errico, F., Cerniglia, L., Cersosimo, M., Paciello, M.: Convinzioni di efficacia personale nella regolazione dell’apprendimento universitario mediato dalle tecnologie. Qwerty-Open Interdisc. J. Technol. Cult. Educ. 10(2), 63–77 (2015)

    Google Scholar 

  33. Pecchinenda, A., Petrucci, M.: Emotion unchained: facial expression modulates gaze cueing under cognitive load. PLoS ONE 11, e0168111 (2016)

    Article  Google Scholar 

  34. Cassano, F., Piccinno, A., Roselli, T., Rossano, V.: Gamification and learning analytics to improve engagement in university courses. In: Di Mascio, T., et al. (eds.) MIS4TEL 2018. AISC, vol. 804, pp. 156–163. Springer, Cham (2019). https://doi.org/10.1007/978-3-319-98872-6_19

    Chapter  Google Scholar 

  35. De Carolis, B., Ferilli, S., Novielli, N., Leuzzi, F., Rotella, F.: Social attitude recognition in multimodal interaction with a pedagogical agent. J. E-Learn. Knowl. Soc. 8(3), 141–151 (2012). https://doi.org/10.20368/1971-8829/649

    Article  Google Scholar 

  36. Malerba, D., et al.: Advanced programming of intelligent social robots.  J. E-Learn. Knowl. Soc.  15(2) (2019). https://doi.org/10.20368/1971-8829/1611

  37. D’Errico, F., Paciello, M., Cerniglia, L.: When emotions enhance students’ engagement in e-learning processes. J. e-Learn. Knowl. Soc. 12(4) (2016)

    Google Scholar 

  38. Bosch, N., D’Mello, S.K., Ocumpaugh, J., Baker, R.S., Shute, V.: Using video to automatically detect learner affect in computer-enabled classrooms. ACM Trans. Interact. Intell. Syst. 6(2), Article 17, 26 (2016). https://doi.org/10.1145/2946837

  39. Whitehill, J., Serpell, Z., Lin, Y.-C., Foster, A., Movellan, J.R.: The faces of engagement: automatic recognition of student engagement from facial expressions. IEEE Trans. Affect. Comput. 5(1), 86–98 (2014)

    Article  Google Scholar 

  40. Zeng, Z., Pantic, M., Roisman, G.I., Huang, T.S.: A survey of affect recognition methods: audio, visual, and spontaneous expressions. IEEE Trans. Pattern Anal. Mach. Intell. 31(1), 39–58 (2009)

    Article  Google Scholar 

  41. Temdee, P.: Smart learning environment: paradigm shift for online learning (2020). https://doi.org/10.5772/intechopen.85787

  42. Scheffler, I.: In Praise of the Cognitive Emotions. Routledge, New York (1991)

    Google Scholar 

  43. Bassi, M., Steca, P., Delle Fave, A., Caprara, G.V.: Academic self-efficacy beliefs and quality of experience in learning. J. Youth Adolesc. 36(3), 301–312 (2007). https://doi.org/10.1007/s10964-006-9069-y

    Article  Google Scholar 

  44. Chen, S., Dai, J., Yan, Y.: Classroom teaching feedback system based on emotion detection. In: 9th International Conference on Education and Social Science (ICESS 2019), pp. 940–946 (2019)

    Google Scholar 

  45. Zhang, K., Zhang, Z., Li, Z., Qiao, Y.: Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Sig. Process. Lett. 23(10), 1499–1503 (2016)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science, University of Bari, Bari, Italy

    Berardina De Carolis & Nicola Macchiarulo

  2. Department For.Psi.Com, University of Bari, Bari, Italy

    Francesca D’Errico

  3. Psychology Faculty, Uninettuno University, Rome, Italy

    Marinella Paciello

  4. Hero Srl, Salmour, Italy

    Giuseppe Palestra

Authors
  1. Berardina De Carolis
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Francesca D’Errico
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Nicola Macchiarulo
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Marinella Paciello
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Giuseppe Palestra
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Nicola Macchiarulo .

Editor information

Editors and Affiliations

  1. Giustino Fortunato University, Benevento, Italy

    Laura Sara Agrati

  2. Universidad Internacional de La Rioja, Logroño, La Rioja, Spain

    Daniel Burgos

  3. University of Pisa, Pisa, Italy

    Pietro Ducange

  4. University of Foggia, Foggia, Italy

    Pierpaolo Limone

  5. University of Bari Aldo Moro, Bari, Italy

    Loredana Perla

  6. eCampus University, Novedrate, Como, Italy

    Pietro Picerno

  7. eCampus University, Novedrate, Como, Italy

    Paolo Raviolo

  8. Open University in the Netherlands, Heerlen, The Netherlands

    Christian M. Stracke

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

De Carolis, B., D’Errico, F., Macchiarulo, N., Paciello, M., Palestra, G. (2021). Recognizing Cognitive Emotions in E-Learning Environment. In: Agrati, L.S., et al. Bridges and Mediation in Higher Distance Education. HELMeTO 2020. Communications in Computer and Information Science, vol 1344. Springer, Cham. https://doi.org/10.1007/978-3-030-67435-9_2

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-67435-9_2

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-67434-2

  • Online ISBN: 978-3-030-67435-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation