Skip to main content
SpringerLink
Log in

FullExpression Using Transfer Learning in the Classification of Human Emotions

  • Conference paper
  • First Online:
Ambient Intelligence – Software and Applications (ISAmI 2020)

Part of the book series: Advances in Intelligent Systems and Computing ((AISC,volume 1239))

Included in the following conference series:

Abstract

During human evolution emotion expression became an important social tool that contributed to the complexification of societies. Human-computer interaction is commonly present in our daily life, and the industry is struggling for solutions that can analyze human emotions, to improve workers safety and security, as well as processes optimization. In this work we present a software built using the transfer-learning technique on a deep learning model, and conclude about how it can classify human emotions through facial expression analysis. A Convolutional Neuronal Network model was trained and used in a web application. Several tools were created to facilitate the software development process, including the training and validation processes. Data was collected by combining several facial expression emotion databases. Software evaluation revealed an accuracy in identifying the correct emotions close to 80% .

The present work has been developed under the EUREKA – ITEA3 Project CyberFactory#1 (ITEA-17032) and Project CyberFactory#1PT (ANI–P2020 40124) co-funded by Portugal 2020).

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Massey, D.S.: A brief history of human society: the origin and role of emotion in social life: 2001 presidential address. Am. Sociol. Rev. 67(1), 1–29 (2002). https://www.jstor.org/stable/3088931. http://www.jstor.org/stable/3088931?origin=crossref

  2. Derks, D., Fischer, A.H., Bos, A.E.: The role of emotion in computer-mediated communication: a review. Comput. Hum. Behav. 24(3), 766–785 (2008)

    Article  Google Scholar 

  3. Hockenbury, D., Hockenbury, S.: Discovering Psychology. Worth Publishers, New York (2007)

    Google Scholar 

  4. Cherry, K.: The James-Lange Theory of Emotion (2018). https://www.verywellmind.com/what-is-the-james-lange-theory-of-emotion-2795305

  5. Bluhm, C.: Handbook on facial expression of emotion, November 2013. https://search.proquest.com/docview/1518034163?accountid=13904

  6. Ekman, P., Friesen, W.V.: Measuring facial movement. Environ. Psychol. Nonverbal Behav. 1(1), 55–75 (1976)

    Google Scholar 

  7. Ko, B.C.: A brief review of facial emotion recognition based on visual information. Sensors (Switzerland) 18(2), 401 (2018)

    Article  Google Scholar 

  8. Picard, R.W.: Affective computing. Technical report, MIT Media Laboratory, Cambridge (1995)

    Google Scholar 

  9. Gershgorn, D.: The data that transformed AI research – and possibly the world (2017). https://qz.com/1034972/the-data-that-changed-the-direction-of-ai-research-and-possibly-the-world/

  10. Seif, G.: I’ll tell you why Deep Learning is so popular and in demand, pp. 2–5 (2018). https://medium.com/swlh/ill-tell-you-why-deep-learning-is-so-popular-and-in-demand-5aca72628780

  11. Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., Adam, H.: MobileNets: efficient convolutional neural networks for mobile vision applications, April 2017. http://arxiv.org/abs/1704.04861

  12. Yang, L., Hanneke, S., Carbonell, J.: A theory of transfer learning with applications to active learning. Mach. Learn. 90(2), 161–189 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  13. Kaggle Inc.. Kaggle Machine Learning & Data Science Survey 2017, Technical report 2017. https://www.kaggle.com/surveys/2017

  14. Sun, C., Shrivastava, A., Singh, S., Gupta, A.: Revisiting unreasonable effectiveness of data in deep learning era arXiv:1707.02968 [cs] (2017). http://openaccess.thecvf.com/content ICCV 2017/papers/Sun Revisiting Unreasonable Effectiveness ICCV 2017 paper.pdf

  15. Abdat, F., Maaoui, C., Pruski, A.: Human-computer interaction using emotion recognition from facial expression. In: Proceedings - UKSim 5th European Modelling Symposium on Computer Modelling and Simulation. EMS 2011, pp. 196–201 (2011)

    Google Scholar 

  16. Kanade, T., Cohn, J., Tian, Y.: Comprehensive database for facial expression analysis. In: Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580), vol. 899, no. 3, pp. 46–53. IEEE Computer Society (2017). http://ieeexplore.ieee.org/document/840611/

  17. Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z., Matthews, I.: The extended Cohn-Kanade Dataset (CK+): a complete dataset for action unit and emotion-specified expression. In: n 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops, vol. 45, no. 7, pp. 94–101. IEEE, June 2010. http://ieeexplore.ieee.org/document/5543262/

  18. Velusamy, S., Kannan, H., Anand, B., Sharma, A., Navathe, B.: A method to infer emotions from facial action units. In: ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings, pp. 2028–2031 (2011)

    Google Scholar 

  19. Sadeghi, H., Raie, A.A., Mohammadi, M.R.: Facial expression recognition using geometric normalization and appearance representation. In: Iranian Conference on Machine Vision and Image Processing, MVIP, pp. 159–163 (2013)

    Google Scholar 

  20. Suchitra, S.P., Tripathi, S.: Real-time emotion recognition from facial images using Raspberry Pi II. In: 2016 3rd International Conference on Signal Processing and Integrated Networks (SPIN), pp. 666–670 (2016). http://ieeexplore.ieee.org/document/7566780/

  21. Verma, R., Dabbagh, M.Y.: Fast facial expression recognition based on local binary patterns. In: Canadian Conference on Electrical and Computer Engineering, pp. 1–4 (2013)

    Google Scholar 

  22. Lyons, M., Akamatsu, S., Kamachi, M., Gyoba, J.: Coding facial expressions with Gabor wavelets. In: Proceedings - 3rd IEEE International Conference on Automatic Face and Gesture Recognition, FG 1998, pp. 200–205 (1998)

    Google Scholar 

  23. Sun, J.M., Pei, X.S., Zhou, S.S.: Facial emotion recognition in modern distant education system using SVM. In: Proceedings of the 7th International Conference on Machine Learning and Cybernetics, ICMLC, vol. 6, pp. 3545–3548 (2008)

    Google Scholar 

  24. Nicolai, A., Choi, A.: Facial emotion recognition using fuzzy systems. In: Proceedings - 2015 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2015, pp. 2216–2221 (2016)

    Google Scholar 

  25. Patil, G., Suja, P.: Emotion recognition from 3D videos using optical flow method. In: 2017 International Conference On Smart Technologies For Smart Nation (SmartTechCon), pp. 825–829 (2017). https://ieeexplore.ieee.org/document/8358488/

  26. Zhang, X., Yin, L., Cohn, J.F., Canavan, S., Reale, M., Horowitz, A., Liu, P.: A high-resolution spontaneous 3D dynamic facial expression database. In: 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), vol. 38, pp. 1–6. IEEE, April 2013. http://ieeexplore.ieee.org/document/6553788/

  27. Viola, P., Jones, M.: Robust real-time object detection paul. Int. J. Comput. Vis. 4(57), 4 (2001)

    Google Scholar 

  28. Valstar, M.F., Pantic, M.: Fully automatic recognition of the temporal phases of facial actions. IEEE Trans. Syst. Man Cybern. Part B (Cybern.) 42(1), 28–43 (2012). http://ieeexplore.ieee.org/document/6020812/

  29. Peng, Z.Y., Zhou, Y., Wang, P.: Multi-pose face detection based on adaptive skin color and structure model. In: CIS 2009 - 2009 International Conference on Computational Intelligence and Security, vol. 1, pp. 325–329 (2009)

    Google Scholar 

  30. Peng, Z.Y., Zhu, Y.H., Zhou, Y.: Real-time facial expression recognition based on adaptive Canny operator edge detection. In: 2010 International Conference on MultiMedia and Information Technology, MMIT 2010, vol. 2, pp. 154–157 (2010)

    Google Scholar 

  31. Mistry, K., Zhang, L., Neoh, S.C., Jiang, M., Hossain, A., Lafon, B.: Intelligent appearance and shape based facial emotion recognition for a humanoid robot. In: SKIMA 2014–8th International Conference on Software. Knowledge, Information Management and Applications (2014)

    Google Scholar 

  32. Suk, M., Prabhakaran, B.: Real-time mobile facial expression recognition system-a case study. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, pp. 132–137 (2014)

    Google Scholar 

  33. Lopes, A.T., De Aguiar, E., Oliveira-Santos, T.: A facial expression recognition system using convolutional networks. In: Brazilian Symposium of Computer Graphic and Image Processing, vol. 2015, pp. 273–280, October 2015

    Google Scholar 

  34. Mollahosseini, A., Chan, D., Mahoor, M.H.: Going deeper in facial expression recognition using deep neural networks. In: 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1–10. IEEE, March 2016. http://ieeexplore.ieee.org/document/7477450/

  35. Kennedy, B., Balint, A.: EmotionNet. https://github.com/co60ca/EmotionNet

  36. Tautkute, I., Trzcinski, T., Bielski, A.: I know how you feel: emotion recognition with facial landmarks, pp. 1991–1993 (2018). http://arxiv.org/abs/1805.00326

  37. Viola, P., Jones, M.: Rapid object detection using a boosted cascade of simple features. In: Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001, vol. 1, no. 1-3, pp. I–511–I–518. IEEE Computer Society (1997). http://ieeexplore.ieee.org/document/990517/

  38. Sheetjs, “xlsx.”. http://sheetjs.com/opensource

  39. “charjs”. https://www.chartjs.org/

  40. “jszip”. https://stuk.github.io/jszip/

  41. “file saver”. https://www.npmjs.com/package/file-saver

  42. “Tensorflow.js.” https://js.tensorflow.org/

  43. Lundqvist, D., Flykt, A., Ohman, A.: The Karolinska Directed Emotional Faces - KDEF (1998). https://www.emotionlab.se/kdef/

  44. Chen, L.-F., Yen, Y.-S.: Taiwanese facial expression image database. Institute of Brain Science, National Yang-Ming University, Brain Mapping Laboratory (2007)

    Google Scholar 

  45. Righi, G., Peissig, J.J., Tarr, M.J.: Recognizing disguised faces. Visual Cogn. 20(2), 143–169 (2012). http://www.tandfonline.com/doi/abs/10.1080/13506285.2012.654624

Download references

Author information

Authors and Affiliations

  1. School of Engineering, Polytechnic of Porto (ISEP/IPP), Porto, Portugal

    Ricardo Rocha & Isabel Praça

  2. GECAD – Knowledge Engineering and Decision Support Research Centre, Porto, Portugal

    Isabel Praça

Authors
  1. Ricardo Rocha
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Isabel Praça
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Isabel Praça .

Editor information

Editors and Affiliations

  1. Departamento de Informática, University of Minho, ALGORITMI Center, Braga, Portugal

    Paulo Novais

  2. DIBRIS, University of Genoa, Genoa, Italy

    Gianni Vercelli

  3. Data Management Group, Technical University of Catalonia, Barcelona, Barcelona, Spain

    Josep L. Larriba-Pey

  4. Department Computer Science and Artificial Intelligence, ETS de Ingenierias Informática y de Telecomunicación, University of Granada, Granada, Spain

    Francisco Herrera

  5. BISITE Research Group, University of Salamanca, Salamanca, Salamanca, Spain

    Pablo Chamoso

Rights and permissions

Reprints and permissions

Copyright information

© 2021 The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Rocha, R., Praça, I. (2021). FullExpression Using Transfer Learning in the Classification of Human Emotions. In: Novais, P., Vercelli, G., Larriba-Pey, J.L., Herrera, F., Chamoso, P. (eds) Ambient Intelligence – Software and Applications . ISAmI 2020. Advances in Intelligent Systems and Computing, vol 1239. Springer, Cham. https://doi.org/10.1007/978-3-030-58356-9_8

Download citation

Publish with us

Policies and ethics

Search

Navigation