Skip to main content
SpringerLink
Log in

Hybrid Transfer Learning Approach for Emotion Analysis of Occluded Facial Expressions

  • Conference paper
  • First Online:
Recent Trends in Image Processing and Pattern Recognition (RTIP2R 2023)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 2026))

  • 100 Accesses

Abstract

The ability to recognise and interpret emotional expressions is crucial since emotions play a significant role in our daily lives. Emotions are multifaceted phenomena that affect our behavior, perception, and cognition. As a result, numerous machine-learning and deep-learning algorithms for emotion analysis have been studied in previous works. Finding emotion in an obscured face, such as one covered by a scarf or hidden in shadow, is considerably harder than in a complete face, though. This study explores the effectiveness of deep learning models in occluded facial emotion analysis through a transfer learning approach. The performance of two individual pre-trained models, MobileNetV2 and EfficientNetB3, is compared alongside a hybrid model that combines both approaches. This comparison is conducted using the FER-2013 dataset. The dataset consists of 35,887 images and categorizes emotions into seven emotional categories. The results indicate that the hybrid model attained the highest accuracy, with a score of 93.04% for faces occluded at the top and 92.63% for faces occluded at the bottom. Additionally, the study suggests that top-occluded faces displayed more pronounced emotional expressions in comparison to bottom-occluded faces. Overall, these findings imply that hybrid architecture, which was developed as a state-of-the-art model in the study, proves to be effective for analyzing emotions in facial expressions that are partially obscured.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 79.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 99.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Li, X., et al.: EEG based emotion recognition: a tutorial and review. ACM Comput. Surv. 55, 1–57 (2023). https://doi.org/10.1145/3524499

  2. Nita, S., Bitam, S., Heidet, M., Mellouk, A.: A new data augmentation convolutional neural network for human emotion recognition based on ECG signals. Biomed. Sig. Process. Control. 75, 103580 (2022). https://doi.org/10.1016/j.bspc.2022.103580

    Article  Google Scholar 

  3. Wei, Y., et al.: A real-time and two-dimensional emotion recognition system based on EEG and HRV using machine learning. In: 2023 IEEE/SICE International Symposium on System Integration (SII), pp. 1–6. IEEE (2023)

    Google Scholar 

  4. Kipli, K., et al.: GSR signals features extraction for emotion recognition. In: Kaiser, M.S., Bandyopadhyay, A., Ray, K., Singh, R., Nagar, V. (eds.) Proceedings of Trends in Electronics and Health Informatics. LNNS, vol. 376, pp. 329–338. Springer, Singapore (2022). https://doi.org/10.1007/978-981-16-8826-3_28

    Chapter  Google Scholar 

  5. Tamulis, Ž., Vasiljevas, M., Damaševičius, R., Maskeliunas, R., Misra, S.: Affective computing for ehealth using low-cost remote internet of things-based EMG platform. In: Ghosh, U., Chakraborty, C., Garg, L., Srivastava, G. (eds.) Intelligent Internet of Things for Healthcare and Industry. Internet of Things, pp. 67–81. Springer, Cham (2022). https://doi.org/10.1007/978-3-030-81473-1_3

  6. Yang, K., et al.: Mobile emotion recognition via multiple physiological signals using convolution-augmented transformer. In: Proceedings of the 2022 International Conference on Multimedia Retrieval, pp. 562–570. ACM, New York, NY, USA (2022)

    Google Scholar 

  7. Bharathiraja, N., Sakthivel, M., Deepa, T., Hariprasad, S., Ragasudha, N.: Design and implementation of selection algorithm based human emotion recognition system. In: 2023 7th International Conference on Trends in Electronics and Informatics (ICOEI), pp. 1348–1353. IEEE (2023)

    Google Scholar 

  8. Pradhan, A., Srivastava, S.: Hierarchical extreme puzzle learning machine-based emotion recognition using multimodal physiological signals. Biomed. Sig. Process. Control. 83, 104624 (2023). https://doi.org/10.1016/j.bspc.2023.104624

    Article  Google Scholar 

  9. Hies, O., Lewis, M.B.: Beyond the beauty of occlusion: medical masks increase facial attractiveness more than other face coverings. Cogn. Research 7, 1 (2022). https://doi.org/10.1186/s41235-021-00351-9

    Article  Google Scholar 

  10. Pamod, D., Joseph, C., Palanisamy, V., Lekamge, S.: Emotion analysis of occluded facial expressions - a review of literature. In: 2022 ASU International Conference in Emerging Technologies for Sustainability and Intelligent Systems (ICETSIS), pp. 423–429. IEEE (2022). https://doi.org/10.1109/ICETSIS55481.2022.9888947

  11. Ekenel, H.K., Stiefelhagen, R.: Why is facial occlusion a challenging problem? In: Tistarelli, M., Nixon, M.S. (eds.) ICB 2009. LNCS, vol. 5558, pp. 299–308. Springer, Heidelberg (2009). https://doi.org/10.1007/978-3-642-01793-3_31

    Chapter  Google Scholar 

  12. Houshmand, B., Mefraz Khan, N.: Facial expression recognition under partial occlusion from virtual reality headsets based on transfer learning. In: 2020 IEEE Sixth International Conference on Multimedia Big Data (BigMM), pp. 70–75. IEEE (2020). https://doi.org/10.1109/BigMM50055.2020.00020

  13. Feng, X., Pietikäinen, M., Hadid, A.: Facial expression recognition based on local binary patterns. Pattern Recogn. Image Anal. 17, 592–598 (2007). https://doi.org/10.1134/S1054661807040190

    Article  Google Scholar 

  14. Xiao-Xu, Q., Wei, J.: Application of wavelet energy feature in facial expression recogni-tion. In: 2007 International Workshop on Anti-counterfeiting, Security and Identification (ASID), pp. 169–174. IEEE (2007).https://doi.org/10.1109/IWASID.2007.373720

  15. Lee, C.-C., Shih, C.-Y., Lai, W.-P., Lin, P.-C.: An improved boosting algorithm and its AP-plication to facial emotion recognition. J. Ambient. Intell. Humaniz. Comput. 3, 11–17 (2012). https://doi.org/10.1007/s12652-011-0085-8

    Article  Google Scholar 

  16. Chang, C.-Y., Huang, Y.-C.: Personalized facial expression recognition in indoor environ-ments. In: The 2010 International Joint Conference on Neural Networks (IJCNN), pp. 1–8. IEEE (2010). https://doi.org/10.1109/IJCNN.2010.5596316

  17. Alshamsi, H., Meng, H., Li, M.: Real time facial expression recognition app development on mobile phones. In: 2016 12th International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery (ICNC-FSKD), pp. 1750–1755. IEEE (2016).https://doi.org/10.1109/FSKD.2016.7603442

  18. Shan, C., Gong, S., McOwan, P.W.: Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis. Comput. 27, 803–816 (2009).https://doi.org/10.1016/j.imavis.2008.08.005

  19. Pranav, E., Kamal, S., Satheesh Chandran, C., Supriya, M.H.: Facial emotion recognition using deep convolutional neural network. In: 2020 6th International Conference on Advanced Computing and Communication Systems (ICACCS), pp. 317–320. IEEE (2020).https://doi.org/10.1109/ICACCS48705.2020.9074302

  20. Pons, G., Masip, D.: Supervised committee of convolutional neural networks in automated facial expression analysis. IEEE Trans. Affect. Comput. 9, 343–350 (2018).https://doi.org/10.1109/TAFFC.2017.2753235

  21. Ding, H., Zhou, S.K., Chellappa, R.: FaceNet2ExpNet: regularizing a deep face recogni-tion net for expression recognition. In: 2017 12th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2017), pp. 118–126. IEEE (2017). https://doi.org/10.1109/TAFFC.2017.2753235

  22. Li, J., et al.: Facial expression recognition by transfer learning for small datasets. In: Yang, C.-N., Peng, S.-L., Jain, L.C. (eds.) SICBS 2018. AISC, vol. 895, pp. 756–770. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-16946-6_62

    Chapter  Google Scholar 

  23. Pandeya, Y.R., Bhattarai, B., Lee, J.: Deep-learning-based multimodal emotion classification for music videos. Sensors 21, 4927 (2021). https://doi.org/10.3390/s21144927

    Article  Google Scholar 

  24. Shirian, A., Tripathi, S., Guha, T.: Dynamic emotion modeling with learnable graphs and graph inception network. IEEE Trans. Multimed. 24, 780–790 (2022). https://doi.org/10.1109/TMM.2021.3059169

    Article  Google Scholar 

  25. Kosti, R., Alvarez, J., Recasens, A., Lapedriza, A.: Context based emotion recognition using EMOTIC dataset. IEEE Trans. Pattern Anal. Mach. Intell. 1 (2019).https://doi.org/10.1109/TPAMI.2019.2916866

  26. Dutta, S., Ganapathy, S.: Multimodal transformer with learnable frontend and self attention for emotion recognition. In: ICASSP 2022–2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6917–6921. IEEE (2022). https://doi.org/10.1109/TPAMI.2019.2916866

  27. Wei, M., Zheng, W., Zong, Y., Jiang, X., Lu, C., Liu, J.: A novel micro-expression recognition approach using attention-based magnification-adaptive networks. In: ICASSP 2022–2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2420–2424. IEEE (2022). https://doi.org/10.1109/ICASSP43922.2022.9747723

  28. Dhankhar, P.: ResNet-50 and VGG-16 for recognizing Facial Emotions. Int. J. Innov. Eng. Technol. (IJIET)(2019)

    Google Scholar 

  29. Chowdary, M.K., Nguyen, T.N., Hemanth, D.J.: Deep learning-based facial emotion recognition for human-computer interaction applications. Neural Comput. Appl. (2021). https://doi.org/10.1007/s00521-021-06012-8

    Article  Google Scholar 

  30. Ramirez Cornejo, J.Y., Pedrini, H.: Emotion recognition from occluded facial expressions using weber local descriptor. In: 2018 25th International Conference on Systems, Signals and Image Processing (IWSSIP), pp. 1–5. IEEE (2018). https://doi.org/10.1109/ICASSP43922.2022.9747232

  31. Xia, C., Wang, X., Hu, M., Ren, F.: Facial expression recognition under partial occlusion based on fusion of global and local features. In: Yu, H., Dong, J. (eds.) Ninth International Conference on Graphic and Image Processing (ICGIP 2017), pp. 150. SPIE (2018). https://doi.org/10.1117/12.2303417

  32. Li, Y., Zeng, J., Shan, S., Chen, X.: Patch-gated CNN for occlusion-aware facial expression recognition. In: 2018 24th International Conference on Pattern Recognition (ICPR), pp. 2209–2214. IEEE (2018). https://doi.org/10.1109/ICPR.2018.8545853

  33. Mao, X., Wei, C., Qian, Z., Li, M., Fang, X.: Facial expression recognition based on transfer learning from deep convolutional networks. In: 2015 11th International Conference on Natural Computation (ICNC), pp. 702–708. IEEE (2015).https://doi.org/10.1109/ICNC.2015.7378076

  34. Pan, B., Wang, S., Xia, B.: Occluded facial expression recognition enhanced through privileged information. In: Proceedings of the 27th ACM International Conference on Multimedia, pp. 566–573. ACM, New York, NY, USA (2019). https://doi.org/10.1145/3343031.3351049

  35. Song, L., Gong, D., Li, Z., Liu, C., Liu, W.: Occlusion robust face recognition based on mask learning with pairwise differential siamese network. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 773–782. IEEE (2019).https://doi.org/10.1109/ICCV.2019.00086

  36. Shrivastava, H., et al.: Facefetch: an efficient and scalable face retrieval system that uses your visual memory. In: 2019 IEEE Fifth International Conference on Multimedia Big Data (BigMM), pp. 338–347. IEEE (2019). https://doi.org/10.1109/BigMM.2019.00014

  37. Towner, H., Slater, M.: Reconstruction and recognition of occluded facial expressions using PCA. In: Paiva, A.C.R., Prada, R., Picard, R.W. (eds.) ACII 2007. LNCS, vol. 4738, pp. 36–47. Springer, Heidelberg (2007). https://doi.org/10.1007/978-3-540-74889-2_4

    Chapter  Google Scholar 

  38. Zhang, L., Tjondronegoro, D., Chandran, V.: Random Gabor based templates for facial expression recognition in images with facial occlusion. Neurocomputing 145, 451–464 (2014). https://doi.org/10.1016/j.neucom.2014.05.008

    Article  Google Scholar 

  39. Jiang, B., Jia, K.: Research of robust facial expression recognition under facial occlusion condition. In: Zhong, N., Callaghan, V., Ghorbani, A.A., Hu, B. (eds.) AMT 2011. LNCS, vol. 6890, pp. 92–100. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-23620-4_13

    Chapter  Google Scholar 

  40. Kotsia, I., Buciu, I., Pitas, I.: An analysis of facial expression recognition under partial facial image occlusion. Image Vis. Comput. 26, 1052–1067 (2008). https://doi.org/10.1016/j.imavis.2007.11.004

    Article  Google Scholar 

  41. No Title. https://paperswithcode.com/dataset/fer2013

Download references

Author information

Authors and Affiliations

  1. Department of Computing and Information Systems, Sabaragamuwa University of Sri Lanka, Belihuloya, Sri Lanka

    Dilshan Pamod, Ashen Iranga Hewarathna & Sugeeswari Lekamge

  2. Department of Computer Science, Faculty of Engineering, Friedrich-Alexander-University Erlangen-Nuremberg (FAU), Erlangen, Germany

    Joseph Charles

  3. Department of Software Engineering, Sabaragamuwa University of Sri Lanka, Belihuloya, Sri Lanka

    Palanisamy Vigneshwaran

  4. School of Information Technology, Deakin University, Geelong, Australia

    Selvarajah Thuseethan

Authors
  1. Dilshan Pamod
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Joseph Charles
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ashen Iranga Hewarathna
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Palanisamy Vigneshwaran
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Sugeeswari Lekamge
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Selvarajah Thuseethan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Dilshan Pamod .

Editor information

Editors and Affiliations

  1. University of South Dakota, Vermillion, SD, USA

    KC Santosh

  2. University of Derby, Derby, UK

    Aaisha Makkar

  3. University of Derby, Derby, UK

    Myra Conway

  4. Indian Institute of Information Technology, Bhopal, India

    Ashutosh K. Singh

  5. University of Clermont Auvergne, Aubière, France

    Antoine Vacavant

  6. ENSA - Cadi Ayyad University, Marrakesh, Morocco

    Anas Abou el Kalam

  7. Halmstad University, Halmstad, Sweden

    Mohamed-Rafik Bouguelia

  8. Central University of Karnataka, Kalaburagi, India

    Ravindra Hegadi

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Pamod, D., Charles, J., Hewarathna, A.I., Vigneshwaran, P., Lekamge, S., Thuseethan, S. (2024). Hybrid Transfer Learning Approach for Emotion Analysis of Occluded Facial Expressions. In: Santosh, K., et al. Recent Trends in Image Processing and Pattern Recognition. RTIP2R 2023. Communications in Computer and Information Science, vol 2026. Springer, Cham. https://doi.org/10.1007/978-3-031-53082-1_31

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-53082-1_31

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-53081-4

  • Online ISBN: 978-3-031-53082-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation