Skip to main content
SpringerLink
Log in

Context-Aware Facial Expression Recognition Using Deep Convolutional Neural Network Architecture

  • Conference paper
  • First Online:
Intelligent Human Computer Interaction (IHCI 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14531))

Included in the following conference series:

  • 89 Accesses

Abstract

A frame of reference, which includes additional contextual information, can provide a more accurate and comprehensive understanding of the individual’s emotional state. This context might encompass factors such as the person’s surroundings, body language, gestures, tone of voice, and the specific situation or events taking place. Previous research in this field has often struggled to recognize emotions within a contextual framework. However, by considering contextual elements in addition to facial expressions, we can gain a more nuanced and precise picture of the individual’s emotions. In this paper, we used both context-aware datasets (Emotic, CAER, and CAER-S) and only the facial emotion datasets (Affectnet and AEFW) to signify the context. In this Emotic dataset images are labeled with 26 emotional categories. We utilized these datasets to build a convolutional neural network model that effectively examines both the individual and the overall scenario to accurately identify a wide range of information pertaining to emotional states. The features obtained from these two modules are combined using a specialized fusion network. Through this approach, we demonstrate the significance of emotion recognition within a visual context.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 59.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 79.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Ekman, P.: Cross-cultural Studies of Facial Expression. Darwin and Facial Expression, pp. 169–220. Malor Books, Los Altos (2006)

    Google Scholar 

  2. Ekman, P., Friesen, W.V.: Constants across cultures in the face and emotion. J. Pers. Soc. Psychol. 17, 124–129 (1971)

    Article  Google Scholar 

  3. Fridlund, A.J.: Human facial expression: an evolutionary view. Nature 373, 569 (1995)

    Google Scholar 

  4. Kosti, R., Alvarez, J.M., Recasens, A., Lapedriza, A.: Emotion recognition in context. In: CVPR (2017)

    Google Scholar 

  5. Lee, J., Kim, S., Kim, S., Park, J., Sohn, K.: Context-aware emotion recognition networks. IEEE explore (2019)

    Google Scholar 

  6. Soleymani, M Pantic, M.; Pun, T. Multimodal Emotion Recognition in Response to Videos. IEEE Trans. Affect. Comput. 2012,3, 211–223

    Google Scholar 

  7. Noroozi, F., Marjanovic, M., Njegus, A., Escalera, S., Anbarjafari, G.: Audio-visual emotion recognition in video clips. IEEE Trans. Affect. Comput. 10, 60–75 (2019)

    Article  Google Scholar 

  8. Li, X., Peng, X., Ding, C.: Sequential interactive biased network for context-aware emotion recognition. In: 2021 IEEE International Joint Conference on Biometrics (IJCB), pp. 1–6. IEEE (2021)

    Google Scholar 

  9. Zhang, D., et al.: Multi-modal multi-label emotion recognition with heterogeneous hierarchical message passing. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, no. 16, pp. 14338–14346 (2021)

    Google Scholar 

  10. Li, Y., Lu, G., Li, J., Zhang, Z., Zhang, D.: Facial expression recognition in the wild using multi-level features and attention mechanisms. IEEE Trans. Affect. Comput. (2020)

    Google Scholar 

  11. Caon, M., Angelini, L., Yue, Y., Khaled, O.A., Mugellini, E.: Context-aware multimodal sharing of emotions. In: Kurosu, M. (ed.) HCI 2013. LNCS, vol. 8008, pp. 19–28. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-39342-6_3

    Chapter  Google Scholar 

  12. Wu, S., Zhou, L., Hu, Z., Liu, J.: Hierarchical context-based emotion recognition with scene graphs. IEEE Trans. Neural Netw. Learn. Syst. (2022)

    Google Scholar 

  13. Mehendale, N.: Facial emotion recognition using convolutional neural networks (FERC). SN Appl. Sci. 2(3), 1–8 (2020). https://doi.org/10.1007/s42452-020-2234-1

    Article  Google Scholar 

  14. Mohan, K., Seal, A., Krejcar, O., Yazidi, A.: FER-net: facial expression recognition using deep neural net. Neural Comput. Appl. 33, 9125–9136 (2021)

    Article  Google Scholar 

  15. Johannßen, D., Biemann, C.: Neural classification with attention assessment of the implicit-association test OMT and prediction of subsequent academic success. In: KONVENS (2019)

    Google Scholar 

  16. Shenoy, A., Sardana, A.: Multilogue-net: a context aware RNN for multi-modal emotion detection and sentiment analysis in conversation (2020). arXiv preprint arXiv:2002.08267

  17. Bendjillali, R.I., Beladgham, M., Merit, K., Taleb-Ahmed, A.: Improved facial expression recognition based on DWT feature for deep CNN. Electronics 8, 324 (2019)

    Article  Google Scholar 

  18. Deng, J., Dong, W., Socher, R., Li, L., Li, K., Fei-Fei, L.: Imagenet: a large-scale hierarchical image database. In CVPR (2009)

    Google Scholar 

  19. B. Zhou, A. Khosla, A. Lapedriza, A. Torralba, and A. Oliva. Places: An image database for deep scene understanding. CoRR, abs/1610.02055, 2015

    Google Scholar 

  20. Ismatov, A., Enriquez, V.G., Singh, M.: FaceHub: facial recognition data management in blockchain. In: Lee, S.-W., Singh, I., Mohammadian, M. (eds.) Blockchain Technology for IoT Applications. BT, pp. 135–153. Springer, Singapore (2021). https://doi.org/10.1007/978-981-33-4122-7_7

    Chapter  Google Scholar 

  21. Lin, T., etal.: Microsoft COCO: common objects in context. CoRR abs/1405.0312 (2014)

    Google Scholar 

  22. Zhou, B., Zhao, H., Puig, X., Fidler, S., Barriuso, A., Torralba, A.: Semantic understanding of scenes through ade20k dataset (2016)

    Google Scholar 

  23. Prinz, J.: Which emotions are basic. Emot. Evol. Rational. 69, 88 (2004)

    Google Scholar 

  24. Mollahosseini, A., Hasani, B., Mahoor, M.H.: Affectnet: a database for facial expression, valence, and arousal computing in the wild. IEEE Trans. Affect. Comput. 10(1), 18–31 (2017)

    Article  Google Scholar 

  25. Dhall, A., Goecke, R., Lucey, S., Gedeon, T., et al.: Collecting large, richly annotated facial-expression databases from movies. IEEE Multimedia (2012)

    Google Scholar 

  26. Jiang, Y.G., Xu, B., Xue, X.: Predicting emotions in user-generated videos. In: AAAI (2014)

    Google Scholar 

  27. Tran, D., Bourdev, L., Fergus, R., Torresani, L., Paluri, M.: Learning spatiotemporal features with 3d convolutional networks. In: ICCV (2015)

    Google Scholar 

  28. You, Q., Jin, H., Luo, J.: Visual sentiment analysis by attending on local image regions. In: AAAI (2017)

    Google Scholar 

  29. Ko, B.C.: A brief review of facial emotion recognition based on visual information. Sensors 18, 401 (2018)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science, Banasthali Vidyapith, Radha Kishnpura, Rajasthan, 304022, India

    Abha Jain, Swati Nigam & Rajiv Singh

  2. Centre for Artificial Intelligence, Banasthali Vidyapith, Radha Kishnpura, Rajasthan, 304022, India

    Swati Nigam & Rajiv Singh

Authors
  1. Abha Jain
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Swati Nigam
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Rajiv Singh
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Swati Nigam .

Editor information

Editors and Affiliations

  1. Soongsil University, Seoul, Korea (Republic of)

    Bong Jun Choi

  2. Saint Louis University, St. Louis, MO, USA

    Dhananjay Singh

  3. Indian Institute of Information Technology, Allahabad, India

    Uma Shanker Tiwary

  4. Pukyong National University, Busan, Korea (Republic of)

    Wan-Young Chung

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Jain, A., Nigam, S., Singh, R. (2024). Context-Aware Facial Expression Recognition Using Deep Convolutional Neural Network Architecture. In: Choi, B.J., Singh, D., Tiwary, U.S., Chung, WY. (eds) Intelligent Human Computer Interaction. IHCI 2023. Lecture Notes in Computer Science, vol 14531. Springer, Cham. https://doi.org/10.1007/978-3-031-53827-8_13

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-53827-8_13

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-53826-1

  • Online ISBN: 978-3-031-53827-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation