Skip to main content
SpringerLink
Log in

Deep Hybrid Neural Networks for Facial Expression Classification

  • Conference paper
  • First Online:
Book cover Computer Vision and Image Processing (CVIP 2019)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1148))

Included in the following conference series:

Abstract

Facial Expression evaluation has become necessary for human machine interaction, behavior analysis and also forensic and clinical evaluation. Deep convolutional neural networks (CNN) have been largely used for facial expression recognition but due to locality of convolution, CNNs results in lower accuracy when trained with facial expression data of varying ethnicity and emotion intensity. Recurrent neural networks (RNN) are used to work with sequential data and used to predict the sequences. We propose CNN-RNN network approach, a hybrid network, wherein the outputs from CNN and RNN have been concatenated to predict the final emotion, similarly a CNN model followed by a RNN layers has been designed that gives promising results. The proposed hybrid models are evaluated on two publically available datasets CK+ and JAFFE which provide us with variation in ethnicity and emotion intensity. Promising results have been obtained with this hybrid approach when compared to various machine learning and deep learning methods.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z., Matthews, I.: The extended Cohn-Kanade dataset (CK+): a complete dataset for action unit and emotion-specified expression. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 94–101. IEEE (2010)

    Google Scholar 

  2. Lyons, M.J., Akamatsu, S., Kamachi, M., Gyoba, J., Budynek, J.: The Japanese female facial expression (JAFFE) database (1998)

    Google Scholar 

  3. Ekman, P.: Pictures of Facial Affect. Consulting Psychologists Press, USA (1976)

    Google Scholar 

  4. Ekman, P.: Facial expression and emotion. Am. Psychol. 48(4), 384 (1993)

    Article  Google Scholar 

  5. Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: International Conference on Computer Vision and Pattern Recognition (CVPR 2005), San Diego, United States, pp. 886–893 (June 2005)

    Google Scholar 

  6. Shan, C., Gong, S., McOwan, P.W.: Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis. Comput. 27(6), 803–816 (2009)

    Article  Google Scholar 

  7. Cootes, T.F., Edwards, G.J., Taylor, C.J.: Active appearance models. In: Burkhardt, H., Neumann, B. (eds.) ECCV 1998. LNCS, vol. 1407, pp. 484–498. Springer, Heidelberg (1998). https://doi.org/10.1007/BFb0054760

    Chapter  Google Scholar 

  8. Ou, J., Bai, X.B., Pei, Y., Ma, L., Liu, W.: Automatic facial expression recognition using Gabor filter and expression analysis. In: IEEE International Conference on Computer Modeling and Simulation, pp. 215–218 (2010)

    Google Scholar 

  9. Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1097–1105 (2012)

    Google Scholar 

  10. Szegedy, C., et al.: Going deeper with convolutions. In: CVPR (2015)

    Google Scholar 

  11. Donahue, J., et al.: Long-term recurrent convolutional networks for visual recognition and description. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2625–2634 (2015)

    Google Scholar 

  12. Liang, M., Hu, X.: Recurrent convolutional neural network for object recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3367–3375 (2015)

    Google Scholar 

  13. Wang, J., Yi, Y., Mao, J., Huang, Z., Huang, C., Xu, W.: CNN-RNN: a unified framework for multi-label image classification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2285–2294 (2016)

    Google Scholar 

  14. Zuo, Z., et al.: Convolutional recurrent neural networks: learning spatial dependencies for image representation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 18–26 (2015)

    Google Scholar 

  15. Khorrami, P., Paine, T.L., Brady, K., Dagli, C., Huang, T.S.: How deep neural networks can improve emotion recognition on video data. In: IEEE Conference on Image Processing (ICIP) (2016)

    Google Scholar 

  16. Visin, F., Kastner, K., Cho, K., Matteucci, M., et al.: ReNet: a recurrent neural network based alternative to convolutional networks. arXiv:1505.00393 (2015)

  17. Graves, A., Mohamed, A.R., Hinton, G.: Speech recognition with deep recurrent neural networks. In: Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 6645–6649 (2013)

    Google Scholar 

  18. Sanin, A., Sanderson, C., Harandi, M.T., Lovell, B.C.: Spatiotemporal covariance descriptors for action and gesture recognition. In: IEEE Workshop on Applications of Computer Vision (2013)

    Google Scholar 

  19. Jain, S., Hu, C., Aggarwal, J.K.: Facial expression recognition with temporal modeling of shapes. In: Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 1642–1649 (2011)

    Google Scholar 

  20. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)

  21. Connie, T., Al-Shabi, M., Cheah, W.P., Goh, M.: Facial expression recognition using a hybrid CNN–SIFT aggregator. In: Phon-Amnuaisuk, S., Ang, S.-P., Lee, S.-Y. (eds.) MIWAI 2017. LNCS (LNAI), vol. 10607, pp. 139–149. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-69456-6_12

    Chapter  Google Scholar 

  22. Kaya, H., Gürpınar, F., Salah, A.A.: Video-based emotion recognition in the wild using deep transfer learning and score fusion. Image Vis. Comput. 65, 66–75 (2017)

    Article  Google Scholar 

  23. Lowe, D.G.: Distinctive image features from scale-invariant key points. Int. J. Comput. Vis. 60(2), 91–110 (2004)

    Article  Google Scholar 

  24. Le, Q.V., Jaitly, N., Hinton, G.E.: A simple way to initialize recurrent networks of rectified linear units. arXiv:1504.00941 (2015)

  25. Minaee, S., Abdolrashidi, A.: Deep-emotion: facial expression recognition using attentional convolutional network. arXiv preprint arXiv:1902.01019 (2019)

  26. Shan, K., Guo, J., You, W., Lu, D., Bie, R.: Automatic facial expression recognition based on a deep convolutional-neural-network structure. In: 2017 IEEE 15th International Conference on Software Engineering Research Management and Applications (SERA), pp. 123–128 (2017)

    Google Scholar 

  27. Kim, J.-H., Kim, B.-G., Roy, P.P., Jeong, D.-M.: Efficient facial expression recognition algorithm based on hierarchical deep neural network structure. IEEE Access 7, 41273–41285 (2019)

    Article  Google Scholar 

  28. Jain, N., et al.: Hybrid deep neural networks for face emotion recognition. Pattern Recogn. Lett. 115, 101–106 (2018)

    Article  Google Scholar 

  29. Visin, F., Kastner, K., Courville, A., Bengio, Y., et al.: ReSeg: a recurrent neural network for object segmentation arXiv:1511.07053 (2015)

  30. Patil, P., Murala, S.: FgGAN: a cascaded unpaired learning for background estimation and foreground segmentation. In: 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1770–1778. IEEE (2019)

    Google Scholar 

  31. Patil, P., Murala, S., Dhall, A., Chaudhary, S.: MsEDNet: multi-scale deep saliency learning for moving object detection. In: 2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pp. 1670–1675. IEEE (2018)

    Google Scholar 

  32. Patil, P.W., Murala, S.: MSFgNet: a novel compact end-to-end deep network for moving object detection. IEEE Trans. Intell. Transp. Syst. 20(11), 4066–4077 (2018)

    Article  Google Scholar 

  33. Dudhane, A., Murala, S.: CDNet: single image de-hazing using unpaired adversarial training. In: 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1147–1155. IEEE (2019)

    Google Scholar 

  34. Dudhane, A., Murala, S.: C^2MSNet: a novel approach for single image haze removal. In: 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1397–1404. IEEE (2018)

    Google Scholar 

  35. Choi, Y., Choi, M., Kim, M., Ha, J.-W., Kim, S., Choo, J.: StarGAN: unified generative adversarial networks for multi-domain image-to-image translation. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8789–8797 (2018)

    Google Scholar 

  36. Zhu, J.-Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: The IEEE International Conference on Computer Vision (ICCV), pp. 2223–2232 (2017)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Shri Guru Gobind Singhji Institute of Engineering and Technology, Nanded, India

    Aakash Babasaheb Jadhav, Sairaj Laxman Burewar, Ajay Ashokrao Waghumbare & Anil Balaji Gonde

Authors
  1. Aakash Babasaheb Jadhav
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sairaj Laxman Burewar
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ajay Ashokrao Waghumbare
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Anil Balaji Gonde
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Aakash Babasaheb Jadhav .

Editor information

Editors and Affiliations

  1. Malaviya National Institute of Technology, Jaipur, Rajasthan, India

    Neeta Nain

  2. Malaviya National Institute of Technology, Jaipur, Rajasthan, India

    Santosh Kumar Vipparthi

  3. Indian Institute of Technology Roorkee, Roorkee, Uttarakhand, India

    Balasubramanian Raman

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Jadhav, A.B., Burewar, S.L., Waghumbare, A.A., Gonde, A.B. (2020). Deep Hybrid Neural Networks for Facial Expression Classification. In: Nain, N., Vipparthi, S., Raman, B. (eds) Computer Vision and Image Processing. CVIP 2019. Communications in Computer and Information Science, vol 1148. Springer, Singapore. https://doi.org/10.1007/978-981-15-4018-9_26

Download citation

  • DOI: https://doi.org/10.1007/978-981-15-4018-9_26

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-15-4017-2

  • Online ISBN: 978-981-15-4018-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation