Skip to main content

Advertisement

Springer Nature Link
Log in

Enhancing facial expression recognition through generative adversarial networks-based augmentation

  • Original Article
  • Published:
International Journal of System Assurance Engineering and Management Aims and scope Submit manuscript

Abstract

Emotion plays a significant role in our daily lives. It can describe the inner feelings and state of an individual and contribute to the communication process. Human–machine interaction is possible as a result of the application of these expressions. Facial expression recognition requires a significant amount of facial images as input data. However, such datasets pose challenges related to image quality and sample imbalance. Since facial expressions exhibit a high degree of diversity, accurately classifying them is a challenging task, particularly for expressions that have fewer samples. Building an efficient and reliable system requires a substantial amount of data. This study aims to address the issue of class imbalance in facial expression datasets by developing and implementing a deep learning-based classification model that uses synthetic images generated through Generative Adversarial Networks. The goal is to improve recognition accuracy for each expression. The effectiveness of the proposed augmentation technique is compared with simple augmentation techniques using VGG16 and the proposed DCNN Model. GAN-based augmentation and the proposed deep learning model outperformed by a large margin on the FER-2013 dataset.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19

Similar content being viewed by others

Explore related subjects

Discover the latest articles and news from researchers in related subjects, suggested using machine learning.

Data availability

Datasets are derived from public resources and made available with the respective articles as mentioned in the literature.

References

  • Agrawal A, Mittal N (2020) Using cnn for facial expression recognition: a study of the effects of kernel size and number of filters on accuracy. Vis Comput 36:2020

    Article  Google Scholar 

  • Arora T, Soni R (2021) A review of techniques to detect the GAN-generated fake images. Generative Adversarial Networks for Image-to-Image Translation, 125–159

  • Breuer R, Kimmel R (2017) A deep learning perspective on the origin of facial expressions. arxiv. preprint 2017

  • Brunet PM, Cowie R (2012) Towards a conceptual framework of research on social signal processing. J Multimodal User Interfaces 6:2012

    Google Scholar 

  • Cai J, Meng Z, Khan AS, O’Reilly J, Li Z, Han S, Tong Y (2021) In Identity-free facial expression recognition using conditional generative adversarial network, ICIP, 2021. In: 2021 IEEE International Conference on Image Processing, pp 1344–1348

  • Ekman P, Friesen WV (1971) Constants across cultures in the face and emotion. J Pers Soc Psychol 17:1971

    Article  Google Scholar 

  • Eleyan A (2017) Comparative study on facial expression recognition using gabor and dual-tree complex wavelet transforms. Int J Eng Appl Sci 2017:1–13

    Google Scholar 

  • Gao F, Yang Y, Wang J, Sun J, Yang E, Zhou H (2018) A deep convolutional generative adversarial networks (dcgans)-based semi-supervised method for object recognition in synthetic aperture radar (sar) images. Remote Sensing 10:2018

    Article  Google Scholar 

  • Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. Adv Neural Inf Process Syst 27:2014

    Google Scholar 

  • Gupta S, Kumar P, Tekchandani RK (2023) Facial emotion recognition based real-time learner engagement detection system in online learning context using deep learning models. Multimedia Tools Appl 82(8):11365–11394

    Article  Google Scholar 

  • Hasani B, Mahoor MH (2017) Facial expression recognition using enhanced deep 3d convolutional neural networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 30–40. 2017

  • Hassan T, Seuß D, Wollenberg J, Weitz K, Kunz M, Lautenbacher S, Schmid U (2019) Automatic detection of pain from facial expressions: a survey. IEEE Trans Pattern Anal Mach Intell 43:2019

    Google Scholar 

  • Hossain MS, Muhammad G (2016) Audio-visual emotion recognition using multi-directional regression and ridgelet transform. J Multimodal User Interfaces 10(4):2016

    Article  Google Scholar 

  • IqbalQuraishi M, Choudhury J, De M, Chakraborty P (2012) A framework for the recognition of human emotion using soft computing models. Int J Comput Appl 40:2012

    Google Scholar 

  • Ivanko D, Karpov A, Fedotov D, Kipyatkova I, Ryumin D, Ivanko D, Zelezny M (2018) Multimodal speech recognition: increasing accuracy using high-speed video data 12:319–328

  • Kaggle (2013) FER2013: facial expression recognition challenge. Retrieved from https://www.kaggle.com/c/challenges-in-representation-learning-facial-expression-recognition-challenge/data

  • Karpov A, Mporas I (2018) Speech communication integrated with other modalities. J Multimodal User Interfaces 12:2018

    Article  Google Scholar 

  • Krishna R (2020) Real-time facial expression recognition using cnn. Int J Adv Res Ideas Innov Technol 2020:576–580

    Google Scholar 

  • Li S, Deng W (2020) Deep facial expression recognition: a survey. IEEE Trans Affect Comput 2020:1–20

    Google Scholar 

  • Li J, Jin K, Zhou D, Kubota N, Ju Z (2020) Attention mechanism-based cnn for facial expression recognition. Neurocomputing 411:340–350

    Article  Google Scholar 

  • Li K, Jin Y, Akram MW, Han R, Chen J (2020) Facial expression recognition with convolutional neural networks via a new face cropping and rotation strategy. Vis Comput 362:2020

    Google Scholar 

  • Ma H, Celik T (2019) Fer-net: facial expression recognition using densely connected convolutional network. Electron Lett 55:2019

    Article  Google Scholar 

  • Mahajan S, Rani R (2021) Text detection and localization in scene images: a broad review. Artif Intell Rev 54(6):2021

    Article  Google Scholar 

  • Mahajan S, Rani R (2022) Word level script identification using convolutional neural network enhancement for scenic images. Trans Asian Low-Resour Lang Inf Process 21:2022

    Google Scholar 

  • Mayya V, Pai RM, Pai MM (2016) Automatic facial expression recognition using dcnn. Procedia Comput Sci 93:2016

    Article  Google Scholar 

  • Mozaffari L, Brekke MM, Gajaruban B, Purba D, Zhang J (2023) Facial expression recognition using deep neural network. In: 2023 3rd International Conference on applied artificial intelligence (ICAPAI), pp 1–9. IEEE

  • Porcu S, Floris A, Atzori L (2020) Evaluation of data augmentation techniques for facial expression recognition systems. Electronics 9:2020

    Article  Google Scholar 

  • Punuri SB, Kuanar SK, Kolhar M, Mishra TK, Alameen A, Mohapatra H, Mishra SR (2023) Efficient net-XGBoost: an implementation for facial emotion recognition using transfer learning. Mathematics 11(3):776

    Article  Google Scholar 

  • Salman FZ, Madani A (2018) Emotion recognition from facial expression based on fiducial points detection and using neural network. Int J Electr Comput Eng 2018:52–59

  • Schuller BW, Zhang Y, Weninger F (2018) Three recent trends in paralinguistics on the way to omniscient machine intelligence. J Multimodal User Interfaces 12:2018

    Article  Google Scholar 

  • Singh S, Nasoz F (2020) facial expression recognition with convolutional neural networks. 2020(10):324–0328

  • Soni R, Arora T (2021) A review of the techniques of images using GAN. Generative Adversarial Networks for Image-to-Image Translation, pp 99–123

  • Tian C, Ma Y, Cammon J, Fang F, Zhang Y, Meng M (2023) Dual-encoder VAE-GAN with spatiotemporal features for emotional EEG data augmentation. IEEE Trans Neural Syst Rehabil Eng

  • Vepuri KS (2021) Improving facial emotion recognition with image processing and deep learning. Master’s Projects

  • Verma V, Rani R (2021) Recognition of facial expressions using a deep neural network. In: 2021 8th International conference on signal processing and integrated networks (SPIN), pp 585–590. IEEE

  • Wang K, Peng X, Yang J, Meng D, Qiao Y (2020) Region attention networks for pose and occlusion robust facial expression recognition. IEEE Trans Image Process 29:2020

    Google Scholar 

  • Yang L, Tian Y, Song Y, Yang N, Ma K, Xie L (2020) A novel feature separation model exchange-gan for facial expression recognition. Knowl-Based Syst 204:2020

    Article  Google Scholar 

  • Yang D, Alsadoon A (2018) An emotion recognition model based on facial recognition in virtual learning environment. In: International conference on smart computing and communications, pp 4–10, 2018. ICSCC

  • Yang H, Ciftci U, Yin L (2018) Facial expression recognition by de-expression residue learning. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2168–2177

  • Yin L, Wei X, Sun Y, Wang J, Rosato MJ (2006) April. In A 3D facial expression database for facial behavior research. In: 7th international conference on automatic face and gesture recognition IEEE, pp 211–216

  • Zeng J, Shan S, Chen X (2018a) Facial expression recognition with inconsistently annotated datasets. In: Proceedings of the European conference on computer vision, pp 222–237

  • Zeng J, Shan S, Chen X (2018b) Facial expression recognition with inconsistently annotated datasets. In: Proceedings of the European conference on computer vision, pp 222–237

  • Zhang F, Zhang T, Mao Q, Xu C (2018) Joint pose and expression modeling for facial expression recognition. In: 2018 IEEE/CVF conference on computer vision and pattern recognition, pp 3359–3368

Download references

Funding

This research received no external funding.

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, NIT, Jalandhar, Punjab, 144011, India

    Rajneesh Rani, Shefali Arora, Vipan Verma & Shilpa Mahajan

  2. Department of MIS and Analytics, International Management Institute, Kolkata, West Bengal, India

    Ruchi Sharma

Authors
  1. Rajneesh Rani
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Shefali Arora
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Vipan Verma
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Shilpa Mahajan
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Ruchi Sharma
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Rajneesh Rani.

Ethics declarations

Conflict of interest

All authors certify that they have no conflict of interest, affiliations with, or involvement in any organization or entity with any financial interest or non-financial interest in the subject matter or materials discussed in this manuscript.

Research involving human participants and/or animals

All authors certify that no research has been conducted on human participants or animals. Dataset has been used for the same.

Informed consent

All authors certify that informed consent has been made prior to submission.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Rani, R., Arora, S., Verma, V. et al. Enhancing facial expression recognition through generative adversarial networks-based augmentation. Int J Syst Assur Eng Manag 15, 1037–1056 (2024). https://doi.org/10.1007/s13198-023-02186-7

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13198-023-02186-7

Keywords