Skip to main content

Advertisement

Log in

PRATIT: a CNN-based emotion recognition system using histogram equalization and data augmentation

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Emotions are spontaneous feelings that are accompanied by fluctuations in facial muscles, which leads to facial expressions. Categorization of these facial expressions as one of the seven basic emotions - happy, sad, anger, disgust, fear, surprise, and neutral is the intention behind Emotion Recognition. This is a difficult problem because of the complexity of human expressions, but is gaining immense popularity due to its vast number of applications such as predicting behavior. Using deeper architectures has enabled researchers to achieve state-of-the-art performance in emotion recognition. Motivated from the aforementioned discussion, in this paper, we propose a model named as PRATIT, used for facial expression recognition that uses specific image preprocessing steps and a Convolutional Neural Network (CNN) model. In PRATIT, preprocessing techniques such as grayscaling, cropping, resizing, and histogram equalization have been used to handle variations in the images. CNNs accomplish better accuracy with larger datasets, but there are no freely accessible datasets with adequate information for emotion recognition with deep architectures. Therefore, to handle the aforementioned issue, we have applied data augmentation in PRATIT, which helps in further fine tuning the model for performance improvement. The paper presents the effects of histogram equalization and data augmentation on the performance of the model. PRATIT with the usage of histogram equalization during image preprocessing and data augmentation surpasses the state-of-the-art results and achieves a testing accuracy of 78.52%.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Alghowinem S, Goecke R, Wagner M, Epps J, Breakspear M, Parker G (2012) From joyous to clinically depressed: mood detection using spontaneous speech. In: Proceedings of the twenty-fifth international Florida artificial intelligence research society conference, pp 141–146

  2. Alghowinem S, Goecke R, Cohn JF, Wagner M, Parker G, Breakspear M (2015) Cross-cultural detection of depression from nonverbal behaviour. In: 2015 11th IEEE International conference and workshops on automatic face and gesture recognition (FG), vol 1. IEEE, pp 1–8

  3. Ali G, Iqbal MA, Choi TS (2016) Boosted nne collections for multicultural facial expression recognition. Pattern Recogn 55:14–27

    Article  Google Scholar 

  4. Alzubi JA, Bharathikannan B, Tanwar S, Manikandan R, Khanna A, Thaventhiran C (2019) Boosted neural network ensemble classification for lung cancer disease diagnosis. Appl Soft Comput 80:579–591

    Article  Google Scholar 

  5. Anjum MA, Javed MY (2005) Face recognition vs image resolution. In: Information and communication technologies. IEEE, pp 109–112

  6. Bhatia S, Hayat M, Breakspear M, Parker G, Goecke R (2017) A video-based facial behaviour analysis approach to melancholia. In: Automatic face & gesture recognition. IEEE, pp 754–761

  7. Bista U (2018) USPS Dataset. http://www-i6.informatik.rwthaachen.de/keysers/usps.html, last Accessed: 2015-10-28

  8. Cao C, Hou Q, Zhou K (2014) Displaced dynamic expression regression for real-time facial tracking and animation. ACM Trans Graph (TOG) 33(4):43

    Google Scholar 

  9. Chang Cc, Lin Cj (2001) Libsvm: a library for support vector machines. Software available at http://www.csie.ntu.edu.tw/cjlin/libsvm

  10. Chen Y, Yang J, Qian J (2017) Recurrent neural network for facial landmark detection. Neurocomputing 219:26–38

    Article  Google Scholar 

  11. Chrysos GG, Antonakos E, Zafeiriou S, Snape P (2015) Offline deformable face tracking in arbitrary videos. In: Proceedings of the IEEE international conference on computer vision workshops, pp 1–9

  12. Chuang CF, Shih FY (2006) Recognizing facial action units using independent component analysis and support vector machine. Pattern Recogn 39(9):1795–1798

    Article  Google Scholar 

  13. De Silva LC, Miyasato T, Nakatsu R (1997) Facial emotion recognition using multi-modal information. In: Information, communications and signal processing, vol 1. IEEE, pp 397–401

  14. De Silva CR, Ranganath S, De Silva LC (2008) Cloud basis function neural network: a modified rbf network architecture for holistic facial expression recognition. Pattern Recogn 41(4):1241–1253

    Article  Google Scholar 

  15. Dhall A, Goecke R, Lucey S, Gedeon T (2011) Acted facial expressions in the wild database. Australian National University, Canberra, Australia. Technical Report TR-CS-11, 2:1

  16. Dhall A, Goecke R, Lucey S, Gedeon T (2011) Static facial expression analysis in tough conditions: data, evaluation protocol and benchmark. Computer vision workshops (ICCV Workshops). IEEE, pp 2106–2112

  17. Giannakakis G, Pediaditis M, Manousos D, Kazantzaki E, Chiarugi F, Simos PG, Marias K, Tsiknakis M (2017) Stress and anxiety detection using facial cues from videos. Biomed Signal Process Control 31:89–101

    Article  Google Scholar 

  18. Gross R, Matthews I, Cohn E, Kanade T, Baker S (2008) The CMU multi-PIE face database. http://www.cs.cmu.edu/afs/cs/project/PIE/MultiPie/Multi-Pie/Home.html, last Accessed: 2015-10-28

  19. Guo Y, Tao D, Yu J, Xiong H, Li Y, Tao D (2016) Deep neural networks with relativity learning for facial expression recognition. Multimedia & expo workshops (ICMEW). IEEE, pp 1–6

  20. Jain N, Kumar S, Kumar A, Shamsolmoali P, Zareapoor M (2018) Hybrid deep neural networks for face emotion recognition. Pattern Recogn Lett 115:101–106

    Article  Google Scholar 

  21. Jeni LA, Cohn JF, De La Torre F (2013) Facing imbalanced data–recommendations for the use of performance metrics, IEEE

  22. Kaya H, Gurpinar F, Salah AA (2017) Video-based emotion recognition in the wild using deep transfer learning and score fusion. Image Vis Comput 65:66–75

    Article  Google Scholar 

  23. Khorrami P, Paine T, Huang T (2015) Do deep neural networks learn facial action units when doing expression recognition?. In: Proceedings of the IEEE international conference on computer vision workshops, pp 19–27

  24. Kumar GR, Kumar RK, Sanyal G (2017) Facial emotion analysis using deep convolution neural network, IEEE

  25. Laboratory BM (2008) TFEID:Taiwanese facial expression image database. http://bml.ym.edu.tw/tfeid/, last Accessed: 2015-10-28

  26. Labs L (2017) What emotion recognition is: you may not even notice its importance. https://medium.com/@looxid.labs/what-emotion-recognition-is-you-may-not-even-notice-its-importance-d1edd0a4250a, last Accessed: 2018-08-05

  27. Levi G, Hassner T (2015) Emotion recognition in the wild via convolutional neural networks and mapped binary patterns. In: Proceedings of the 2015 ACM on international conference on multimodal interaction. ACM, pp 503–510

  28. Lopes AT, de Aguiar E, De Souza AF, Oliveira-Santos T (2017) Facial expression recognition with convolutional neural networks: coping with few data and the training sample order. Pattern Recogn 61:610–628

    Article  Google Scholar 

  29. Lopes AT, de Aguiar E, Souza AFD, Oliveira-Santos T (2017) Facial expression recognition with convolutional neural networks: coping with few data and the training sample order. Pattern Recogn 61:610–628

    Article  Google Scholar 

  30. Lyons MJ, Budynek J, Akamatsu S (1999) Automatic classification of single facial images. IEEE Trans Pattern Anal Mach Intell 21(12):1357–1362

    Article  Google Scholar 

  31. Mollahosseini A, Chan D, Mahoor MH (2016) Going deeper in facial expression recognition using deep neural networks. In: Applications of computer vision (WACV). IEEE, pp 1–10

  32. Ng HW, Nguyen VD, Vonikakis V, Winkler S (2015) Deep learning for emotion recognition on small datasets using transfer learning. In: Proceedings of the 2015 ACM on international conference on multimodal interaction. ACM, pp 443–449

  33. Nicolle J, Bailly K, Chetouani M (2016) Real-time facial action unit intensity prediction with regularized metric learning. Image Vis Comput 52:1–14

    Article  Google Scholar 

  34. Ojeme B, Mbogho A (2016) Selecting learning algorithms for simultaneous identification of depression and comorbid disorders. Procedia Comput Sci 96:1294–1303

    Article  Google Scholar 

  35. Patii J, Patei R, Kothiya S (2017) Human mental states recognition under face occlusion, IEEE

  36. Pierre-Luc Carrier AC (2013) Challenges in representation learning: facial expression recognition challenge—kaggle. https://www.kaggle.com/c/challenges-in-representation-learning-facial-expression-recognition-challenge/data https://www.kaggle.com/c/challenges-in-representation-learning-facial-expression-recognition-challenge/data, last Accessed: 2018-04-03

  37. Shankar K, Lakshmanaprabu SK, Tanwar S, Rodrigues JJ, Roy NR (2019) Alzheimer detection using group grey wolf optimization based features with convolutional classifier. Comput Electr Eng 77:230–243

    Article  Google Scholar 

  38. Shen J, Zafeiriou S, Chrysos GG, Kossaifi J, Tzimiropoulos G, Pantic M (2015) The first facial landmark tracking in-the-wild challenge: benchmark and results. In: Proceedings of the IEEE international conference on computer vision workshops, pp 50–58

  39. Simard PY, Steinkraus D, Platt JC, et al. (2003) Best practices for convolutional neural networks applied to visual document analysis. In: ICDAR, vol 3, pp 958–962

  40. Tzimiropoulos G (2015) Project-out cascaded regression with an application to face alignment. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3659–3667

  41. University R (2017) RadBoud dataset. http://www.socsci.ru.nl:8180/RaFD2/RaFD?p=main, last Accessed: 2015-10-28

  42. Valstar M, Schuller B, Smith K, Eyben F, Jiang B, Bilakhia S, Schnieder S, Cowie R, Pantic M (2013) Avec 2013: the continuous audio/visual emotion and depression recognition challenge. In: Proceedings of the 3rd ACM international workshop on audio/visual emotion challenge. ACM, pp 3–10

  43. Viola P, Jones M (2001) Rapid object detection using a boosted cascade of simple features. In: Computer vision and pattern recognition, vol 1. IEEE, pp I–I

  44. Wang Z, Qu Z (2017) Research on web text classification algorithm based on improved cnn and svm. In: Communication technology (ICCT). IEEE, pp 1958–1961

  45. Wang SH, Phillips P, Dong ZC, Zhang YD (2018) Intelligent facial emotion recognition based on stationary wavelet entropy and jaya algorithm. Neurocomputing 272:668–676

    Article  Google Scholar 

  46. Xie L, Shen J, Han J, Zhu L, Shao L (2017) Dynamic multi-view hashing for online image retrieval. In: Proceedings of the 26th international joint conference on artificial intelligence IJCAI’17. AAAI Press, pp 3133–3139

  47. Yang Y, Fairbairn C, Cohn JF (2013) Detecting depression severity from vocal prosody. IEEE Trans Affect Comput 4(2):142–150

    Article  Google Scholar 

  48. Yang D, Alsadoon A, Prasad P, Singh A, Elchouemi A (2018) An emotion recognition model based on facial recognition in virtual learning environment. Procedia Comput Sci 125:2–10

    Article  Google Scholar 

  49. Yin L, Wei X, Sun Y, Wang J, Rosato MJ (2006) A 3d facial expression database for facial behavior research. In: Automatic face and gesture recognition. IEEE, pp 211–216

  50. Yu Z, Zhang C (2015) Image based static facial expression recognition with multiple deep network learning. In: Proceedings of the 2015 ACM on international conference on multimodal interaction. ACM, pp 435–442

  51. Zhang B, Quan C, Ren F (2016) Study on cnn in the recognition of emotion in audio and images, IEEE

  52. Zhu L, Shen J, Xie L, Cheng Z (2016) Unsupervised topic hypergraph hashing for efficient mobile image retrieval. IEEE Trans Cybern 47(11):3941–3954

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Sudeep Tanwar.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mungra, D., Agrawal, A., Sharma, P. et al. PRATIT: a CNN-based emotion recognition system using histogram equalization and data augmentation. Multimed Tools Appl 79, 2285–2307 (2020). https://doi.org/10.1007/s11042-019-08397-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-019-08397-0

Keywords

Navigation