Abstract
Emotions are spontaneous feelings that are accompanied by fluctuations in facial muscles, which leads to facial expressions. Categorization of these facial expressions as one of the seven basic emotions - happy, sad, anger, disgust, fear, surprise, and neutral is the intention behind Emotion Recognition. This is a difficult problem because of the complexity of human expressions, but is gaining immense popularity due to its vast number of applications such as predicting behavior. Using deeper architectures has enabled researchers to achieve state-of-the-art performance in emotion recognition. Motivated from the aforementioned discussion, in this paper, we propose a model named as PRATIT, used for facial expression recognition that uses specific image preprocessing steps and a Convolutional Neural Network (CNN) model. In PRATIT, preprocessing techniques such as grayscaling, cropping, resizing, and histogram equalization have been used to handle variations in the images. CNNs accomplish better accuracy with larger datasets, but there are no freely accessible datasets with adequate information for emotion recognition with deep architectures. Therefore, to handle the aforementioned issue, we have applied data augmentation in PRATIT, which helps in further fine tuning the model for performance improvement. The paper presents the effects of histogram equalization and data augmentation on the performance of the model. PRATIT with the usage of histogram equalization during image preprocessing and data augmentation surpasses the state-of-the-art results and achieves a testing accuracy of 78.52%.







Similar content being viewed by others
References
Alghowinem S, Goecke R, Wagner M, Epps J, Breakspear M, Parker G (2012) From joyous to clinically depressed: mood detection using spontaneous speech. In: Proceedings of the twenty-fifth international Florida artificial intelligence research society conference, pp 141–146
Alghowinem S, Goecke R, Cohn JF, Wagner M, Parker G, Breakspear M (2015) Cross-cultural detection of depression from nonverbal behaviour. In: 2015 11th IEEE International conference and workshops on automatic face and gesture recognition (FG), vol 1. IEEE, pp 1–8
Ali G, Iqbal MA, Choi TS (2016) Boosted nne collections for multicultural facial expression recognition. Pattern Recogn 55:14–27
Alzubi JA, Bharathikannan B, Tanwar S, Manikandan R, Khanna A, Thaventhiran C (2019) Boosted neural network ensemble classification for lung cancer disease diagnosis. Appl Soft Comput 80:579–591
Anjum MA, Javed MY (2005) Face recognition vs image resolution. In: Information and communication technologies. IEEE, pp 109–112
Bhatia S, Hayat M, Breakspear M, Parker G, Goecke R (2017) A video-based facial behaviour analysis approach to melancholia. In: Automatic face & gesture recognition. IEEE, pp 754–761
Bista U (2018) USPS Dataset. http://www-i6.informatik.rwthaachen.de/keysers/usps.html, last Accessed: 2015-10-28
Cao C, Hou Q, Zhou K (2014) Displaced dynamic expression regression for real-time facial tracking and animation. ACM Trans Graph (TOG) 33(4):43
Chang Cc, Lin Cj (2001) Libsvm: a library for support vector machines. Software available at http://www.csie.ntu.edu.tw/cjlin/libsvm
Chen Y, Yang J, Qian J (2017) Recurrent neural network for facial landmark detection. Neurocomputing 219:26–38
Chrysos GG, Antonakos E, Zafeiriou S, Snape P (2015) Offline deformable face tracking in arbitrary videos. In: Proceedings of the IEEE international conference on computer vision workshops, pp 1–9
Chuang CF, Shih FY (2006) Recognizing facial action units using independent component analysis and support vector machine. Pattern Recogn 39(9):1795–1798
De Silva LC, Miyasato T, Nakatsu R (1997) Facial emotion recognition using multi-modal information. In: Information, communications and signal processing, vol 1. IEEE, pp 397–401
De Silva CR, Ranganath S, De Silva LC (2008) Cloud basis function neural network: a modified rbf network architecture for holistic facial expression recognition. Pattern Recogn 41(4):1241–1253
Dhall A, Goecke R, Lucey S, Gedeon T (2011) Acted facial expressions in the wild database. Australian National University, Canberra, Australia. Technical Report TR-CS-11, 2:1
Dhall A, Goecke R, Lucey S, Gedeon T (2011) Static facial expression analysis in tough conditions: data, evaluation protocol and benchmark. Computer vision workshops (ICCV Workshops). IEEE, pp 2106–2112
Giannakakis G, Pediaditis M, Manousos D, Kazantzaki E, Chiarugi F, Simos PG, Marias K, Tsiknakis M (2017) Stress and anxiety detection using facial cues from videos. Biomed Signal Process Control 31:89–101
Gross R, Matthews I, Cohn E, Kanade T, Baker S (2008) The CMU multi-PIE face database. http://www.cs.cmu.edu/afs/cs/project/PIE/MultiPie/Multi-Pie/Home.html, last Accessed: 2015-10-28
Guo Y, Tao D, Yu J, Xiong H, Li Y, Tao D (2016) Deep neural networks with relativity learning for facial expression recognition. Multimedia & expo workshops (ICMEW). IEEE, pp 1–6
Jain N, Kumar S, Kumar A, Shamsolmoali P, Zareapoor M (2018) Hybrid deep neural networks for face emotion recognition. Pattern Recogn Lett 115:101–106
Jeni LA, Cohn JF, De La Torre F (2013) Facing imbalanced data–recommendations for the use of performance metrics, IEEE
Kaya H, Gurpinar F, Salah AA (2017) Video-based emotion recognition in the wild using deep transfer learning and score fusion. Image Vis Comput 65:66–75
Khorrami P, Paine T, Huang T (2015) Do deep neural networks learn facial action units when doing expression recognition?. In: Proceedings of the IEEE international conference on computer vision workshops, pp 19–27
Kumar GR, Kumar RK, Sanyal G (2017) Facial emotion analysis using deep convolution neural network, IEEE
Laboratory BM (2008) TFEID:Taiwanese facial expression image database. http://bml.ym.edu.tw/tfeid/, last Accessed: 2015-10-28
Labs L (2017) What emotion recognition is: you may not even notice its importance. https://medium.com/@looxid.labs/what-emotion-recognition-is-you-may-not-even-notice-its-importance-d1edd0a4250a, last Accessed: 2018-08-05
Levi G, Hassner T (2015) Emotion recognition in the wild via convolutional neural networks and mapped binary patterns. In: Proceedings of the 2015 ACM on international conference on multimodal interaction. ACM, pp 503–510
Lopes AT, de Aguiar E, De Souza AF, Oliveira-Santos T (2017) Facial expression recognition with convolutional neural networks: coping with few data and the training sample order. Pattern Recogn 61:610–628
Lopes AT, de Aguiar E, Souza AFD, Oliveira-Santos T (2017) Facial expression recognition with convolutional neural networks: coping with few data and the training sample order. Pattern Recogn 61:610–628
Lyons MJ, Budynek J, Akamatsu S (1999) Automatic classification of single facial images. IEEE Trans Pattern Anal Mach Intell 21(12):1357–1362
Mollahosseini A, Chan D, Mahoor MH (2016) Going deeper in facial expression recognition using deep neural networks. In: Applications of computer vision (WACV). IEEE, pp 1–10
Ng HW, Nguyen VD, Vonikakis V, Winkler S (2015) Deep learning for emotion recognition on small datasets using transfer learning. In: Proceedings of the 2015 ACM on international conference on multimodal interaction. ACM, pp 443–449
Nicolle J, Bailly K, Chetouani M (2016) Real-time facial action unit intensity prediction with regularized metric learning. Image Vis Comput 52:1–14
Ojeme B, Mbogho A (2016) Selecting learning algorithms for simultaneous identification of depression and comorbid disorders. Procedia Comput Sci 96:1294–1303
Patii J, Patei R, Kothiya S (2017) Human mental states recognition under face occlusion, IEEE
Pierre-Luc Carrier AC (2013) Challenges in representation learning: facial expression recognition challenge—kaggle. https://www.kaggle.com/c/challenges-in-representation-learning-facial-expression-recognition-challenge/data https://www.kaggle.com/c/challenges-in-representation-learning-facial-expression-recognition-challenge/data, last Accessed: 2018-04-03
Shankar K, Lakshmanaprabu SK, Tanwar S, Rodrigues JJ, Roy NR (2019) Alzheimer detection using group grey wolf optimization based features with convolutional classifier. Comput Electr Eng 77:230–243
Shen J, Zafeiriou S, Chrysos GG, Kossaifi J, Tzimiropoulos G, Pantic M (2015) The first facial landmark tracking in-the-wild challenge: benchmark and results. In: Proceedings of the IEEE international conference on computer vision workshops, pp 50–58
Simard PY, Steinkraus D, Platt JC, et al. (2003) Best practices for convolutional neural networks applied to visual document analysis. In: ICDAR, vol 3, pp 958–962
Tzimiropoulos G (2015) Project-out cascaded regression with an application to face alignment. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3659–3667
University R (2017) RadBoud dataset. http://www.socsci.ru.nl:8180/RaFD2/RaFD?p=main, last Accessed: 2015-10-28
Valstar M, Schuller B, Smith K, Eyben F, Jiang B, Bilakhia S, Schnieder S, Cowie R, Pantic M (2013) Avec 2013: the continuous audio/visual emotion and depression recognition challenge. In: Proceedings of the 3rd ACM international workshop on audio/visual emotion challenge. ACM, pp 3–10
Viola P, Jones M (2001) Rapid object detection using a boosted cascade of simple features. In: Computer vision and pattern recognition, vol 1. IEEE, pp I–I
Wang Z, Qu Z (2017) Research on web text classification algorithm based on improved cnn and svm. In: Communication technology (ICCT). IEEE, pp 1958–1961
Wang SH, Phillips P, Dong ZC, Zhang YD (2018) Intelligent facial emotion recognition based on stationary wavelet entropy and jaya algorithm. Neurocomputing 272:668–676
Xie L, Shen J, Han J, Zhu L, Shao L (2017) Dynamic multi-view hashing for online image retrieval. In: Proceedings of the 26th international joint conference on artificial intelligence IJCAI’17. AAAI Press, pp 3133–3139
Yang Y, Fairbairn C, Cohn JF (2013) Detecting depression severity from vocal prosody. IEEE Trans Affect Comput 4(2):142–150
Yang D, Alsadoon A, Prasad P, Singh A, Elchouemi A (2018) An emotion recognition model based on facial recognition in virtual learning environment. Procedia Comput Sci 125:2–10
Yin L, Wei X, Sun Y, Wang J, Rosato MJ (2006) A 3d facial expression database for facial behavior research. In: Automatic face and gesture recognition. IEEE, pp 211–216
Yu Z, Zhang C (2015) Image based static facial expression recognition with multiple deep network learning. In: Proceedings of the 2015 ACM on international conference on multimodal interaction. ACM, pp 435–442
Zhang B, Quan C, Ren F (2016) Study on cnn in the recognition of emotion in audio and images, IEEE
Zhu L, Shen J, Xie L, Cheng Z (2016) Unsupervised topic hypergraph hashing for efficient mobile image retrieval. IEEE Trans Cybern 47(11):3941–3954
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Mungra, D., Agrawal, A., Sharma, P. et al. PRATIT: a CNN-based emotion recognition system using histogram equalization and data augmentation. Multimed Tools Appl 79, 2285–2307 (2020). https://doi.org/10.1007/s11042-019-08397-0
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-019-08397-0