Abstract
Road accidents have a significant impact on increasing death rates. In addition to weather, roads, and vehicles, human error constitutes these accidents’ main reason. So, driver-safety technology is one of the common research areas, whether in academia or industry. The driver’s behavior is influenced by his feelings such as anger or sadness, as well as the physical distraction factors such as using mobile or drinking. Recognition of the driver’s emotions is crucial in expecting the driver’s behavior and dealing with it. In this work, the Convolutional Neural Network (CNN) model is employed to implement a Facial Expression Recognition (FER) approach to identify the driver’s emotions. The proposed CNN model has achieved considerable performance in prediction and classification tasks. However, it is similar to other deep learning approaches that have a lack of transparency and interpretability. We use Explainable Artificial Intelligence (XAI) techniques that generate interpretations for decisions and provide human-explainable representations to address this shortage. We utilize two visualization methods of XAI approaches to support our decision of choosing the architecture of the proposed FER model. Our model achieves accuracies of 92.85%, 99.28%, 88.88%, and 100% for the JAFFE, CK+, KDEF, and KMU-FED datasets, respectively.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Adadi, A., Berrada, M.: Peeking inside the black-box: a survey on explainable artificial intelligence (XAI). IEEE Access 6, 52138–52160 (2018)
Dagher, I., Dahdah, E., Al Shakik, M.: Facial expression recognition using three-stage support vector machines. Visual Comput. Ind. Biomedi. Art 2(1), 1–9 (2019). https://doi.org/10.1186/s42492-019-0034-5
Dong, Y., Hu, Z., Uchimura, K., Murayama, N.: Driver inattention monitoring system for intelligent vehicles: a review. IEEE Trans. Intell. Transp. Syst. 12(2), 596–614 (2010)
Du, G., Wang, Z., Gao, B., Mumtaz, S., Abualnaja, K.M., Du, C.: A convolution bidirectional long short-term memory neural network for driver emotion recognition. IEEE Trans. Intell. Transp. Syst. 1–9 (2020)
Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT Press, Cambridge (2016)
Guettas, A., Ayad, S., Kazar, O.: Driver state monitoring system: a review. In: Proceedings of the 4th International Conference on Big Data and Internet of Things. BDIoT 2019, Association for Computing Machinery, New York, NY, USA (2019). https://doi.org/10.1145/3372938.3372966
Harel, J., Koch, C., Perona, P.: Saliency map tutorial (2012). https://www.techylib.com/en/view/skillfulwolverine/saliency_map_tutorial
Jeong, M., Ko, B.C.: Driver’s facial expression recognition in real-time for safe driving. Sensors 18(12), 4270 (2018)
Kandeel, A.A., Rahmanian, M., Zulkernine, F.H., Abbas, H., Hassanein, H.S.: Facial expression recognition using a simplified convolutional neural network model. In: 2020 International Conference on Communications, Signal Processing, and their Applications (ICCSPA 2020)(Accepted). Sharjah, United Arab Emirates (2020)
Knapton, S.: Which emotion raises the risk of a car crash by nearly 10 times? February 2016. https://www.telegraph.co.uk/news/science/science-news/12168472/Which-emotion-raises-the-risk-of-a-car-crash-by-nearly-10-times.html?
Ko, B.C.: A brief review of facial emotion recognition based on visual information. Sensors 18(2), 401 (2018)
Kowalczuk, Z., Czubenko, M., Merta, T.: Emotion monitoring system for drivers. IFAC-PapersOnLine 52(8), 200–205 (2019)
Krishna, M.G., Srinivasulu, A.: Face detection system on Adaboost algorithm using HAAR classifiers. Int. J. Mod. Eng. Res. 2(5), 3556–3560 (2012)
Lee, K.W., Yoon, H.S., Song, J.M., Park, K.R.: Convolutional neural network-based classification of driver’s emotion during aggressive and smooth driving using multi-modal camera sensors. Sensors 18(4), 957 (2018)
Li, K., Jin, Y., Akram, M.W., Han, R., Chen, J.: Facial expression recognition with convolutional neural networks via a new face cropping and rotation strategy. Visual Comput. 36(2), 391–404 (2019). https://doi.org/10.1007/s00371-019-01627-4
Lopes, A.T., de Aguiar, E., De Souza, A.F., Oliveira-Santos, T.: Facial expression recognition with convolutional neural networks: coping with few data and the training sample order. Pattern Recogn. 61, 610–628 (2017)
Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z., Matthews, I.: The extended cohn-kanade dataset (ck+): a complete dataset for action unit and emotion-specified expression. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops, pp. 94–101 (2010)
Lundqvist, D., Flykt, A., Öhman, A.: The karolinska directed emotional faces (kdef). CD ROM from Department of Clinical Neuroscience, Psychology Section, Karolinska Institutet 91, 630 (1998)
Lyons, M., Kamachi, M., Gyoba, J.: The Japanese Female Facial Expression (JAFFE) Database, April 1998. https://doi.org/10.5281/zenodo.3451524
Mesken, J.: Determinants and consequences of drivers’ emotions. Stichting Wetenschappelijk Onderzoek Verkeersveiligheid SWOV (2006)
Nonis, F., Dagnes, N., Marcolin, F., Vezzetti, E.: 3d approaches and challenges in facial expression recognition algorithms–a literature review. Appl. Sci. 9(18), 3904 (2019)
Nübling, M., Stößel, U., Hasselhorn, H.M., Michaelis, M., Hofmann, F.: Measuring psychological stress and strain at work-evaluation of the COPSOQ questionnaire in Germany. GMS Psycho-Social Medicine 3 (2006)
Pandey, R.K., Karmakar, S., Ramakrishnan, A., Saha, N.: Improving facial emotion recognition systems using gradient and Laplacian images. arXiv preprint arXiv:1902.05411 (2019)
Puthanidam, R.V., Moh, T.S.: A hybrid approach for facial expression recognition. In: Proceedings of the 12th International Conference on Ubiquitous Information Management and Communication, pp. 1–8 (2018)
Scherer, K.R., Schorr, A., Johnstone, T.: Appraisal Processes in Emotion: Theory, Methods, Research. Oxford University Press, Oxford (2001)
Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 618–626 (2017)
Shorten, C., Khoshgoftaar, T.M.: A survey on image data augmentation for deep learning. J. Big Data 6(1), 60 (2019)
Simonyan, K., Vedaldi, A., Zisserman, A.: Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034 (2013)
Springenberg, J.T., Dosovitskiy, A., Brox, T., Riedmiller, M.: Striving for simplicity: the all convolutional net. arXiv preprint arXiv:1412.6806 (2014)
Viola, P., Jones, M.: Rapid object detection using a boosted cascade of simple features. In: Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001. vol. 1, pp. I-I (2001)
Wang, Q., Jia, K., Liu, P.: Design and implementation of remote facial expression recognition surveillance system based on PCA and KNN algorithms. In: 2015 International Conference on Intelligent Information Hiding and Multimedia Signal Processing (IIH-MSP), pp. 314–317 (2015)
World Health Organization: Global status report on road safety 2018: Summary. World Health Organization, Technical report (2018)
Yang, B., Cao, J., Ni, R., Zhang, Y.: Facial expression recognition using weighted mixture deep neural network based on double-channel facial images. IEEE Access 6, 4630–4640 (2017)
Zhang, J., Mei, X., Liu, H., Yuan, S., Qian, T.: Detecting negative emotional stress based on facial expression in real time. In: 2019 IEEE 4th International Conference on Signal and Image Processing (ICSIP), pp. 430–434 (2019)
Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2921–2929 (2016)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Kandeel, A.A., Abbas, H.M., Hassanein, H.S. (2021). Explainable Model Selection of a Convolutional Neural Network for Driver’s Facial Emotion Identification. In: Del Bimbo, A., et al. Pattern Recognition. ICPR International Workshops and Challenges. ICPR 2021. Lecture Notes in Computer Science(), vol 12666. Springer, Cham. https://doi.org/10.1007/978-3-030-68780-9_53
Download citation
DOI: https://doi.org/10.1007/978-3-030-68780-9_53
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-68779-3
Online ISBN: 978-3-030-68780-9
eBook Packages: Computer ScienceComputer Science (R0)
