Abstract
Recognizing emotions in controlled conditions, based on facial expressions, has achieved high accuracies in the past years. This is still a challenging task for robots working in real-world scenarios due to different factors such as illumination, pose variation or occlusions. One of the next barriers of science is to give sociable robots the ability to fully engage in emotional interactions with users. In this paper a real-time emotion recognition system using a YOLO-based facial detection system and an ensemble CNN for sociable robots, is proposed. Experiments have been carried out on the most challenging database, FER 2013, giving a performance of 72.47% on test sets, achieving current standards.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Bartlett, M.S., Littlewort, G., Frank, M., Lainscsek, C., Fasel, I., Movellan, J.: Recognizing facial expression: machine learning and application to spontaneous behavior. In: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPRV 2005), vol. 2, pp. 568–573, June 2005
Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2005), San Diego, CA, USA, vol. 1, pp. 886–893. IEEE (2005)
Devries, T., Biswaranjan, K., Taylor, G.W.: Multi-task learning of facial landmarks and expression. In: 2014 Canadian Conference on Computer and Robot Vision, pp. 98–103, May 2014
Diehl, J.J., Schmitt, L.M., Villano, M., Crowell, C.R.: The clinical use of robots for individuals with autism spectrum disorders: a critical review. Res. Autism Spectr. Disord. 6(1), 249–262 (2012)
Ekman, P.: Pictures of Facial Affect. Consulting Psychologists Press, Palo Alto (1976)
Fong, T., Nourbakhsh, I., Dautenhahn, K.: A survey of socially interactive robots. Robot. Auton. Syst. 42(3–4), 143–166 (2003)
Goodfellow, I.J., et al.: Challenges in representation learning: a report on three machine learning contests. arXiv:1307.0414 [cs, stat], July 2013
Guo, Y., Tao, D., Yu, J., Xiong, H., Li, Y., Tao, D.: Deep neural networks with relativity learning for facial expression recognition. In: 2016 IEEE International Conference on Multimedia Expo Workshops (ICMEW), pp. 1–6, July 2016
Itzcovich, I.: Yolo-face-detection (2018). https://github.com/iitzco/faced
Jack, R.E., Garrod, O.G., Yu, H., Caldara, R., Schyns, P.G.: Facial expressions of emotion are not culturally universal. Proc. Natl. Acad. Sci. 109(19), 7241–7244 (2012)
Jeong, S., et al.: A social robot to mitigate stress, anxiety, and pain in hospital pediatric care. In: Proceedings of the Tenth Annual ACM/IEEE International Conference on Human-Robot Interaction Extended Abstracts - HRI 2015, Extended Abstracts, Portland, Oregon, USA, pp. 103–104. ACM Press (2015)
Kim, B.K., Dong, S.Y., Roh, J., Kim, G., Lee, S.Y.: Fusing aligned and non-aligned face information for automatic affect recognition in the wild: a deep learning approach. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Las Vegas, NV, USA, pp. 1499–1508. IEEE, June 2016
Li, S., Deng, W.: Deep facial expression recognition: a survey. arXiv:1804.08348 [cs], April 2018
Lin, M., Chen, Q., Yan, S.: Network in network. arXiv:1312.4400 [cs], December 2013
Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z., Matthews, I.: The extended cohn-kanade dataset (CK+): a complete dataset for action unit and emotion-specified expression. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops, pp. 94–101, June 2010
Mollahosseini, A., Chan, D., Mahoor, M.H.: Going deeper in facial expression recognition using deep neural networks. In: 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1–10, March 2016
Ojala, T., Pietikäinen, M., Harwood, D.: A comparative study of texture measures with classification based on featured distributions. Pattern Recognit. 29(1), 51–59 (1996)
Pramerdorfer, C., Kampel, M.: Facial expression recognition using convolutional neural networks: state of the art. arXiv:1612.02903 [cs], December 2016
Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You only look once: unified, real-time object detection. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, pp. 779–788. IEEE, June 2016
Scassellati, B., Admoni, H., Mataric, M.: Robots for use in autism research. Ann. Rev. Biomed. Eng. 14(1), 275–294 (2012)
Springenberg, J.T., Dosovitskiy, A., Brox, T., Riedmiller, M.: Striving for simplicity: the all convolutional net. arXiv:1412.6806 [cs], December 2014
Tang, Y.: Deep learning using linear support vector machines. arXiv:1306.0239 [cs, stat], June 2013
Tapus, A., Mataric, M.J., Scassellati, B.: The grand challenges in socially assistive robotics, p. 7
Viola, P., Jones, M.: Rapid object detection using a boosted cascade of simple features. In: Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2001, vol. 1, p. I. IEEE (2001)
Yang, S., Luo, P., Loy, C.C., Tang, X.: WIDER FACE: a face detection benchmark. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, pp. 5525–5533. IEEE, June 2016
Yu, R., et al.: Use of a therapeutic, socially assistive pet robot (PARO) in improving mood and stimulating social interaction and communication for people with dementia: study protocol for a randomized controlled trial. JMIR Res. Protoc. 4(2), e45 (2015)
Acknowledgements
We want to acknowledge to Programa de Ayudas a Grupos de Excelencia de la Región de Murcia, from Fundación Séneca, Agencia de Ciencia y Tecnología de la Región de Murcia.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Benamara, N.K. et al. (2019). Real-Time Emotional Recognition for Sociable Robotics Based on Deep Neural Networks Ensemble. In: Ferrández Vicente, J., Álvarez-Sánchez, J., de la Paz López, F., Toledo Moreo, J., Adeli, H. (eds) Understanding the Brain Function and Emotions. IWINAC 2019. Lecture Notes in Computer Science(), vol 11486. Springer, Cham. https://doi.org/10.1007/978-3-030-19591-5_18
Download citation
DOI: https://doi.org/10.1007/978-3-030-19591-5_18
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-19590-8
Online ISBN: 978-3-030-19591-5
eBook Packages: Computer ScienceComputer Science (R0)