Abstract
Facial expressions are dynamic processes that evolve over temporal segments, including onset, apex, offset, and neutral. However, previous works on automatic facial expression analysis have mainly focused on the recognition of discrete emotions, neglecting the continuous nature of these processes. Additionally, facial images captured from videos in the wild often have varying resolutions due to fixed-lens cameras. To address these problems, our objective is to develop a robust facial expression recognition classifier that provides good performance in such challenging environments. We evaluated several state-of-the-art models on labeled and unlabeled collections and analyzed their performance at different scales. To improve performance, we filtered the probabilities provided by each classifier and demonstrated that this improves decision-making consistency by more than 10%, leading to accuracy improvement. Finally, we combined the models’ backbones into a temporal-sequence classifier, leveraging this consistency-performance trade-off and achieving an additional improvement of 9.6%.
This work is partially funded by the the Spanish Ministry of Science and Innovation under project PID2021-122402OB-C22, and by the ACIISI-Gobierno de Canarias and European FEDER funds under project, ProID2021010012, ULPGC Facilities Net, and Grant EIS 2021 04.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Akbar, M.T., Ilmi, M.N., Rumayar, I.V., Moniaga, J., Chen, T.K., Chowanda, A.: Enhancing game experience with facial expression recognition as dynamic balancing. Proc. Comput. Sci. 157, 388–395 (2019)
Aminbeidokhti, M., Pedersoli, M., Cardinal, P., Granger, E.: Emotion recognition with spatial attention and temporal softmax pooling. In: Karray, F., Campilho, A., Yu, A. (eds.) Image Analysis and Recognition, pp. 323–331 (2019)
Baddar, W.J., Ro, Y.M.: Mode variational LSTM robust to unseen modes of variation: application to facial expression recognition. In: AAAI Conference on Artificial Intelligence. vol. 33, pp. 3215–3223 (2019)
Barrett, L.F., Adolphs, R., Marsella, S., Martinez, A.M., Pollak, S.D.: Emotional expressions reconsidered: challenges to inferring emotion from human facial movements. Psychol. Sci. Public Interest 20(1), 1–68 (2019)
Bell, S., Bala, K.: Learning visual similarity for product design with convolutional neural networks. ACM Trans. Graph. 34(4), 98:1–98:10 (2015)
Deng, J., Guo, J., Zhou, Y., Yu, J., Kotsia, I., Zafeiriou, S.: RetinaFace: Single-stage dense face localisation in the wild. CoRR abs/1905.00641 (2019)
Dhall, A.: EmotiW 2019: automatic emotion, engagement and cohesion prediction tasks. In: 2019 International Conference on Multimodal Interaction, pp. 546–550 (2019)
Dhall, A., Asthana, A., Goecke, R., Gedeon, T.: Emotion recognition using PHOG and LPQ features. In: 2011 IEEE International Conference on Automatic Face Gesture Recognition, pp. 878–883 (2011)
Ekman, P., Friesen, W.: Unmasking the Face: A Guide to Recognizing Emotions from Facial Expressions. Prentice Hall, Hoboken (1975)
Fan, Y., Lam, J.C.K., Li, V.O.K.: Video-based emotion recognition using deeply-supervised neural networks. In: 20th ACM International Conference on Multimodal Interaction, pp. 584–588 (2018)
Feng, D., Ren, F.: Dynamic facial expression recognition based on two-stream-CNN with LBP-TOP. In: 2018 5th IEEE International Conference on Cloud Computing and Intelligence Systems, pp. 355–359 (2018)
Freire-Obregón, D., Castrillón-Santana, M.: An evolutive approach for smile recognition in video sequences. Int. J. Pattern Recogn. Artif. Intell. 29, 1550006 (2015)
Freire-Obregón, D., Castrillón-Santana, M., Barra, P., Bisogni, C., Nappi, M.: An attention recurrent model for human cooperation detection. Comput. Vis. Image Underst. 197–198, 102991 (2020)
Freire-Obregón, D., Rosales-Santana, K., Marín-Reyes, P.A., Penate-Sanchez, A., Lorenzo-Navarro, J., Castrillón-Santana, M.: Improving user verification in human-robot interaction from audio or image inputs through sample quality assessment. Pattern Recogn. Lett. 149, 179–184 (2021)
Happy, S.L., Routray, A.: Automatic facial expression recognition using features of salient facial patches. IEEE Trans. Affect. Comput. 6(1), 1–12 (2015)
Howard, A.G., et al.: MobileNets: Efficient convolutional neural networks for mobile vision applications. CoRR abs/1704.04861 (2017)
Hu, P., Cai, D., Wang, S., Yao, A., Chen, Y.: Learning supervised scoring ensemble for emotion recognition in the wild. In: 19th ACM International Conference on Multimodal Interaction, pp. 553–560 (2017)
Jain, V., Learned-Miller., E.: FDDB: A benchmark for face detection in unconstrained settings. Tech. rep., University of Massachusetts, Amherst (2010)
Kumar, V., Rao, S., Yu, L.: Noisy student training using body language dataset improves facial expression recognition. In: Computer Vision - ECCV 2020 Workshops, pp. 756–773 (2020)
Liu, C., Tang, T., Lv, K., Wang, M.: Multi-feature based emotion recognition for video clips. In: 20th ACM International Conference on Multimodal Interaction, pp. 630–634 (2018)
Luan, P., Huynh, V., Tuan Anh, T.: Facial expression recognition using residual masking network. In: IEEE 25th International Conference on Pattern Recognition, pp. 4513–4519 (2020)
Marras, M., Marín-Reyes, P., Lorenzo-Navarro, J., Castrillón-Santana, M., Fenu, G.: AveROBOT: an audio-visual dataset for people re-identification and verification in human-robot interaction. In: Proceedings of the 8th International Conference on Pattern Recognition Applications and Methods, pp. 255–265 (2019)
Meng, D., Peng, X., Wang, K., Qiao, Y.: Frame attention networks for facial expression recognition in videos. In: 2019 IEEE International Conference on Image Processing, pp. 3866–3870 (2019)
Pantic, M., Patras, I.: Dynamics of facial expression: recognition of facial actions and their temporal segments from face profile image sequences. IEEE Trans. Syst. Man Cybern. 36(2), 433–449 (2006)
Penate-Sanchez, A., Freire-Obregón, D., Lorenzo-Melián, A., Lorenzo-Navarro, J., Castrillón-Santana, M.: TGC20ReId: a dataset for sport event re-identification in the wild. Pattern Recog. Lett. 138, 355–361 (2020)
Saeed, A., Al-Hamadi, A., Niese, R., Elzobi, M.: Effective geometric features for human emotion recognition. In: 2012 IEEE 11th International Conference on Signal Processing. vol. 1, pp. 623–627 (2012)
Savchenko, A.V.: Facial expression and attributes recognition based on multi-task learning of lightweight neural networks. In: 2021 IEEE 19th International Symposium on Intelligent Systems and Informatics, pp. 119–124 (2021)
Tan, M., Le, Q.V.: EfficientNet: Rethinking model scaling for convolutional neural networks. CoRR abs/1905.11946 (2019)
Uchida, M.C., et al.: Identification of muscle fatigue by tracking facial expressions. PLoS ONE 13(12), e0208834 (2018)
Vyas, A.S., Prajapati, H.B., Dabhi, V.K.: Survey on face expression recognition using CNN. In: 2019 5th International Conference on Advanced Computing Communication Systems (ICACCS), pp. 102–106 (2019)
Yang, S., Luo, P., Loy, C.C., Tang, X.: WIDER FACE: a face detection benchmark. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 5525–5533. IEEE, Hawai, USA (2016)
Yao, A., Cai, D., Ping Hu, S.W., Sha, L., Chen, Y.: HoloNet: towards robust emotion recognition in the wild. In: 18th ACM International Conference on Multimodal Interaction, pp. 472–478 (2016)
Zhang, F., Zhang, T., Mao, Q., Xu, C.: Geometry guided pose-invariant facial expression recognition. IEEE Trans. Image Process. 29, 4445–4460 (2020)
Zhang, N., Luo, J., Gao, W.: Research on face detection technology based on MTCNN. In: 2020 International Conference on Computer Network, Electronic and Automation, pp. 154–158 (2020)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Freire-Obregón, D., Hernández-Sosa, D., Santana, O.J., Lorenzo-Navarro, J., Castrillón-Santana, M. (2023). Towards Facial Expression Robustness in Multi-scale Wild Environments. In: Foresti, G.L., Fusiello, A., Hancock, E. (eds) Image Analysis and Processing – ICIAP 2023. ICIAP 2023. Lecture Notes in Computer Science, vol 14233. Springer, Cham. https://doi.org/10.1007/978-3-031-43148-7_16
Download citation
DOI: https://doi.org/10.1007/978-3-031-43148-7_16
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-43147-0
Online ISBN: 978-3-031-43148-7
eBook Packages: Computer ScienceComputer Science (R0)