Abstract
Widespread use of masks was mandated in many countries as a direct result of the covid-19 pandemic. This meant that mask wearing, which was previously restricted to specialized occupations or cities with high levels of pollution became the norm in many places of the world. This has obvious implications for any system that uses facial images to infer user state. This work attempts to gauge the effect of mask wearing on such systems. Arousal classification is used in this study due to its well-studied nature in image processing literature. Using “Affect in the wild” video dataset, the “masks” were synthetically placed on the facial images extracted from videos. A binary classification between high and low arousal shows that there is a drop in accuracy when using masks. However, this drop is larger in across subject classification than within subject classification. The study shows that it is feasible to develop effective user state classification models even with mask cover.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Mollahosseini, A., Hasani, B., Mahoor, M.H.: Affectnet: a database for facial expression, valence, and arousal computing in the wild. IEEE Trans. Affect. Comput. 10(1), 18–31 (2017)
Kollias, D., et al.: Deep affect prediction in-the-wild: Aff-wild database and challenge, deep architectures, and beyond. Int. J. Comput. Vis. 127, 907–929 (2019)
Machajdik, J., Hanbury, A.: Affective image classification using features inspired by psychology and art theory. In: Proceedings of the 18th ACM international conference on Multimedia, pp. 83–92, October 2010
Nhan, B.R., Chau, T.: Classifying affective states using thermal infrared imaging of the human face. IEEE Trans. Biomed. Eng. 57(4), 979–987 (2009)
Bandara, D., Velipasalar, S., Bratt, S., Hirshfield, L.: Building predictive models of emotion with functional near-infrared spectroscopy. Int. J. Hum Comput Stud. 110, 75–85 (2018)
Buciu, I., Kotsia, I., Pitas, I.: Recognition of facial expressions in presence of partial occlusion. In: Proceedings of the 9th Panhellenic Conference on Informatics (PCI 2003), November 2003sss
Kotsia, I., Buciu, I., Pitas, I.: An analysis of facial expression recognition under partial facial image occlusion. Image Vis. Comput. 26(7), 1052–1067 (2008)
Bourel, F., Chibelushi, C.C., Low, A.A.: Recognition of facial expressions in the presence of occlusion. In: BMVC, pp. 1–10 (2001)
Zhang, L., Tjondronegoro, D., Chandran, V.: Random gabor based templates for facial expression recognition in images with facial occlusion. Neurocomputing 145, 451–464 (2014)
Towner, H., Slater, M.: Reconstruction and recognition of occluded facial expressions using PCA. In: Paiva, A.C.R., Prada, R., Picard, R.W. (eds.) ACII. LNCS, vol. 4738, pp. 36–47. Springer, Heidelberg (2007). https://doi.org/10.1007/978-3-540-74889-2_4
Huang, X., Zhao, G., Zheng, W., Pietikäinen, M.: Towards a dynamic expression recognition system under facial occlusion. Pattern Recogn. Lett. 33(16), 2181–2191 (2012)
Jiang, B., Jia, K.: Research of robust facial expression recognition under facial occlusion condition. In: Zhong, N., Callaghan, V., Ghorbani, A.A., Hu, B. (eds.) AMT. LNCS, vol. 6890, pp. 92–100. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-23620-4_13
Miyakoshi, Y., Kato, S.: Facial emotion detection considering partial occlusion of face using Bayesian network. In: 2011 IEEE Symposium on Computers & Informatics, pp. 96–101. IEEE, March 2011
Cotter, S.F.: Sparse representation for accurate classification of corrupted and occluded facial expressions. In: 2010 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 838–841. IEEE, March 2010
Hammal, Z., Arguin, M., Gosselin, F.: Comparing a novel model based on the transferable belief model with humans during the recognition of partially occluded facial expressions. J. Vis. 9(2), 22 (2009)
Roy, B., Nandy, S., Ghosh, D., Dutta, D., Biswas, P., Das, T.: MOXA: A deep learning based unmanned approach for real-time monitoring of people wearing medical masks. Trans. Indian Natl. Acad. Eng. 5(3), 509–518 (2020). https://doi.org/10.1007/s41403-020-00157-z
Loey, M., Manogaran, G., Taha, M.H.N., Khalifa, N.E.M.: Fighting against COVID-19: a novel deep learning model based on YOLO-v2 with ResNet-50 for medical face mask detection. Sustain. Cities Soc. 65, 102600 (2021)
Bandara, D., Hirshfield, L., Velipasalar, S.: Classification of affect using deep learning on brain blood flow data. J. Near Infrared Spectrosc. 27(3), 206–219 (2019)
Buciu, I., Kotsia, I., Pitas, I.: Facial expression analysis under partial occlusion. In: Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing ICASSP 2005, vol. 5, pp. v–453. IEEE, March 2005
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, pp. 770–778 (2016). https://doi.org/10.1109/CVPR.2016.90
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Bandara, D. (2021). User State Detection Using Facial Images with Mask Cover. In: Stephanidis, C., Antona, M., Ntoa, S. (eds) HCI International 2021 - Posters. HCII 2021. Communications in Computer and Information Science, vol 1420. Springer, Cham. https://doi.org/10.1007/978-3-030-78642-7_10
Download citation
DOI: https://doi.org/10.1007/978-3-030-78642-7_10
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-78641-0
Online ISBN: 978-3-030-78642-7
eBook Packages: Computer ScienceComputer Science (R0)