Abstract
Clinical investigations have demonstrated that depression patients’ facial expression mimicry and cognitive capacities are substantially weakened, therefore their facial expressions have great uncertainty. Great uncertainty and limited data of depression patients make facial expression recognition (FER) based on depression patients a difficult endeavor. In this paper, we proposed Depression Vision Transformer (Dep-ViT) to solve the above problems. Firstly, we invited 164 subjects to participate in the Voluntary Facial Expression Mimicry (VFEM) experiment. VFEM contains seven expressions, including neutrality, anger, Disgust, Fear, happiness, sadness and surprise. We employed Person correlation analysis to characterize the action units (AUs) in VFEM at the same time. Secondly, to limit the uncertainty, each small sample in Dep-ViT had a block composed of Squeeze-Excitation (SE) and the self attention layer for the local attention information and sample importance. The sample label that received the least attention will be re-labeled using the Rank Regularization. Thirdly, in addition to the label of the VFEM itself, we manually labeled each expression image of the VFEM again, and used the manual label to assist the model training. The results showed that Dep-ViT obtains excellent results, with an accuracy of 0.417 in the VFEM.
Supported by Qilu University of Technology (Shandong Academy of Sciences).
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Amos, B., Ludwiczuk, B., Satyanarayanan, M., et al.: Openface: a general-purpose face recognition library with mobile applications. CMU Sch. Comput. Sci. 6(2) (2016)
Baltrusaitis, T., Robinson, P., Morency, L.P.: Openface: an open source facial behavior analysis toolkit. In: IEEE Winter Conference on Applications of Computer Vision (2016)
Dhall, A., Goecke, R., Lucey, S., Gedeon, T.: Collecting large, richly annotated facial-expression databases from movies. IEEE Multimedia 19(03), 34–41 (2012)
Dosovitskiy, A., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)
Du, S., Tao, Y., Martinez, A.M.: Compound facial expressions of emotion. Proc. Natl. Acad. Sci. USA 111(15), E1454 (2014)
Ekman, P., Friesen, W.V.: Constants across cultures in the face and emotion. J. Pers. Soc. Psychol. 17(2), 124 (1971)
Gaebel, W., Wölwer, W.: Facial expression and emotional face recognition in schizophrenia and depression. Eur. Arch. Psychiatry Clin. Neurosci. 242(1), 46–52 (1992)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7132–7141 (2018)
Kahou, S.E., et al.: Combining modality specific deep neural networks for emotion recognition in video. In: Proceedings of the 15th ACM on International Conference on Multimodal Interaction, pp. 543–550 (2013)
Lundqvist, D., Flykt, A., Öhman, A.: Karolinska directed emotional faces. Cogn. Emot. 22(6), 1094–1118 (1998)
Minaee, S., Minaei, M., Abdolrashidi, A.: Deep-emotion: facial expression recognition using attentional convolutional network. Sensors 21(9), 3046 (2021)
Schaefer, K.L., Baumann, J., Rich, B.A., Luckenbaugh, D.A., Zarate, C.A., Jr.: Perception of facial emotion in adults with bipolar or unipolar depression and controls. J. Psychiatr. Res. 44(16), 1229–1235 (2010)
Shan, C., Gong, S., McOwan, P.W.: Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis. Comput. 27(6), 803–816 (2009)
Wang, K., Peng, X., Yang, J., Lu, S., Qiao, Y.: Suppressing uncertainties for large-scale facial expression recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6897–6906 (2020)
Wexler, B.E., Levenson, L., Warrenburg, S., Price, L.H.: Decreased perceptual sensitivity to emotion-evoking stimuli in depression. Psychiatry Res. 51(2), 127–138 (1994)
Zhang, K., Zhang, Z., Li, Z., Qiao, Y.: Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Process. Lett. 23(10), 1499–1503 (2016)
Zhi, R., Flierl, M., Ruan, Q., Kleijn, W.B.: Graph-preserving sparse nonnegative matrix factorization with application to facial expression recognition. IEEE Trans. Syst. Man Cybern. Part B (Cybern.) 41(1), 38–52 (2010)
Zwick, J.C., Wolkenstein, L.: Facial emotion recognition, theory of mind and the role of facial mimicry in depression. J. Affect. Disord. 210, 90–99 (2017)
Acknowledgment
This work was supported by the Shandong Provincial Natural Science Foundation, China (Grant No: ZR2021MF079, ZR2020MF039). the National Natural Science Foundation of China (Grant No: 81573829). The Key Research and Development Program of Shandong Province (Grant No.2020CXGC010901).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Ye, J., Fu, G., Liu, Y., Cheng, G., Wang, Q. (2022). Dep-ViT: Uncertainty Suppression Model Based on Facial Expression Recognition in Depression Patients. In: Pimenidis, E., Angelov, P., Jayne, C., Papaleonidas, A., Aydin, M. (eds) Artificial Neural Networks and Machine Learning – ICANN 2022. ICANN 2022. Lecture Notes in Computer Science, vol 13531. Springer, Cham. https://doi.org/10.1007/978-3-031-15934-3_10
Download citation
DOI: https://doi.org/10.1007/978-3-031-15934-3_10
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-15933-6
Online ISBN: 978-3-031-15934-3
eBook Packages: Computer ScienceComputer Science (R0)