Cited By
View all- Liu HCao RLi SWang YZhang XXu HSun XWang LQian PSun ZGao KLi F(2024)ViT-Based Face Diagnosis Images Analysis for Schizophrenia DetectionBrain Sciences10.3390/brainsci1501003015:1(30)Online publication date: 29-Dec-2024
Deep Learning Architectures for Facial Emotion Analysis
Authors: 
Bryan Chandra, Grace Oktaviani Kindy, Kelvin Susanto Gunawan, Gavin Putra Satria, Ivan Sebastian Edbert, Derwin SuhartonoAuthors Info & Claims
Bryan Chandra, Grace Oktaviani Kindy, Kelvin Susanto Gunawan, Gavin Putra Satria, Ivan Sebastian Edbert, Derwin SuhartonoAuthors Info & ClaimsPages 241 - 249
Abstract
Mental illnesses, which affect 1 in 4 people globally, can harm a person's life if they are not detected and treated properly. To visually assess a person's mental condition, facial emotion recognition plays a huge role. One way to help automate this process is to use a trained model that can detect a person's emotion using their face as an input. With the advances of technologies capable of detecting a person's emotion, various models have been developed. To find out which model works better to detect a person's emotion, a study to compare EfficientNet-B2 and VGG19 is done. Both models are trained and evaluated against the FER+ dataset with 30 epochs. On the test split of the dataset, VGG19 achieved an accuracy of 90.58%. EfficientNet-B2 on the other hand achieved an accuracy of 78.22% on the test split of the dataset. Both models achieved low accuracy and F1 scores on disgust, fear, and contempt emotions.
References
[1]
World Health Organization. Mental health: Strengthening our response. Retrieved April 4, 2022 from https://www.who.int/news-room/fact-sheets/detail/mental-health-strengthening-our-response
[2]
E. H. Brown, Jennifer L. Young, and Nicole Martinez-Martin. 2022. Psychiatric genomics, mental health equity, and intersectionality: A framework for research and practice. In Frontiers in Psychiatry, Vol. 13. https://doi.org/10.3389/fpsyt.2022.1061705
[3]
Mats Granlund, Christine Imms, Gillian King, Anna Karin Andersson, Lilly Augustine, Rob Brooks, Henrik Danielsson, Jennifer Gothilander, Magnus Ivarsson, Lars-Olov Lundqvist, Frida Lygnegård, and Lena Almqvist. 2021. Definitions and Operationalization of Mental Health Problems, Wellbeing and Participation Constructs in Children with NDD: Distinctions and Clarifications. In International Journal of Environmental Research and Public Health, Vol. 18, No. 4, Article 1656. https://doi.org/10.3390/ijerph18041656
[4]
World Health Organization. Adolescent Mental Health. Retrieved April 6, 2022 from https://www.who.int/news-room/fact-sheets/detail/adolescent-mental-health
[5]
Saloni Dattani, Lucas Rodés-Guirao, Hannah Ritchie and Max Roser. Our World in Data. Mental Health. Retrieved April 7, 2022 from https://ourworldindata.org/mental-health
[6]
Surapon Nochaiwong, Chidchanok Ruengorn, Kednapa Thavorn, Brian Hutton, Ratanaporn Awiphan, Chabaphai Phosuya, Yongyuth Ruanta, Nahathai Wongpakaran, and Tinakon Wongpakaran. 2021. Global prevalence of mental health issues among the general population during the coronavirus disease-2019 pandemic: a systematic review and meta-analysis. In Scientific Reports, Vol. 11, No. 1, Article 10173. https://doi.org/10.1038/s41598-021-89700-8
[7]
Md. Mosfikur Rahman, Mohd. Saifuzzaman, Akash Ahmed, Mahfuja Ferdousi Mahin, and Syeda Farjana Shetu. 2021. Impact of COVID-19 on mental health: A quantitative analysis of anxiety and depression based on regular life and internet use. In Current Research in Behavioral Sciences, Vol. 2, Article 100037. https://doi.org/10.1016/j.crbeha.2021.100037
[8]
NHS Digital. Mental Health of Children and Young People in England, 2022: Follow-up to the 2017 Survey, Part 1 - Mental Health. Retrieved April 30, 2022 from https://digital.nhs.uk/data-and-information/publications/statistical/mental-health-of-children-and-young-people-in-england/2022-follow-up-to-the-2017-survey/part-1—mental-health
[9]
Muhammad Arsyad Subu, Del Fatma Wati, Netrida Netrida, Vetty Priscilla, Jacqueline Maria Dias, Mini Sarah Abraham, Shameran Slewa-Younan, and Nabeel Al-Yateem. 2023. Types of stigma experienced by patients with mental illness and mental health nurses in Indonesia: a qualitative content analysis. In International Journal of Mental Health Systems, Vol. 17, No. 1, Article 16. https://doi.org/10.1186/s13033-021-00502-x
[10]
Marco Colizzi, Antonio Lasalvia, and Mirella Ruggeri. 2020. Prevention and early intervention in youth mental health: Is it time for a multidisciplinary and trans-diagnostic model for care? In International Journal of Mental Health Systems, Vol. 14, No. 1, Article 23. https://doi.org/10.1186/s13033-020-00356-9
[11]
Agnes Binagwaho and Joyeuse Senga. 2021. Children and Adolescent Mental Health in a Time of COVID-19: A Forgotten Priority. In Annals of Global Health, Vol. 87, No. 1, Article 57. https://doi.org/10.5334/aogh.3330
[12]
Benjamin G Druss, Lydia Chwastiak, John Kern, Joseph J Parks, Martha C Ward, and Lori E Raney. 2018. Psychiatry's Role in Improving the Physical Health of Patients With Serious Mental Illness: A Report From the American Psychiatric Association. In Psychiatric Services, Vol. 69, No. 3, pp. 254-256. https://doi.org/10.1176/appi.ps.201700359
[13]
Getinet Ayano, Sileshi Demelash, Zegeye Yohannes, Kibrom Haile, Mikiyas Tulu, Dawit Assefa, Abel Tesfaye, Kelemua Haile, Melat Solomon, Asrat Chaka, and Light Tsegay. 2021. Misdiagnosis, detection rate, and associated factors of severe psychiatric disorders in specialized psychiatry centers in Ethiopia. In Ann Gen Psychiatry, Vol. 20, No. 1, Article 10. https://doi.org/10.1186/s12991-021-00333-7
[14]
Chenyu Ding, Yuying Wu, Xiaoyong Chen, Yue Chen, Zanyi Wu, Zhangya Lin, Dezhi Kang, Wenhua Fang, and Fa Chen. 2022. Global, regional, and national burden and attributable risk factors of neurological disorders: The Global Burden of Disease study 1990–2019. In Frontiers in Public Health, Vol. 10, Article 952161. https://doi.org/10.3389/fpubh.2022.952161
[15]
Yi-Sheng Chao, Chao-Jung Wu, Yi-Chun Lai, Hui-Ting Hsu, Yen-Po Cheng, Hsing-Chien Wu, Shih-Yu Huang, and Wei-Chih Chen. 2022. Why Mental Illness Diagnoses Are Wrong: A Pilot Study on the Perspectives of the Public. In Frontiers in Psychiatry, Vol. 13, pp. 1-9. https://doi.org/10.3389/fpsyt.2022.860487
[16]
Mark Zimmerman. 2016. A Review of 20 Years of Research on Overdiagnosis and Underdiagnosis in the Rhode Island Methods to Improve Diagnostic Assessment and Services (MIDAS) Project. In Canadian Journal of Psychiatry, Vol. 61, No. 2, pp. 71-79. https://doi.org/10.1177/0706743715625935
[17]
Michael W. Kraus. 2017. Voice-only communication enhances empathic accuracy. In American Psychologist, Vol. 72, No. 7, pp. 644-654. https://doi.org/10.1037/amp0000147
[18]
Rianne Gomes E Claudino, Laysa Karen Soares de Lima, Erickson Duarte Bonifácio de Assis, and Nelson Torro. 2019. Facial expressions and eye tracking in individuals with social anxiety disorder: a systematic review. In Psicol. Refl. Crít., Vol. 32, No. 1, pp. 1-9. https://doi.org/10.1186/s41155-019-0121-8
[19]
David Matsumoto and Hyisung C. Hwang. 2018. Microexpressions Differentiate Truths From Lies About Future Malicious Intent. In Frontiers in Psychology, Vol. 9, pp. 2545-2554. https://doi.org/10.3389/fpsyg.2018.02545
[20]
Nour Malek, Daniel Messinger, Andy Yuan Lee Gao, Eva Krumhuber, Whitney Mattson, Ridha Joober, Karim Tabbane, and Julio C Martinez-Trujillo. 2019. Generalizing Duchenne to sad expressions with binocular rivalry and perception ratings. In Emotion, Vol. 19, No. 2, pp. 234-241. https://doi.org/10.1037/emo0000410
[21]
Xiaoyang Wang, Yilin Wang, Mingjie Zhou, Baobin Li, Xiaoqian Liu, and Tingshao Zhu. 2020. Identifying Psychological Symptoms Based on Facial Movements. In Frontiers in Psychiatry, Vol. 11, Article 607890. https://doi.org/10.3389/fpsyt.2020.607890
[22]
Gabrielle Simcock, Larisa T. McLoughlin, Tamara De Regt, Kathryn M. Broadhouse, Denise Beaudequin, Jim Lagopoulos, and Daniel F. Hermens. 2020. Associations between Facial Emotion Recognition and Mental Health in Early Adolescence. In International Journal of Environmental Research and Public Health, Vol. 17, No. 1, Article 330. https://doi.org/10.3390/ijerph17010330
[23]
Kristian Kersting. 2018. Machine Learning and Artificial Intelligence: Two Fellow Travelers on the Quest for Intelligent Behavior in Machines. In Frontiers in Big Data, Vol. 1, Article 6. https://doi.org/10.3389/fdata.2018.00006
[24]
Ngumimi Karen Iyortsuun, Soo-Hyung Kim, Min Jhon, Hyung-Jeong Yang, and Sudarshan Pant. 2023. A Review of Machine Learning and Deep Learning Approaches on Mental Health Diagnosis. In Healthcare, Vol. 11, No. 3, Article 285. https://doi.org/10.3390/healthcare11030285
[25]
Raffaele Cioffi, Marta Travaglioni, Giuseppina Piscitelli, Antonella Petrillo, and Fabio De Felice. 2020. Artificial Intelligence and Machine Learning Applications in Smart Production: Progress, Trends, and Directions. In Sustainability, Vol. 12, No. 2, Article 492. https://doi.org/10.3390/su12020492
[26]
Niklas Kühl, Max Schemmer, Marc Goutier, and Gerhard Satzger. 2022. Artificial intelligence and machine learning. In Electronic Markets, Vol. 32, No. 4, pp. 2235-2244. https://doi.org/10.1007/s12525-022-00598-0
[27]
Iqbal H. Sarker. 2021. Machine Learning: Algorithms, Real-World Applications and Research Directions. In SN Computer Science, Vol. 2, No. 3, Article 160. https://doi.org/10.1007/s42979-021-00592-x
[28]
Chang Su, Zhenxing Xu, Jyotishman Pathak, and Fei Wang. 2020. Deep learning in mental health outcome research: a scoping review. In Transl. Psychiatry, Vol. 10, Article 116. https://doi.org/10.1038/s41398-020-0780-3
[29]
Matthew Squires, Xiaohui Tao, Soman Elangovan, Raj Gururajan, Xujuan Zhou, U Rajendra Acharya, and Yuefeng Li. 2023. Deep learning and machine learning in psychiatry: a survey of current progress in depression detection, diagnosis and treatment. In Brain Inf., Vol. 10, Article 10. https://doi.org/10.1186/s40708-023-00188-6
[30]
Dongdong Liu, Bowen Liu, Tao Lin, Guangya Liu, Guoyu Yang, Dezhen Qi, Ye Qiu, Yuer Lu, Qinmei Yuan, Stella C. Shuai, Xiang Li, Ou Liu, Xiangdong Tang, Jianwei Shuai, Yuping Cao, and Hai Lin. 2022. Measuring depression severity based on facial expression and body movement using deep convolutional neural network. In Front. Psychiatry, Vol. 13, Article 1017064. https://doi.org/10.3389/fpsyt.2022.1017064
[31]
Ying-Li Tian, Takeo Kanade, and Jeffrey F. Cohn. 2005. Facial Expression Analysis. In Handbook of Face Recognition, Springer, New York, NY. https://doi.org/10.1007/0-387-27257-7_12
[32]
IBM. Computer vision. Retrieved April 10, 2022 from https://www.ibm.com/topics/computer-vision
[33]
Zhenjie Song. 2021. Facial Expression Emotion Recognition Model Integrating Philosophy and Machine Learning Theory. In Frontiers in Psychology, Vol. 12. https://doi.org/10.3389/fpsyg.2021.759485
[34]
Daniel Canedo and António J. R. Neves. 2019. Facial Expression Recognition Using Computer Vision: A Systematic Review. In Applied Sciences, Vol. 9. https://doi.org/10.3390/app9214678
[35]
Rawad Abdulghafor, Sherzod Turaev, and Mohammed A H Ali. 2022. Body Language Analysis in Healthcare: An Overview. In Healthcare, Vol. 10, No. 7. https://doi.org/10.3390/healthcare10071251
[36]
Rajib Ghosh and Ditipriya Sinha. 2022. Human Emotion Recognition by Analyzing Facial Expressions, Heart Rate and Blogs Using Deep Learning Method. In Innovations in Systems and Software Engineering. https://doi.org/10.1007/s11334-022-00471-5
[37]
Johannes Keck, Adam Zabicki, Julia Bachmann, Jörn Munzert, and Britta Krüger. 2022. Decoding Spatiotemporal Features of Emotional Body Language in Social Interactions. In Scientific Reports, Vol. 12, No. 1. https://doi.org/10.1038/s41598-022-19267-5
[38]
Gretchen N. Foley, and Julie P. Gentile. 2010. Nonverbal Communication In Psychotherapy. Vol. 7, No. 6.
[39]
Richard Szeliski. Computer Vision: Algorithms and Applications 2nd Edition. Retrieved April 13, 2022 from https://szeliski.org/Book
[40]
Hugh Lloyd-Jukes, Oliver John Gibson, Tracey Wrench, Ade Odunlade, and Lionel Tarassenko. 2021. Vision-Based Patient Monitoring and Management in Mental Health Settings. Journal of Clinical Engineering, vol. 46, no. 1, pp. 36-43. https://doi.org/10.1097/JCE.0000000000000447
[41]
Katie Aafjes-van Doorn, Céline Kamsteeg, Jordan Bate, and Marc Aafjes. 2021. A scoping review of machine learning in psychotherapy research. Psychotherapy Research, vol. 31, no. 1, pp. 92-116. https://doi.org/10.1080/10503307.2020.1808729
[42]
Atul Sajjanhar, ZhaoQi Wu, and Quan Wen. 2018. Deep Learning Models for Facial Expression Recognition. In: 2018 Digital Image Computing: Techniques and Applications (DICTA), Canberra, ACT, Australia, 2018, pp. 1-6. https://doi.org/10.1109/DICTA.2018.8615843.
[43]
Harisu Abdullahi Shehu, Md. Haidar Sharif, and Sahin Uyaver. 2021. Facial expression recognition using deep learning. AIP Conference Proceedings, vol. 2334, no. 1, p. 070003. https://doi.org/10.1063/5.0042221
[44]
Zhanpeng Zhang, Ping Luo, Chen Change Loy, and Xiaoou Tang. 2014. Facial Landmark Detection by Deep Multi-task Learning. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds) Computer Vision – ECCV 2014. ECCV 2014. Lecture Notes in Computer Science, vol. 8694. Springer, Cham. https://doi.org/10.1007/978-3-319-10599-4_7
[45]
Emad Barsoum, Cha Zhang, Cristian Canton Ferrer, and Zhengyou Zhang. 2016. Training Deep Networks for Facial Expression Recognition with Crowd-Sourced Label Distribution. In: ACM International Conference on Multimodal Interaction (ICMI), 2016. https://doi.org/10.48550/arXiv.1608.01041
[46]
Yao Wu and Weigen Qiu. 2017. Facial expression recognition based on improved deep belief networks. AIP Conference Proceedings, vol. 1864, no. 1, pp. 020130. https://doi.org/10.1063/1.4992947
[47]
Jing Li, Kan Jin, Dalin Zhou, Naoyuki Kubota, and Zhaojie Ju. 2020. Attention mechanism-based CNN for facial expression recognition. Neurocomputing, vol. 411, pp. 340-350. https://doi.org/10.1016/j.neucom.2020.06.014
[48]
M. A. H. Akhand, Shuvendu Roy, Nazmul Siddique, Md Abdus Samad Kamal, and Tetsuya Shimamura. 2021. Facial Emotion Recognition Using Transfer Learning in the Deep CNN. Electronics, vol. 10, no. 9, pp. 1036. https://doi.org/10.3390/electronics10091036
[49]
Andrey V. Savchenko. 2022. Frame-level Prediction of Facial Expressions, Valence, Arousal and Action Units for Mobile Devices. arXiv preprint arXiv:2203.13436. https://doi.org/10.48550/arXiv.2203.13436
[50]
Afshin Gholamy, Vladik Kreinovich, Olga Kosheleva. 2018. Why 70/30 or 80/20 Relation Between Training and Testing Sets: A Pedagogical Explanation. Departmental Technical Reports (CS).
[51]
Jianxin Wu. 2017. Introduction to Convolutional Neural Networks. Retrieved April 28, 2022 from Available: https://cs.nju.edu.cn/wujx/paper/CNN.pdf.
Index Terms
- Deep Learning Architectures for Facial Emotion Analysis
Recommendations
Emotion Recognition Through Facial Gestures - A Deep Learning Approach
Mining Intelligence and Knowledge ExplorationAbstractAs defined by some theorists, human emotions are discrete and consistent responses to internal or external events which have significance for an organism. They constitute a major part of our non-verbal communication. Among the human emotions, ...
Analysis of facial emotion expression in eating occasions using deep learning
AbstractEating is experienced as an emotional social activity in any culture. There are factors that influence the emotions felt during food consumption. The emotion felt while eating has a significant impact on our lives and affects different health ...
An efficient deep learning technique for facial emotion recognition
AbstractEmotion recognition from facial images is considered as a challenging task due to the varying nature of facial expressions. The prior studies on emotion classification from facial images using deep learning models have focused on emotion ...
Comments
Information & Contributors
Information
Published In
October 2023
722 pages
Copyright © 2023 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 27 December 2023
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Conference
SIET 2023
SIET 2023: International Conference on Sustainable Information Engineering and Technology
October 24 - 25, 2023
Badung, Bali, Indonesia
Acceptance Rates
Overall Acceptance Rate 45 of 57 submissions, 79%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- View Citations1Total Citations
- 37Total Downloads
- Downloads (Last 12 months)26
- Downloads (Last 6 weeks)3
Reflects downloads up to 05 Mar 2025
Other Metrics
Citations
Cited By
View all- Liu HCao RLi SWang YZhang XXu HSun XWang LQian PSun ZGao KLi F(2024)ViT-Based Face Diagnosis Images Analysis for Schizophrenia DetectionBrain Sciences10.3390/brainsci1501003015:1(30)Online publication date: 29-Dec-2024
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format