Skip to main content

Advertisement

Springer Nature Link
Log in

Emotion detection for supporting depression screening

  • 1224: New Frontiers in Multimedia-based and Multimodal HCI
  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Depression is the most prevalent mental disorder in the world. One of the most adopted tools for depression screening is the Beck Depression Inventory-II (BDI-II) questionnaire. Patients may minimize or exaggerate their answers. Thus, to further examine the patient’s mood while filling in the questionnaire, we propose a mobile application that captures the BDI-II patient’s responses together with their images and speech. Deep learning techniques such as Convolutional Neural Networks analyze the patient’s audio and image data. The application displays the correlation between the patient’s emotional scores and DBI-II scores to the clinician at the end of the questionnaire, indicating the relationship between the patient’s emotional state and the depression screening score. We conducted a preliminary evaluation involving clinicians and patients to assess (i) the acceptability of proposed application for use in clinics and (ii) the patient user experience. The participants were eight clinicians who tried the tool with 21 of their patients. The results seem to confirm the acceptability of the app in clinical practice.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Explore related subjects

Discover the latest articles and news from researchers in related subjects, suggested using machine learning.

Data Availability

The datasets generated and analysed during the current study are available from the corresponding author on reasonable request.

Notes

  1. https://www.who.int/news-room/fact-sheets/detail/depression

  2. https://flutter.dev/

  3. https://cloud.google.com/dialogflow

  4. https://www.kaggle.com/deadskull7/fer2013#_sid=js0

  5. https://www.who.int/ethics/Ethics_basic_concepts_ENG.pdf

  6. https://www.eui.eu/Documents/ServicesAdmin/DeanOfStudies/CodeofEthicsinAcademicResearch.pdf

  7. https://europa.eu/youreurope/business/dealing-with-customers/data-protection/data-protection-gdpr/index_en.htm

  8. https://www.cc.gatech.edu/gvu/user_surveys/survey-1998-10/questions/use.html

  9. https://www.hotjar.com/net-promoter-score/

References

  1. Badshah AM, Ahmad J, Rahim N, Baik SW (2017) Speech emotion recognition from spectrograms with deep convolutional neural network. In: 2017 international conference on platform technology and service (PlatCon). IEEE, pp 1–5

  2. Beck AT (1979) Cognitive therapy of depression guilford press

  3. Beck AT, Steer RA, Brown G (1996) Beck depression inventory–ii psychological assessment

  4. Beck AT, Ward CH, Mendelson M, Mock J, Erbaugh J (1961) An inventory for measuring depression. Archives of general psychiatry 4(6):561–571

    Article  Google Scholar 

  5. Bertero D, Fung P (2017) A first look into a convolutional neural network for speech emotion detection. In: 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, pp 5115–5119

  6. (2022). CDC: depression evaluation measures. https://www.cdc.gov/workplacehealthpromotion/health-strategies/depression/evaluation-measures/index.html. Accessed 15 Dec 2022

  7. De Melo WC, Granger E, Hadid A (2019) Depression detection based on deep distribution learning. In: 2019 IEEE international conference on image processing (ICIP), pp 4544–4548. https://doi.org/10.1109/ICIP.2019.8803467

  8. Denecke K, Vaaheesan S, Arulnathan A (2021) A mental health chatbot for regulating emotions (sermo) - concept and usability test. IEEE Trans Emerging Topics Comput 9(3):1170–1182. https://doi.org/10.1109/TETC.2020.2974478

    Article  Google Scholar 

  9. Deshpande M, Rao V (2017) Depression detection using emotion artificial intelligence. In: 2017 international conference on intelligent sustainable systems (ICISS). IEEE, pp 858–862

  10. Ekman P (1999) Basic emotions. Handbook Cognit Emotion 98 (45-60):16

    Google Scholar 

  11. Ezz-Eldin M, Khalaf AAM, Hamed HFA, Hussein AI (2021) Efficient feature-aware hybrid model of deep learning architectures for speech emotion recognition. IEEE Access 9:19999–20011. https://doi.org/10.1109/ACCESS.2021.3054345

    Article  Google Scholar 

  12. Flint AJ, Black SE, Campbell-Taylor I, Gailey GF, Levinton C (1993) Abnormal speech articulation, psychomotor retardation, and subcortical dysfunction in major depression. J Psychiatric Res 27(3):309–319

    Article  Google Scholar 

  13. Francese R, Attanasio P (2021) Supporting depression screening with multimodal emotion detection. In: Angeli AD, Chittaro L, Gennari R, Marsico MD, Melonio A, Gena C, Russis LD, Spano LD (eds) Proceedings of the 14th biannual conference of the italian SIGCHI chapter, CHItaly ’21, Bozen-Bolzano, Italy, and online (www). ACM, 11-13 July 2021, pp 7–178

  14. Francese R, Risi M, Tortora G (2020) A user-centered approach for detecting emotions with low-cost sensors. Multim Tools Appl 79(47):35885–35907. https://doi.org/10.1007/s11042-020-09576-0

    Article  Google Scholar 

  15. Hamiditabar N, Chalechale A, Kabudian SJ (2022) Determining the severity of depression in speech based on combination of acoustic-space and score-space features. In: 2022 9th Iranian joint congress on fuzzy and intelligent systems (CFIS), pp 1–5. https://doi.org/10.1109/CFIS54774.2022.9756429

  16. Hamilton D, Lane JV, Gaston P, Patton J, Macdonald D, Simpson A, Howie C (2014) Assessing treatment outcomes using a single question: the net promoter score. Bone Joint J 96(5):622–628

    Article  Google Scholar 

  17. Hauke J, Kossowski T (2011) Comparison of values of pearson’s and spearman’s correlation coefficients on the same sets of data. Quaestiones Geographicae 30(2):87–93

    Article  Google Scholar 

  18. He L, Cao C (2018) Automated depression analysis using convolutional neural networks from speech. J Biomed Inf 83:103–111

    Article  Google Scholar 

  19. Jan A, Meng H, Gaus YFBA, Zhang F (2017) Artificial intelligent system for automatic depression level analysis through visual and vocal expressions. IEEE Trans Cognit Develop Syst 10(3):668–680

    Article  Google Scholar 

  20. Korszun A (2002) Facial pain, depression and stress–connections and directions. J Oral Pathology Med 31(10):615–619

    Article  Google Scholar 

  21. Laflamme L, Chipps J, Fangerau H, Juth N, Légaré F, Sawe H, Wallis L (2019) Targeting ethical considerations tied to image-based mobile health diagnostic support specific to clinicians in low-resource settings: the brocher proposition. Glob Health Action 12(1):1666695

    Article  Google Scholar 

  22. Lemey C, Larsen ME, Devylder J, Courtet P, Billot R, Lenca P, Walter M, Baca-García E, Berrouiguet S (2019) Clinicians’ concerns about mobile ecological momentary assessment tools designed for emerging psychiatric problems: prospective acceptability assessment of the memind app. J Med Internet Res 21(4):10111

    Article  Google Scholar 

  23. Likforman-Sulem L, Esposito A, Faundez-Zanuy M, Clémençon S, Cordasco G (2017) Emothaw: a novel database for emotional state recognition from handwriting and drawing. IEEE Trans Human-Mach Syst 47(2):273–284

    Article  Google Scholar 

  24. Lu X, Shi D, Liu Y, Yuan J (2021) Speech depression recognition based on attentional residual network. Frontiers Bioscience-Landmark 26 (12):1746–1759

    Article  Google Scholar 

  25. Mao Q, Dong M, Huang Z, Zhan Y (2014) Learning salient features for speech emotion recognition using convolutional neural networks. IEEE Trans Multimed 16(8):2203–2213

    Article  Google Scholar 

  26. McPherson A, Martin C (2010) A narrative review of the beck depression inventory (bdi) and implications for its use in an alcohol-dependent population. J Psychiatr Ment Health Nurs 17(1):19–30

    Article  Google Scholar 

  27. Morris RR, Schueller SM, Picard RW (2015) Efficacy of a web-based, crowdsourced peer-to-peer cognitive reappraisal platform for depression: randomized controlled trial. J Med Internet Res 17(3):4167

    Article  Google Scholar 

  28. Mouchet-Mages S, Baylé FJ (2022) Sadness as an integral part of depression Dialogues in clinical neuroscience

  29. Mukaka MM (2012) A guide to appropriate use of correlation coefficient in medical research. Malawi Med J 24(3):69–71

    Google Scholar 

  30. Mulay A, Dhekne A, Wani R, Kadam S, Deshpande P, Deshpande P (2020) Automatic depression level detection through visual input. In: 2020 fourth world conference on smart trends in systems, security and sustainability (worldS4). IEEE, pp 19–22

  31. Nakamura R, Mitsukura Y (2018) Feature analysis of electroencephalography in patients with depression. In: 2018 IEEE life sciences conference (LSC). IEEE, pp 53–56

  32. Niu M, Tao J, Liu B, Huang J, Lian Z (2020) Multimodal spatiotemporal representation for automatic depression level detection. IEEE Trans Affect Comput

  33. Pampouchidou A, Simantiraki O, Vazakopoulou C-M, Chatzaki C, Pediaditis M, Maridaki A, Marias K, Simos P, Yang F, Meriaudeau F et al (2017) Facial geometry and speech analysis for depression detection. In: 2017 39th annual international conference of the IEEE engineering in medicine and biology society (EMBC). IEEE, pp 1433–1436

  34. Parada-Cabaleiro E, Costantini G, Batliner A, Schmitt M, Schuller BW (2019) Demos: an italian emotional speech corpus. Lang Resour Eval:1–43

  35. Russell JA (1980) A circumplex model of affect. J Personality Social Psychol 39(6):1161

    Article  Google Scholar 

  36. Schrepp M (2015) User experience questionnaire handbook all you need to know to apply the UEQ successfully in your project

  37. Shi D, Lu X, Liu Y, Yuan J, Pan T, Li Y (2021) Research on depression recognition using machine learning from speech. In: 2021 international conference on asian language processing (IALP), pp 52–56. https://doi.org/10.1109/IALP54817.2021.9675271

  38. Sumali B, Mitsukura Y, Tazawa Y, Kishimoto T (2019) Facial landmark activity features for depression screening. In: 2019 58th annual conference of the society of instrument and control engineers of Japan (SICE), pp 1376–1381. https://doi.org/10.23919/SICE.2019.8859798

  39. Tadalagi M, Joshi AM (2021) Autodep: automatic depression detection using facial expressions based on linear binary pattern descriptor. Med Biol Eng Comput:1–16

  40. Tariman JD, Berry DL, Halpenny B, Wolpin S, Schepp K (2011) Validation and testing of the acceptability e-scale for web-based patient-reported outcomes in cancer care. Appl Nurs Res 24(1):53–58

    Article  Google Scholar 

  41. Tasnim M, Stroulia E (2019) Detecting depression from voice. In: Canadian conference on artificial intelligence. Springer, pp 472–478

  42. Torous J, Onnela J, Keshavan M (2017) New dimensions and new tools to realize the potential of rdoc: digital phenotyping via smartphones and connected devices. Trans Psych 7(3):1053–1053

    Article  Google Scholar 

  43. Valstar M, Schuller B, Smith K, Almaev T, Eyben F, Krajewski J, Cowie R, Pantic M (2014) Avec 2014: 3d dimensional affect and depression recognition challenge. In: Proceedings of the 4th international workshop on audio/visual emotion challenge, pp 3–10

  44. Verde L, Raimo G, Vitale F, Carbonaro B, Cordasco G, Marrone S, Esposito A (2021) A lightweight machine learning approach to detect depression from speech analysis. In: 2021 IEEE 33rd international conference on tools with artificial intelligence (ICTAI), pp 330–335. https://doi.org/10.1109/ICTAI52525.2021.00054

  45. Wohlin C, Runeson P, Hst M, Ohlsson MC, Regnell B, Wessln A (2012) Experimentation in software engineering. Springer

  46. (2017). World health organization: world health day 2017 - depression: let’s talk. https://www.who.int/newsroom/events/detail/2017/04/07/default-calendar/world-health-day-2017. Accessed 15 Dec 2022

  47. Yang L (2019) Multi-modal depression detection and estimation. In: 2019 8th international conference on affective computing and intelligent interaction workshops and demos (ACIIW). IEEE, pp 26–30

  48. Zhou X, Jin K, Shang Y, Guo G (2018) Visually interpretable representation learning for depression recognition from facial images. IEEE Trans Affect Comput 11(3):542–552

    Article  Google Scholar 

Download references

Acknowledgements

Many thanks to all the anonymous participants involved in the evaluation and to the anonymous reviewers that largely improved the paper quality with their valuable suggestions.

Funding

No funding was received for conducting this study.

Author information

Authors and Affiliations

  1. Computer Science Department, Università degli Studi di Salerno, Via Giovanni Paolo II, 132, Fisciano, 84084, (SA), Italy

    Rita Francese

  2. IBM Italia S.p.A., Via G. Porzio, Napoli, 80143, Italy

    Pasquale Attanasio

Authors
  1. Rita Francese
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Pasquale Attanasio
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Rita Francese.

Ethics declarations

Conflict of Interests

The authors have no financial or proprietary interests in any material discussed in this article.

Additional information

Ethical concerns

The evaluation procedure described in this paper obtained the ethical approval n. 0001 by the Ethic Committee of the Computer Science Department of the University of Salerno.

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Pasquale Attanasio contributed equally to this work.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Francese, R., Attanasio, P. Emotion detection for supporting depression screening. Multimed Tools Appl 82, 12771–12795 (2023). https://doi.org/10.1007/s11042-022-14290-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-022-14290-0

Keywords