Skip to main content
SpringerLink
Log in

MMDA: A Multimodal Dataset for Depression and Anxiety Detection

  • Conference paper
  • First Online:
Pattern Recognition, Computer Vision, and Image Processing. ICPR 2022 International Workshops and Challenges (ICPR 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13643))

Included in the following conference series:

Abstract

In recent years, with the development of artificial intelligence, the use of machine learning and other methods for mental illness detection has become a hot topic. However, obtaining data on mental disorders is very difficult, limiting the development of this field. In this paper, we provide a Multimodal dataset for Depression and Anxiety Detection(MMDA). All subjects in the dataset were diagnosed by professional psychologists, and the subjects’ disorders were determined by combining HAMD and HAMA scores. The dataset includes visual, acoustic, and textual information extracted after de-identification of the original interview videos, for a total of 1025 valid data, which is the largest dataset on mental disorders available. We detail the correlations between each modal and symptoms of depression and anxiety, and validate this dataset by machine learning methods to complete the two tasks of classification of anxiety and depression symptoms and regression of HAMA and HAMD scores. We hope that this dataset will be useful for establishing an automatic detection system for mental disorders and motivate more researchers to engage in mental disorders detection.

Y. Jiang and Z. Zhang are equally-contributed authors.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://pan.baidu.com/s/1b1qygk1d_kYX0bTbx_JbSA?pwd=0qgx.

References

  1. Organization, W.H.: Depression key facts [EB/OL]. https://www.who.int/news-room/fact-sheets/detail/depression/. Accessed 13 Sept 2021

  2. Huang, Y., et al.: Prevalence of mental disorders in China: a cross-sectional epidemiological study. Lancet Psychiatry 6(3), 211–224 (2019)

    Article  Google Scholar 

  3. Shen, G., et al.: Depression detection via harvesting social media: a multimodal dictionary learning solution. In: IJCAI, pp. 3838–3844 (2017)

    Google Scholar 

  4. Xezonaki, D., Paraskevopoulos, G., Potamianos, A., Narayanan, S.: Affective conditioning on hierarchical attention networks applied to depression detection from transcribed clinical interviews. In: INTERSPEECH, pp. 4556–4560 (2020)

    Google Scholar 

  5. Ye, J., et al.: Multi-modal depression detection based on emotional audio and evaluation text. J. Affect. Disord. 295, 904–913 (2021)

    Article  Google Scholar 

  6. Guo, W., Yang, H., Liu, Z.: Deep neural networks for depression recognition based on facial expressions caused by stimulus tasks. In: 2019 8th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW), pp. 133–139. IEEE (2019)

    Google Scholar 

  7. Haque, A., Guo, M., Miner, A.S., Fei-Fei, L.: Measuring depression symptom severity from spoken language and 3D facial expressions. arXiv preprint arXiv:1811.08592 (2018)

  8. Alghowinem, S., et al.: Multimodal depression detection: fusion analysis of paralinguistic, head pose and eye gaze behaviors. IEEE Trans. Affect. Comput. 9(4), 478–490 (2016)

    Article  Google Scholar 

  9. Hamilton, M.: A rating scale for depression. J. Neurol. Neurosurg. Psychiatry 23(1), 56 (1960)

    Article  Google Scholar 

  10. Hamilton, M.: The assessment of anxiety states by rating. Br. J. Med. Psychol. 32(1), 50–55 (1959)

    Article  Google Scholar 

  11. Gaudi, G., Kapralos, B., Collins, K.C., Quevedo, A.: Affective computing: an introduction to the detection, measurement, and current applications. In: Virvou, M., Tsihrintzis, G.A., Tsoukalas, L.H., Jain, L.C. (eds.) Advances in Artificial Intelligence-based Technologies. LAIS, vol. 22, pp. 25–43. Springer, Cham (2022). https://doi.org/10.1007/978-3-030-80571-5_3

    Chapter  Google Scholar 

  12. Maas, A.L., Daly, R.E., Pham, P.T., Huang, D., Ng, A.Y., Potts, C.: Learning word vectors for sentiment analysis. In: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 142–150. Association for Computational Linguistics, Portland (2011), http://www.aclweb.org/anthology/P11-1015

  13. Zhang, Z., Luo, P., Loy, C.C., Tang, X.: Learning social relation traits from face images. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3631–3639 (2015)

    Google Scholar 

  14. Miranda-Correa, J.A., Abadi, M.K., Sebe, N., Patras, I.: Amigos: A dataset for affect, personality and mood research on individuals and groups. IEEE Trans. Affect. Comput. 12(2), 479–493 (2018)

    Article  Google Scholar 

  15. Soleymani, M., Lichtenauer, J., Pun, T., Pantic, M.: A multimodal database for affect recognition and implicit tagging. IEEE Trans. Affect. Comput. 3(1), 42–55 (2011)

    Article  Google Scholar 

  16. Mckeown, G.: The semaine database: annotated multimodal records of emotionally colored conversations between a person and a limited agent. IEEE Trans. Affect. Comput. 3(1), 5–17 (2013)

    Article  Google Scholar 

  17. Gong, Y., Poellabauer, C.: Topic modeling based multi-modal depression detection. In: Proceedings of the 7th Annual Workshop on Audio/Visual Emotion Challenge, pp. 69–76 (2017)

    Google Scholar 

  18. Zhang, Z., Lin, W., Liu, M., Mahmoud, M.: Multimodal deep learning framework for mental disorder recognition. In: 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020), pp. 344–350. IEEE (2020)

    Google Scholar 

  19. Garcia-Ceja, E., Riegler, M., Nordgreen, T., Jakobsen, P., Oedegaard, K.J., Tørresen, J.: Mental health monitoring with multimodal sensing and machine learning: a survey. Pervasive Mob. Comput. 51, 1–26 (2018)

    Article  Google Scholar 

  20. Çiftçi, E., Kaya, H., Güleç, H., Salah, A.A.: The turkish audio-visual bipolar disorder corpus. In: 2018 First Asian Conference on Affective Computing and Intelligent Interaction (ACII Asia), pp. 1–6. IEEE (2018)

    Google Scholar 

  21. Gratch, J., et al.: The distress analysis interview corpus of human and computer interviews. In: Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pp. 3123–3128 (2014)

    Google Scholar 

  22. DeVault, D., et al.: Simsensei kiosk: A virtual human interviewer for healthcare decision support. In: Proceedings of the 2014 International Conference on Autonomous Agents and Multi-Agent Systems, pp. 1061–1068 (2014)

    Google Scholar 

  23. Dibeklioğlu, H., Hammal, Z., Cohn, J.F.: Dynamic multimodal measurement of depression severity using deep autoencoding. IEEE J. Biomed. Health Inform. 22(2), 525–536 (2017)

    Article  Google Scholar 

  24. Cai, H., et al.: Modma dataset: a multi-modal open dataset for mental-disorder analysis. arXiv preprint arXiv:2002.09283 (2020)

  25. Spitzer, R.L., Kroenke, K., Williams, J.B., Group, P.H.Q.P.C.S., Group, P.H.Q.P.C.S., et al.: Validation and utility of a self-report version of PRIME-MD: the PHQ primary care study. JAMA 282(18), 1737–1744 (1999)

    Google Scholar 

  26. Xing, Y., et al.: Task-state heart rate variability parameter-based depression detection model and effect of therapy on the parameters. IEEE Access 7, 105701–105709 (2019)

    Article  Google Scholar 

  27. Byun, S., et al.: Detection of major depressive disorder from linear and nonlinear heart rate variability features during mental task protocol. Comput. Biol. Med. 112, 103381 (2019)

    Article  Google Scholar 

  28. Cai, H., et al.: A pervasive approach to EEG-based depression detection. Complexity 2018, 1–13 (2018)

    Google Scholar 

  29. Sun, S., et al.: Graph theory analysis of functional connectivity in major depression disorder with high-density resting state EEG data. IEEE Trans. Neural Syst. Rehabil. Eng. 27(3), 429–439 (2019)

    Article  Google Scholar 

  30. Fiquer, J.T., Moreno, R.A., Brunoni, A.R., Barros, V.B., Fernandes, F., Gorenstein, C.: What is the nonverbal communication of depression? assessing expressive differences between depressive patients and healthy volunteers during clinical interviews. J. Affect. Disord. 238, 636–644 (2018)

    Article  Google Scholar 

  31. Baltrusaitis, T., Zadeh, A., Lim, Y.C., Morency, L.P.: Openface 2.0: facial behavior analysis toolkit. In: 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), pp. 59–66. IEEE (2018)

    Google Scholar 

  32. Yang, Y., Fairbairn, C., Cohn, J.F.: Detecting depression severity from vocal prosody. IEEE Trans. Affect. Comput. 4(2), 142–150 (2012)

    Article  Google Scholar 

  33. Taguchi, T., et al.: Major depressive disorder discrimination using vocal acoustic features. J. Affect. Disord. 225, 214–220 (2018)

    Article  Google Scholar 

  34. Low, L.S.A., Maddage, N.C., Lech, M., Allen, N.: Mel frequency cepstral feature and gaussian mixtures for modeling clinical depression in adolescents. In: 2009 8th IEEE International Conference on Cognitive Informatics, pp. 346–350. IEEE (2009)

    Google Scholar 

  35. Low, L.S.A., Maddage, N.C., Lech, M., Sheeber, L.B., Allen, N.B.: Detection of clinical depression in adolescents’ speech during family interactions. IEEE Trans. Biomed. Eng. 58(3), 574–586 (2010)

    Article  Google Scholar 

  36. Eyben, F., Wöllmer, M., Schuller, B.: Opensmile: the munich versatile and fast open-source audio feature extractor. In: Proceedings of the 18th ACM International Conference on Multimedia, pp. 1459–1462 (2010)

    Google Scholar 

  37. Ive, J., Gkotsis, G., Dutta, R., Stewart, R., Velupillai, S.: Hierarchical neural model with attention mechanisms for the classification of social media text related to mental health. In: Proceedings of the Fifth Workshop on Computational Linguistics and Clinical Psychology: From Keyboard to Clinic, pp. 69–77 (2018)

    Google Scholar 

  38. Sekulić, I., Strube, M.: Adapting deep learning methods for mental health prediction on social media. arXiv preprint arXiv:2003.07634 (2020)

  39. Weerasinghe, J., Morales, K., Greenstadt, R.: “Because... I was told... so much’’: linguistic indicators of mental health status on twitter. Proc. Priv. Enhancing Technol. 2019(4), 152–171 (2019)

    Article  Google Scholar 

  40. Ji, S., Li, X., Huang, Z., Cambria, E.: Suicidal ideation and mental disorder detection with attentive relation networks. Neural Comput. Appl. 34(13), 1–11 (2021)

    Google Scholar 

Download references

Acknowledgement

This work was supported by the National Key R &D Programme of China (2022YFC3803202), Major Project of Anhui Province under Grant 202203a05020011 and General Programmer of the National Natural Science Foundation of China (61976078).

Author information

Authors and Affiliations

  1. Hefei University of Technology, Hefei, China

    Yueqi Jiang, Ziyang Zhang & Xiao Sun

Authors
  1. Yueqi Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ziyang Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xiao Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xiao Sun .

Editor information

Editors and Affiliations

  1. York University, Toronto, ON, Canada

    Jean-Jacques Rousseau

  2. Ontario Tech University, Oshawa, ON, Canada

    Bill Kapralos

Rights and permissions

Reprints and permissions

Copyright information

© 2023 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Jiang, Y., Zhang, Z., Sun, X. (2023). MMDA: A Multimodal Dataset for Depression and Anxiety Detection. In: Rousseau, JJ., Kapralos, B. (eds) Pattern Recognition, Computer Vision, and Image Processing. ICPR 2022 International Workshops and Challenges. ICPR 2022. Lecture Notes in Computer Science, vol 13643. Springer, Cham. https://doi.org/10.1007/978-3-031-37660-3_49

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-37660-3_49

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-37659-7

  • Online ISBN: 978-3-031-37660-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships

Search

Navigation