skip to main content
10.1145/3485447.3512128acmconferencesArticle/Chapter ViewAbstractPublication PagesthewebconfConference Proceedingsconference-collections
research-article

Early Identification of Depression Severity Levels on Reddit Using Ordinal Classification

Published: 25 April 2022 Publication History

Abstract

User-generated text on social media is a promising avenue for public health surveillance and has been actively explored for its feasibility in the early identification of depression. Existing methods in the identification of depression have shown promising results; however, these methods were all focused on treating the identification as a binary classification problem. To date, there has been little effort towards identifying users’ depression severity level and disregard the inherent ordinal nature across these fine-grain levels. This paper aims to make early identification of depression severity levels on social media data. To accomplish this, we built a new dataset based on the inherent ordinal nature over depression severity levels using clinical depression standards on Reddit posts. The posts were classified into 4 depression severity levels covering the clinical depression standards on social media. Accordingly, we reformulate the early identification of depression as an ordinal classification task over clinical depression standards such as Beck’s Depression Inventory and the Depressive Disorder Annotation scheme to identify depression severity levels. With these, we propose a hierarchical attention method optimized to factor in the increasing depression severity levels through a soft probability distribution. We experimented using two datasets (a public dataset having more than one post from each user and our built dataset with a single user post) using real-world Reddit posts that have been classified according to questionnaires built by clinical experts and demonstrated that our method outperforms state-of-the-art models. Finally, we conclude by analyzing the minimum number of posts required to identify depression severity level followed by a discussion of empirical and practical considerations of our study.

References

[1]
Aaron T Beck, Calvin H Ward, Mock Mendelson, Jeremiah Mock, and John Erbaugh. 1961. An inventory for measuring depression. Archives of general psychiatry 4, 6 (1961), 561–571.
[2]
Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150(2020).
[3]
Adrian Benton, Glen Coppersmith, and Mark Dredze. 2017. Ethical research protocols for social media health research. In Proceedings of the First ACL Workshop on Ethics in Natural Language Processing. 94–102.
[4]
David M Blei, Andrew Y Ng, and Michael I Jordan. 2003. Latent dirichlet allocation. the Journal of machine Learning research 3 (2003), 993–1022.
[5]
Leo Breiman. 2001. Random forests. Machine learning 45, 1 (2001), 5–32.
[6]
Sergio G Burdisso, Marcelo Luis Errecalde, and Manuel Montes y Gómez. 2021. Using Text Classification to Estimate the Depression Level of Reddit Users. Journal of Computer Science & Technology 21 (2021).
[7]
Stevie Chancellor, Michael L Birnbaum, Eric D Caine, Vincent MB Silenzio, and Munmun De Choudhury. 2019. A taxonomy of ethical tensions in inferring mental health states from social media. In Proceedings of the conference on fairness, accountability, and transparency. 79–88.
[8]
Stevie Chancellor and Munmun De Choudhury. 2020. Methods in predictive techniques for mental health status on social media: a critical review. NPJ digital medicine 3, 1 (2020), 1–11.
[9]
Junyoung Chung, Çaglar Gülçehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling. CoRR abs/1412.3555(2014). arxiv:1412.3555http://arxiv.org/abs/1412.3555
[10]
Glen Coppersmith, Ryan Leary, Patrick Crutchley, and Alex Fine. 2018. Natural language processing of social media as screening for suicide risk. Biomedical informatics insights 10 (2018), 1178222618792860.
[11]
Corinna Cortes and Vladimir Vapnik. 1995. Support-vector networks. Machine learning 20, 3 (1995), 273–297.
[12]
Munmun De Choudhury, Scott Counts, and Eric Horvitz. 2013. Predicting postpartum changes in emotion and behavior via social media. In Proceedings of the SIGCHI conference on human factors in computing systems. 3267–3276.
[13]
Munmun De Choudhury, Michael Gamon, Scott Counts, and Eric Horvitz. 2013. Predicting depression via social media. In Proceedings of the International AAAI Conference on Web and Social Media, Vol. 7.
[14]
Munmun De Choudhury and Emre Kiciman. 2017. The language of social support in social media and its effect on suicidal ideation risk. In Proceedings of the International AAAI Conference on Web and Social Media, Vol. 11.
[15]
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Association for Computational Linguistics, Minneapolis, Minnesota, 4171–4186. https://doi.org/10.18653/v1/N19-1423
[16]
Adam G Dunn, Kenneth D Mandl, and Enrico Coiera. 2018. Social media interventions for precision public health: promises and risks. NPJ digital medicine 1, 1 (2018), 1–4.
[17]
Jacob Eisenstein, Amr Ahmed, and Eric P Xing. 2011. Sparse additive generative models of text. In Proceedings of the 28th international conference on machine learning (ICML-11). Citeseer, 1041–1048.
[18]
Sindhu Kiranmai Ernala, Michael L Birnbaum, Kristin A Candan, Asra F Rizvi, William A Sterling, John M Kane, and Munmun De Choudhury. 2019. Methodological gaps in predicting mental health states from social media: triangulating diagnostic signals. In Proceedings of the 2019 chi conference on human factors in computing systems. 1–16.
[19]
Eibe Frank and Mark Hall. 2001. A simple approach to ordinal classification. In European Conference on Machine Learning. Springer, 145–156.
[20]
Manas Gaur, Amanuel Alambo, Joy Prakash Sain, Ugur Kursuncu, Krishnaprasad Thirunarayan, Ramakanth Kavuluru, Amit Sheth, Randy Welton, and Jyotishman Pathak. 2019. Knowledge-aware assessment of severity of suicide risk for early intervention. In The World Wide Web Conference. 514–525.
[21]
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long Short-Term Memory. Neural Computation 9, 8 (1997), 1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735
[22]
Y Kim. 2014. Convolutional neural networks for sentence classification., arXiv. preprint (2014).
[23]
Thomas N Kipf and Max Welling. 2016. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907(2016).
[24]
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. arxiv:1909.11942 [cs.CL]
[25]
Xien Liu, Xinxin You, Xiao Zhang, Ji Wu, and Ping Lv. 2020. Tensor graph convolutional networks for text classification. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34. 8409–8416.
[26]
David E Losada and Fabio Crestani. 2016. A test collection for research on depression and language use. In International Conference of the Cross-Language Evaluation Forum for European Languages. Springer, 28–39.
[27]
David E Losada, Fabio Crestani, and Javier Parapar. 2017. eRISK 2017: CLEF lab on early risk prediction on the internet: experimental foundations. In International Conference of the Cross-Language Evaluation Forum for European Languages. Springer, 346–360.
[28]
David E Losada, Fabio Crestani, and Javier Parapar. 2018. Overview of eRisk: early risk prediction on the internet. In International conference of the cross-language evaluation forum for european languages. Springer, 343–361.
[29]
David E Losada, Fabio Crestani, and Javier Parapar. 2019. Overview of erisk 2019 early risk prediction on the internet. In International Conference of the Cross-Language Evaluation Forum for European Languages. Springer, 340–357.
[30]
Danielle L Mowery, Craig Bryan, and Mike Conway. 2015. Towards developing an annotation scheme for depressive disorder symptoms: A preliminary study using Twitter data. In Proceedings of the 2nd Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Reality. 89–98.
[31]
Usman Naseem, Matloob Khushi, Jinman Kim, and Adam G Dunn. 2021. Classifying vaccine sentiment tweets by modelling domain-specific representation and commonsense knowledge into context-aware attentive GRU. arXiv preprint arXiv:2106.09589(2021).
[32]
Usman Naseem and Katarzyna Musial. 2019. Dice: Deep intelligent contextual embedding for twitter sentiment analysis. In 2019 International conference on document analysis and recognition (ICDAR). IEEE, 953–958.
[33]
Usman Naseem, Imran Razzak, Peter Eklund, and Katarzyna Musial. 2020. Towards improved deep contextual embedding for the identification of irony and sarcasm. In 2020 International Joint Conference on Neural Networks (IJCNN). IEEE, 1–7.
[34]
Usman Naseem, Imran Razzak, and Peter W Eklund. 2021. A survey of pre-processing techniques to improve short-text quality: a case study on hate speech detection on twitter. Multimedia Tools and Applications 80, 28 (2021), 35239–35266.
[35]
Usman Naseem, Imran Razzak, Matloob Khushi, Peter W Eklund, and Jinman Kim. 2021. Covidsenti: A large-scale benchmark Twitter data set for COVID-19 sentiment analysis. IEEE Transactions on Computational Social Systems (2021).
[36]
Mark Olfson, Carlos Blanco, and Steven C Marcus. 2016. Treatment of adult depression in the United States. JAMA internal medicine 176, 10 (2016), 1482–1491.
[37]
Ahmed Husseini Orabi, Prasadith Buddhitha, Mahmoud Husseini Orabi, and Diana Inkpen. 2018. Deep learning for depression detection of twitter users. In Proceedings of the Fifth Workshop on Computational Linguistics and Clinical Psychology: From Keyboard to Clinic. 88–97.
[38]
Minsu Park, Chiyoung Cha, and Meeyoung Cha. 2012. Depressive moods of users portrayed in Twitter. (2012).
[39]
Minsu Park, David McDonald, and Meeyoung Cha. 2013. Perception differences between the depressed and non-depressed users in twitter. In Proceedings of the International AAAI Conference on Web and Social Media, Vol. 7.
[40]
James W Pennebaker, Martha E Francis, and Roger J Booth. 2001. Linguistic inquiry and word count: LIWC 2001. Mahway: Lawrence Erlbaum Associates 71, 2001 (2001), 2001.
[41]
Hassan Ramchoun, Mohammed Amine Janati Idrissi, Youssef Ghanou, and Mohamed Ettaouil. 2016. Multilayer Perceptron: Architecture Optimization and Training.Int. J. Interact. Multim. Artif. Intell. 4, 1 (2016), 26–30.
[42]
Guozheng Rao, Chengxia Peng, Li Zhang, Xin Wang, and Zhiyong Feng. 2020. A Knowledge Enhanced Ensemble Learning Model for Mental Disorder Detection on Social Media. In International Conference on Knowledge Science, Engineering and Management. Springer, 181–192.
[43]
Andrew G Reece, Andrew J Reagan, Katharina LM Lix, Peter Sheridan Dodds, Christopher M Danforth, and Ellen J Langer. 2017. Forecasting the onset and course of mental illness with Twitter data. Scientific reports 7, 1 (2017), 1–11.
[44]
Esteban A Ríssola, David E Losada, and Fabio Crestani. 2021. A Survey of Computational Methods for Online Mental State Assessment on Social Media. ACM Transactions on Computing for Healthcare 2, 2 (2021), 1–31.
[45]
Gerard Salton and Michael J McGill. 1983. Introduction to modern information retrieval. mcgraw-hill.
[46]
Ramit Sawhney, Harshit Joshi, Saumya Gandhi, and Rajiv Ratn Shah. 2021. Towards Ordinal Suicide Ideation Detection on Social Media. In Proceedings of the 14th ACM International Conference on Web Search and Data Mining. 22–30.
[47]
Han-Chin Shing, Philip Resnik, and Douglas W Oard. 2020. A prioritization model for suicidality risk assessment. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 8124–8137.
[48]
Ruba Skaik and Diana Inkpen. 2020. Using Social Media for Mental Health Surveillance: A Review. ACM Computing Surveys (CSUR) 53, 6 (2020), 1–31.
[49]
Hajime Sueki. 2015. The association of suicide-related Twitter use with suicidal behaviour: a cross-sectional study of young internet users in Japan. Journal of affective disorders 170 (2015), 155–160.
[50]
Yoshihiko Suhara, Yinzhan Xu, and Alex Sandy Pentland. 2017. Deepmood: Forecasting depressed mood based on self-reported histories via recurrent neural networks. In Proceedings of the 26th International Conference on World Wide Web. Springer, 715–724.
[51]
Michael M Tadesse, Hongfei Lin, Bo Xu, and Liang Yang. 2019. Detection of depression-related posts in reddit social media forum. IEEE Access 7(2019), 44883–44893.
[52]
Truyen Tran, Dinh Phung, Wei Luo, Richard Harvey, Michael Berk, and Svetha Venkatesh. 2013. An integrated framework for suicide risk prediction. In Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining. 1410–1418.
[53]
Elsbeth Turcan and Kathleen McKeown. 2019. Dreaddit: A Reddit dataset for stress analysis in social media. arXiv preprint arXiv:1911.00133(2019).
[54]
Ronghua Xu and Qingpeng Zhang. 2016. Understanding online health groups for depression: social network and linguistic perspectives. Journal of medical Internet research 18, 3 (2016), e63.
[55]
Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alexander J. Smola, and Eduard H. Hovy. 2016. Hierarchical Attention Networks for Document Classification. In HLT-NAACL.
[56]
Liang Yao, Chengsheng Mao, and Yuan Luo. 2019. Graph convolutional networks for text classification. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33. 7370–7377.
[57]
Andrew Yates, Arman Cohan, and Nazli Goharian. 2017. Depression and self-harm risk assessment in online forums. arXiv preprint arXiv:1709.01848(2017).
[58]
Xujuan Zhou, Enrico Coiera, Guy Tsafnat, Diana Arachi, Mei-Sing Ong, Adam G Dunn, 2015. Using social connection information to improve opinion mining: Identifying negative sentiment about HPV vaccines on Twitter. (2015).
[59]
Mark Zimmerman, Jennifer H Martinez, Diane Young, Iwona Chelminski, and Kristy Dalrymple. 2013. Severity classification on the Hamilton depression rating scale. Journal of affective disorders 150, 2 (2013), 384–388.
[60]
Hamad Zogan, Imran Razzak, Shoaib Jameel, and Guandong Xu. 2021. DepressionNet: A Novel Summarization Boosted Deep Framework for Depression Detection on Social Media. arXiv preprint arXiv:2105.10878(2021).

Cited By

View all
  • (2025)Detection of Depression Severity in Social Media Text Using Transformer-Based ModelsInformation10.3390/info1602011416:2(114)Online publication date: 7-Feb-2025
  • (2025)Depression Detection From Social Media Posts Using Emotion Aware Encoders and Fuzzy Based Contrastive NetworksIEEE Transactions on Fuzzy Systems10.1109/TFUZZ.2024.346177633:1(43-53)Online publication date: Jan-2025
  • (2025)A Cross Attention Approach to Diagnostic Explainability Using Clinical Practice Guidelines for DepressionIEEE Journal of Biomedical and Health Informatics10.1109/JBHI.2024.348357729:2(1333-1342)Online publication date: Feb-2025
  • Show More Cited By

Index Terms

  1. Early Identification of Depression Severity Levels on Reddit Using Ordinal Classification
          Index terms have been assigned to the content through auto-classification.

          Recommendations

          Comments

          Information & Contributors

          Information

          Published In

          cover image ACM Conferences
          WWW '22: Proceedings of the ACM Web Conference 2022
          April 2022
          3764 pages
          ISBN:9781450390965
          DOI:10.1145/3485447
          Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

          Sponsors

          Publisher

          Association for Computing Machinery

          New York, NY, United States

          Publication History

          Published: 25 April 2022

          Permissions

          Request permissions for this article.

          Check for updates

          Author Tags

          1. Depression identification
          2. Ordinal classification
          3. Social media

          Qualifiers

          • Research-article
          • Research
          • Refereed limited

          Conference

          WWW '22
          Sponsor:
          WWW '22: The ACM Web Conference 2022
          April 25 - 29, 2022
          Virtual Event, Lyon, France

          Acceptance Rates

          Overall Acceptance Rate 1,899 of 8,196 submissions, 23%

          Contributors

          Other Metrics

          Bibliometrics & Citations

          Bibliometrics

          Article Metrics

          • Downloads (Last 12 months)410
          • Downloads (Last 6 weeks)49
          Reflects downloads up to 28 Feb 2025

          Other Metrics

          Citations

          Cited By

          View all
          • (2025)Detection of Depression Severity in Social Media Text Using Transformer-Based ModelsInformation10.3390/info1602011416:2(114)Online publication date: 7-Feb-2025
          • (2025)Depression Detection From Social Media Posts Using Emotion Aware Encoders and Fuzzy Based Contrastive NetworksIEEE Transactions on Fuzzy Systems10.1109/TFUZZ.2024.346177633:1(43-53)Online publication date: Jan-2025
          • (2025)A Cross Attention Approach to Diagnostic Explainability Using Clinical Practice Guidelines for DepressionIEEE Journal of Biomedical and Health Informatics10.1109/JBHI.2024.348357729:2(1333-1342)Online publication date: Feb-2025
          • (2025)Depression Detection in Social Media: A Comprehensive Review of Machine Learning and Deep Learning TechniquesIEEE Access10.1109/ACCESS.2025.353086213(12789-12818)Online publication date: 2025
          • (2024)A novel multi-model feature generation technique for suicide detectionPeerJ Computer Science10.7717/peerj-cs.230110(e2301)Online publication date: 28-Oct-2024
          • (2024)Depressonify: BERT a deep learning approach of detection of depressionEAI Endorsed Transactions on Pervasive Health and Technology10.4108/eetpht.10.551310Online publication date: 22-Mar-2024
          • (2024)Large Language Models in Psychiatry: Current Applications, Limitations, and Future ScopeBig Data Mining and Analytics10.26599/BDMA.2024.90200467:4(1148-1168)Online publication date: Dec-2024
          • (2024)Mental-LLMProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36435408:1(1-32)Online publication date: 6-Mar-2024
          • (2024)CDME-GAT: Context-Aware Depression Detection Using Multiembedding and Graph Attention Networks in Social Media TextIEEE Transactions on Computational Social Systems10.1109/TCSS.2024.340268911:6(7212-7222)Online publication date: Dec-2024
          • (2024)Calibration of Transformer-Based Models for Identifying Stress and Depression in Social MediaIEEE Transactions on Computational Social Systems10.1109/TCSS.2023.328300911:2(1979-1990)Online publication date: Apr-2024
          • Show More Cited By

          View Options

          Login options

          View options

          PDF

          View or Download as a PDF file.

          PDF

          eReader

          View online with eReader.

          eReader

          HTML Format

          View this article in HTML Format.

          HTML Format

          Figures

          Tables

          Media

          Share

          Share

          Share this Publication link

          Share on social media