skip to main content
10.1145/3565472.3592967acmconferencesArticle/Chapter ViewAbstractPublication PagesumapConference Proceedingsconference-collections
research-article

Leveraging Causal Inference to Measure the Impact of a Mental Health App on Users’ Well-being

Published: 19 June 2023 Publication History

Abstract

As stated in the United Nations’ Sustainable Development Goals, poor mental well-being is one of the biggest problems we are facing worldwide. One possible way of addressing it is through interventions delivered via digital devices since they are scalable, ubiquitous and inexpensive. This is also confirmed by the ever-growing plethora of e-health mobile apps being developed. Although these apps rely to some extent on scientific bases, there is still much work to do to understand the effect of specific digital interventions on app users. To shed light on these effects, we ask what types of interventions within the app have the most significant impact on well-being, and to what extent longer engagement leads to improved outcomes. These questions could be answered with dedicated Randomized Controlled Trials (RCTs), which are generally expensive, time-consuming, and single-purposed. To overcome these difficulties, we adopt instrumental variables on a combination of data collected in an RCT, behavioural data from the app, and a randomized recommender system, to evaluate intervention and app dose-response effects on users’ self-reported well-being. Thus, we present a general causal inference approach for extending results from collected data in RCTs applied in the context of digital health intervention. Following this approach, we show how to measure the impact of different types of activities on the users’ well-being. This allows us to identify the most impactful activities in the app (namely, sleep and relaxation activities), which have direct implications for the app design. On the other hand, we prove the positive effect of longer app usage.

References

[1]
Joshua D. Angrist and Guido W. Imbens. 1995. Two-Stage Least Squares Estimation of Average Causal Effects in Models with Variable Treatment Intensity. J. Amer. Statist. Assoc. 90, 430 (1995), 431–442. https://doi.org/10.1080/01621459.1995.10476535 arXiv:https://www.tandfonline.com/doi/pdf/10.1080/01621459.1995.10476535
[2]
World Medical Association. 2013. World Medical Association Declaration of Helsinki: Ethical Principles for Medical Research Involving Human Subjects. JAMA 310, 20 (11 2013), 2191–2194. https://doi.org/10.1001/jama.2013.281053 arXiv:https://jamanetwork.com/journals/jama/articlepdf/1760318/jsc130006.pdf
[3]
Niranjan Bidargaddi, Peter Musiat, Megan Winsall, Gillian Vogl, Victoria Blake, Stephen Quinn, Simone Orlowski, Gaston Antezana, and Geoffrey Schrader. 2017. Efficacy of a web-based guided recommendation service for a curated list of readily available mental health and well-being mobile apps for young people: randomized controlled trial. Journal of medical Internet research 19, 5 (2017), e141.
[4]
Faith Birnbaum, Dana M Lewis, Rochelle Rosen, and Megan L Ranney. 2015. Patient engagement and the design of digital health. Academic emergency medicine: official journal of the Society for Academic Emergency Medicine 22, 6 (2015), 754.
[5]
Jan-Erik Broman, Hans Smedje, Lena Mallon, and Jerker Hetta. 2008. The Minimal Insomnia Symptom Scale (MISS). Upsala Journal of Medical Sciences 113, 2 (Jul. 2008), 131–142. https://doi.org/10.3109/2000-1967-221
[6]
Silvina Catuara-Solarz, Bartlomiej Skorulski, Iñaki Estella-Aguerri, Claudia Bibiana Avella-Garcia, Sarah Shepherd, Emily Stott, Nicola R Hemmings, Aleix Ruiz de Villa, Laura Schulze, and Sophie Dix. 2022. The Efficacy of “Foundations,” a Digital Mental Health App to Improve Mental Well-being During COVID-19: Proof-of-Principle Randomized Controlled Trial. JMIR Mhealth Uhealth 10, 7 (1 Jul 2022), e30976. https://doi.org/10.2196/30976
[7]
Sheldon Cohen, Tom P Kamarck, and Robin J. Mermelstein. 1983. A global measure of perceived stress.Journal of health and social behavior 24 4 (1983), 385–96.
[8]
Kathryn Connor and Jonathan Davidson. 2003. Development of a new resilience scale: The Connor-Davidson Resilience Scale (CD-RISC). Depression and anxiety 18 (09 2003), 76–82. https://doi.org/10.1002/da.10113
[9]
Munmun De Choudhury, Emre Kiciman, Mark Dredze, Glen Coppersmith, and Mrinal Kumar. 2016. Discovering shifts to suicidal ideation from mental health content in social media. In Proceedings of the 2016 CHI conference on human factors in computing systems. 2098–2110.
[10]
Tara Donker, Katherine Petrie, Judy Proudfoot, Janine Clarke, Mary-Rose Birch, and Helen Christensen. 2013. Smartphones for smarter delivery of mental health programs: a systematic review. Journal of medical Internet research 15, 11 (2013), e2791.
[11]
Catherine K Ettman, Salma M Abdalla, Gregory H Cohen, Laura Sampson, Patrick M Vivier, and Sandro Galea. 2020. Prevalence of depression symptoms in US adults before and during the COVID-19 pandemic. JAMA network open 3, 9 (2020), e2019686–e2019686.
[12]
Ben Glocker, Mirco Musolesi, Jonathan Richens, and Caroline Uhler. 2021. Causality in digital medicine.
[13]
Arthur S. Goldberger. 1972. Structural Equation Methods in the Social Sciences. Econometrica 40, 6 (1972), 979–1001. http://www.jstor.org/stable/1913851
[14]
Chaohui Guo, Hutan Ashrafian, Saira Ghafur, Gianluca Fontana, Clarissa Gardner, and Matthew Prime. 2020. Challenges for the evaluation of digital health solutions—A call for innovative evidence generation approaches. NPJ Digital Medicine 3, 1 (2020), 1–14.
[15]
Anders Gustavsson, Mikael Svensson, Frank Jacobi, Christer Allgulander, Jordi Alonso, Ettore Beghi, Richard Dodel, Mattias Ekman, Carlo Faravelli, Laura Fratiglioni, 2011. Cost of disorders of the brain in Europe 2010. European neuropsychopharmacology 21, 10 (2011), 718–779.
[16]
Edward L Hannan. 2008. Randomized clinical trials and observational studies: guidelines for assessing respective strengths and limitations. JACC: Cardiovascular Interventions 1, 3 (2008), 211–217.
[17]
James J. Heckman. 2000. Causal Parameters and Policy Analysis in Economics: A Twentieth Century Retrospective. The Quarterly Journal of Economics 115, 1 (2000), 45–97. http://www.jstor.org/stable/2586935
[18]
Guido W. Imbens. 2014. Instrumental Variables: An Econometrician’s Perspective. Statist. Sci. 29, 3 (2014), 323 – 358. https://doi.org/10.1214/14-STS480
[19]
Guido W. Imbens and Joshua D. Angrist. 1994. Identification and Estimation of Local Average Treatment Effects. Econometrica 62, 2 (1994), 467–475. http://www.jstor.org/stable/2951620
[20]
Guido W. Imbens and Donald B. Rubin. 2015. Causal Inference for Statistics, Social, and Biomedical Sciences: An Introduction. Cambridge University Press. https://doi.org/10.1017/CBO9781139025751
[21]
Amy H Kaji, David Schriger, and Steven Green. 2014. Looking through the retrospectoscope: reducing bias in emergency medicine chart review studies. Annals of emergency medicine 64, 3 (2014), 292–298.
[22]
Jamie M Kawadler, Nicola Rose Hemmings, Sonia Ponzo, Davide Morelli, Geoffrey Bird, and David Plans. 2020. Effectiveness of a smart phone application BioBase for reducing anxiety and increasing mental wellbeing: a pilot feasibility and acceptability study. (2020).
[23]
Predrag Klasnja, Eric B Hekler, Saul Shiffman, Audrey Boruvka, Daniel Almirall, Ambuj Tewari, and Susan A Murphy. 2015. Microrandomized trials: An experimental design for developing just-in-time adaptive interventions.Health Psychology 34, S (2015), 1220.
[24]
Taisa Kushner and Amit Sharma. 2020. Bursts of activity: Temporal patterns of help-seeking and support in online mental health forums. In Proceedings of The Web Conference 2020. 2906–2912.
[25]
ME Larsen, K Huckvale, and J Torous Nicholas. [n. d.]. J., Birrell, L., Li, E., & Reda, B.(2019). Using science to sell apps: Evaluation of mental health app store quality claims. NPJ Digital Medicine 2 ([n. d.]).
[26]
Eun-Hyun Lee. 2012. Review of the Psychometric Evidence of the Perceived Stress Scale. Asian Nursing Research 6, 4 (2012), 121–127. https://doi.org/10.1016/j.anr.2012.08.004
[27]
Megan A McVay, Gary G Bennett, Dori Steinberg, and Corrine I Voils. 2019. Dose–response research in digital health interventions: Concepts, considerations, and challenges.Health Psychology 38, 12 (2019), 1168.
[28]
World Health Organization 2017. Depression and other common mental disorders: global health estimates. Technical Report. World Health Organization.
[29]
Pamala A Pawloski, Gabriel A Brooks, Matthew E Nielsen, and Barbara A Olson-Bullis. 2019. A systematic review of clinical decision support systems for clinical oncology practice. Journal of the National Comprehensive Cancer Network 17, 4 (2019), 331–338.
[30]
Judea Pearl. 2009. Causality: Models, Reasoning and Inference (2nd ed.). Cambridge University Press, USA.
[31]
Matthias Pierce, Holly Hope, Tamsin Ford, Stephani Hatch, Matthew Hotopf, Ann John, Evangelos Kontopantelis, Roger Webb, Simon Wessely, Sally McManus, 2020. Mental health before and during the COVID-19 pandemic: a longitudinal probability sample survey of the UK population. The Lancet Psychiatry 7, 10 (2020), 883–892.
[32]
Svenja Pieritz, Mohammed Khwaja, A Aldo Faisal, and Aleksandar Matic. 2021. Personalised Recommendations in Mental Health Apps: The Impact of Autonomy and Data Sharing. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–12.
[33]
Sonia Ponzo, Davide Morelli, Jamie M Kawadler, Nicola Rose Hemmings, Geoffrey Bird, and David Plans. 2020. Efficacy of the digital therapeutic mobile app biobase to reduce stress and improve mental well-being among university students: randomized controlled trial. JMIR mHealth and uHealth 8, 4 (2020), e17767.
[34]
Koustuv Saha and Amit Sharma. 2020. Causal factors of effective psychosocial outcomes in online mental health communities. In Proceedings of the International AAAI Conference on Web and Social Media, Vol. 14. 590–601.
[35]
Koustuv Saha, Benjamin Sugar, John Torous, Bruno Abrahao, Emre Kıcıman, and Munmun De Choudhury. 2019. A social media study on the effects of psychiatric medication use. In Proceedings of the International AAAI Conference on Web and Social Media, Vol. 13. 440–451.
[36]
Hannah Ritchie Saloni Dattani and Max Roser. 2021. Mental Health. Our World in Data (2021). https://ourworldindata.org/mental-health.
[37]
Robert William Sanson-Fisher, Billie Bonevski, Lawrence W. Green, and Cate D’Este. 2007. Limitations of the Randomized Controlled Trial in Evaluating Population-Based Health Interventions. American Journal of Preventive Medicine 33, 2 (2007), 155–161. https://doi.org/10.1016/j.amepre.2007.04.007
[38]
Amit Sharma, Jake M. Hofman, and Duncan J. Watts. 2015. Estimating the Causal Impact of Recommendation Systems from Observational Data. In Proceedings of the Sixteenth ACM Conference on Economics and Computation (Portland, Oregon, USA) (EC ’15). Association for Computing Machinery, New York, NY, USA, 453–470. https://doi.org/10.1145/2764468.2764488
[39]
Robert L. Spitzer, Kurt Kroenke, Janet B. W. Williams, and Bernd Löwe. 2006. A Brief Measure for Assessing Generalized Anxiety Disorder: The GAD-7. Archives of Internal Medicine 166, 10 (05 2006), 1092–1097. https://doi.org/10.1001/archinte.166.10.1092
[40]
James H. Stock and Francesco Trebbi. 2003. Retrospectives Who Invented Instrumental Variable Regression?The Journal of Economic Perspectives 17, 3 (2003), 177–194. http://www.jstor.org/stable/3216828
[41]
Daniel J Taylor, Kenneth L Lichstein, H Heith Durrence, Brant W Reidel, and Andrew J Bush. 2005. Epidemiology of insomnia, depression, and anxiety. Sleep 28, 11 (2005), 1457–1464.
[42]
Christian Topp, Søren Østergaard, Susan Søndergaard, and Per Bech. 2015. The WHO-5 Well-Being Index: A systematic review of the literature. Psychotherapy and psychosomatics 84 (04 2015), 167–76. https://doi.org/10.1159/000376585
[43]
Helma Torkamaan and Jürgen Ziegler. 2022. Recommendations as Challenges: Estimating Required Effort and User Ability for Health Behavior Change Recommendations. In 27th International Conference on Intelligent User Interfaces. 106–119.
[44]
André Calero Valdez, Martina Ziefle, Katrien Verbert, Alexander Felfernig, and Andreas Holzinger. 2016. Recommender systems for health informatics: state-of-the-art and future perspectives. In Machine learning for health informatics. Springer, 391–414.
[45]
Joep van Agteren, Matthew Iasiello, Laura Lo, Jonathan Bartholomaeus, Zoe Kopsaftis, Marissa Carey, and Michael Kyrios. 2021. A systematic review and meta-analysis of psychological interventions to improve mental wellbeing. Nature Human Behaviour (2021), 1–22.
[46]
Abraham Wald. 1940. The Fitting of Straight Lines if Both Variables are Subject to Error. The Annals of Mathematical Statistics 11, 3 (1940), 284 – 300. https://doi.org/10.1214/aoms/1177731868
[47]
Philip G Wright. 1928. The Tariff on Animal and Vegetable Oils. Macmillan.
[48]
Tianchen Wu, Xiaoqian Jia, Huifeng Shi, Jieqiong Niu, Xiaohan Yin, Jialei Xie, and Xiaoli Wang. 2020. Prevalence of mental health problems during the COVID-19 pandemic: A systematic review and meta-analysis. Journal of affective disorders (2020).
[49]
Longqi Yang, Yin Cui, Yuan Xuan, Chenyang Wang, Serge Belongie, and Deborah Estrin. 2018. Unbiased offline recommender evaluation for missing-not-at-random implicit feedback. In Proceedings of the 12th ACM Conference on Recommender Systems. 279–287.
[50]
Liangyu Zhu, Wenbin Lu, and Rui Song. 2020. Causal Effect Estimation and Optimal Dose Suggestions in Mobile Health. In Proceedings of the 37th International Conference on Machine Learning(Proceedings of Machine Learning Research, Vol. 119), Hal Daumé III and Aarti Singh (Eds.). PMLR, 11588–11598. https://proceedings.mlr.press/v119/zhu20c.html

Cited By

View all
  • (2024)Ranking the causal impact of recommendations under collider bias in k-spots recommender systemsACM Transactions on Recommender Systems10.1145/36431392:2(1-29)Online publication date: 14-May-2024

Index Terms

  1. Leveraging Causal Inference to Measure the Impact of a Mental Health App on Users’ Well-being

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Conferences
      UMAP '23: Proceedings of the 31st ACM Conference on User Modeling, Adaptation and Personalization
      June 2023
      333 pages
      ISBN:9781450399326
      DOI:10.1145/3565472
      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

      Sponsors

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 19 June 2023

      Permissions

      Request permissions for this article.

      Check for updates

      Author Tags

      1. Causal inference
      2. E-health
      3. Instrumental variables
      4. Mobile HCI
      5. RCT new insights
      6. Recommender system evaluation

      Qualifiers

      • Research-article
      • Research
      • Refereed limited

      Conference

      UMAP '23
      Sponsor:

      Acceptance Rates

      Overall Acceptance Rate 162 of 633 submissions, 26%

      Upcoming Conference

      UMAP '25

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)73
      • Downloads (Last 6 weeks)5
      Reflects downloads up to 18 Feb 2025

      Other Metrics

      Citations

      Cited By

      View all
      • (2024)Ranking the causal impact of recommendations under collider bias in k-spots recommender systemsACM Transactions on Recommender Systems10.1145/36431392:2(1-29)Online publication date: 14-May-2024

      View Options

      Login options

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      HTML Format

      View this article in HTML Format.

      HTML Format

      Figures

      Tables

      Media

      Share

      Share

      Share this Publication link

      Share on social media