Vanessa Richter
ABSTRACT
We present a scalable multimodal dialog platform for the remote digital assessment and monitoring of schizophrenia. Patients diagnosed with schizophrenia and healthy controls interacted with Tina, a virtual conversational agent, as she guided them through a brief set of structured tasks, while their speech and facial video was streamed in real-time to a back-end analytics module. Patients were concurrently assessed by trained raters on validated clinical scales. We find that multiple speech and facial biomarkers extracted from these data streams show significant differences (as measured by effect sizes) between patients and controls, and furthermore, machine learning models built on such features can classify patients and controls with high sensitivity and specificity. We further investigate, using correlation analysis between the extracted metrics and standardized clinical scales for the assessment of schizophrenia symptoms, how such speech and facial biomarkers can provide further insight into schizophrenia symptomatology.
- Anzar Abbas, Bryan J Hansen, Vidya Koesmahargyo, Vijay Yadav, Paul J Rosenfield, Omkar Patil, Marissa F Dockendorf, Matthew Moyer, Lisa A Shipley, M Mercedez Perez-Rodriguez, 2022. Facial and Vocal Markers of Schizophrenia Measured Using Remote Smartphone Assessments: Observational Study. JMIR Formative Research 6, 1 (2022), e26276.Google Scholar
Cross Ref
- Donald Addington, Jean Addington, and B Schissel. 2000. Calgary Depression Scale for Schizophrenia (CDSS). American Psychiatric Association. Task Force for the Handbook of Psychiatric Measures. American Psychiatric Association. Washington DC (2000), 504–507.Google Scholar
- Nancy C Andreasen and Scott Olsen. 1982. Negative v positive schizophrenia: Definition and validation. Archives of general psychiatry 39, 7 (1982), 789–794.Google ScholarCross Ref
- Thomas RE Barnes. 1989. A rating scale for drug-induced akathisia. The British Journal of Psychiatry 154, 5 (1989), 672–676.Google ScholarCross Ref
- Paul Boersma and Vincent Van Heuven. 2001. Speak and unSpeak with PRAAT. Glot International 5, 9/10 (2001), 341–347.Google Scholar
- Veronica Boschi, Eleonora Catricala, Monica Consonni, Cristiano Chesi, Andrea Moro, and Stefano F Cappa. 2017. Connected speech in neurodegenerative language disorders: a review. Frontiers in psychology 8 (2017), 269.Google ScholarCross Ref
- Debsubhra Chakraborty, Zixu Yang, Yasir Tahir, Tomasz Maszczyk, Justin Dauwels, Nadia Thalmann, Jianmin Zheng, Yogeswary Maniam, Nur Amirah, Bhing Leet Tan, 2018. Prediction of negative symptoms of schizophrenia from emotion related low-level speech signals. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 6024–6028.Google ScholarDigital Library
- Cheryl M Corcoran, Vijay A Mittal, Carrie E Bearden, Raquel E Gur, Kasia Hitczenko, Zarina Bilgrami, Aleksandar Savic, Guillermo A Cecchi, and Phillip Wolff. 2020. Language as a biomarker for psychosis: A natural language processing approach. Schizophrenia research 226 (2020), 158–166.Google Scholar
- Michael A Covington, SL Anya Lunden, Sarah L Cristofaro, Claire Ramsay Wan, C Thomas Bailey, Beth Broussard, Robert Fogarty, Stephanie Johnson, Shayi Zhang, and Michael T Compton. 2012. Phonetic measures of reduced tongue movement correlate with negative symptom severity in hospitalized patients with first-episode schizophrenia-spectrum disorders. Schizophrenia research 142, 1-3 (2012), 93–95.Google ScholarCross Ref
- Paul Ekman and Wallace V Friesen. 1978. Facial action coding system. Environmental Psychology & Nonverbal Behavior (1978).Google Scholar
- Wolfgang Gaebel and Wolfgang Woelwer. 2004. Facial expression in the course of schizophrenia and depression. European archives of psychiatry and clinical neuroscience 254 (11 2004), 335–42. https://doi.org/10.1007/s00406-004-0510-5Google Scholar
- Luis F. Gomez, Aythami Morales, Juan R. Orozco-Arroyave, Roberto Daza, and Julian Fierrez. 2021. Improving Parkinson Detection using Dynamic Features from Evoked Expressions in Video. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). 1562–1570. https://doi.org/10.1109/CVPRW53098.2021.00172Google ScholarCross Ref
- William Guy. 1976. ECDEU assessment manual for psychopharmacology. US Department of Health, Education, and Welfare, Public Health Service ….Google Scholar
- Stanley R Kay, Abraham Fiszbein, and Lewis A Opler. 1987. The positive and negative syndrome scale (PANSS) for schizophrenia. Schizophrenia bulletin 13, 2 (1987), 261–276.Google Scholar
- Brian Kirkpatrick, Gregory P Strauss, Linh Nguyen, Bernard A Fischer, David G Daniel, Angel Cienfuegos, and Stephen R Marder. 2011. The brief negative symptom scale: psychometric properties. Schizophrenia bulletin 37, 2 (2011), 300–305.Google Scholar
- Rony Krell, Wenqing Tang, Katrin Hänsel, Michael Sobolev, Sunghye Cho, Sarah Berretta, and Sunny X Tang. 2021. Lexical and acoustic correlates of clinical speech disturbance in schizophrenia. In International Workshop on Health Intelligence. Springer, 27–35.Google Scholar
- Daniel M Low, Kate H Bentley, and Satrajit S Ghosh. 2020. Automated assessment of psychiatric disorders using speech: A systematic review. Laryngoscope Investigative Otolaryngology 5, 1 (2020), 96–116.Google Scholar
- Manas K Mandal, Rakesh Pandey, and Akhouri B Prasad. 1998. Facial expressions of emotions and schizophrenia: a review. Schizophrenia bulletin 24, 3 (1998), 399–412.Google ScholarCross Ref
- Patrick E McKight and Julius Najab. 2010. Kruskal-wallis test. The corsini encyclopedia of psychology(2010), 1–1.Google Scholar
- Michael Neumann, Oliver Roesler, Jackson Liscombe, Hardik Kothare, David Suendermann-Oeft, David Pautler, Indu Navar, Aria Anvar, Jochen Kumm, Raquel Norel, Ernest Fraenkel, Alex Sherman, James Berry, Gary Pattee, Jun Wang, Jordan Green, and Vikram Ramanarayanan. 2021. Investigating the Utility of Multimodal Conversational Technology and Audiovisual Analytic Measures for the Assessment and Monitoring of Amyotrophic Lateral Sclerosis at Scale. Brno, Czech Republic, 4783–4787. https://doi.org/10.21437/Interspeech.2021-1801Google Scholar
- Lena Palaniyappan. 2021. More than a biomarker: could language be a biosocial marker of psychosis?npj Schizophrenia 7, 1 (2021), 1–5.Google Scholar
- Alberto Parola, Ilaria Gabbatore, Laura Berardinelli, Rogerio Salvini, and Francesca M Bosco. 2021. Multimodal assessment of communicative-pragmatic features in schizophrenia: a machine learning approach. NPJ schizophrenia 7, 1 (2021), 1–9.Google Scholar
- Alberto Parola, Arndis Simonsen, Vibeke Bliksted, and Riccardo Fusaroli. 2020. Voice patterns in schizophrenia: A systematic review and Bayesian meta-analysis. Schizophrenia research 216 (2020), 24–40.Google Scholar
- Joerg Pueschel, Hans Stassen, G. Bomben, Christian Scharfetter, and Daniel Hell. 1998. Speaking behavior and speech sound characteristics in acute schizophrenia. Journal of psychiatric research 32 (03 1998), 89–97. https://doi.org/10.1016/S0022-3956(98)00046-6Google Scholar
- Vikram Ramanarayanan, Adam C Lammert, Hannah P Rowe, Thomas F Quatieri, and Jordan R Green. 2022. Speech as a Biomarker: Opportunities, Interpretability, and Challenges. Perspectives of the ASHA Special Interest Groups (2022), 1–8.Google Scholar
- Vikram Ramanarayanan, Oliver Roesler, Michael Neumann, David Pautler, Doug Habberstad, Andrew Cornish, Hardik Kothare, Vignesh Murali, Jackson Liscombe, Dirk Schnelle-Walka, 2020. Toward Remote Patient Monitoring of Speech, Video, Cognitive and Respiratory Biomarkers Using Multimodal Dialog Technology.. In INTERSPEECH. 492–493.Google Scholar
- Viliam Rapcan, Shona D’Arcy, Sherlyn Yeap, Natasha Afzal, Jogin Thakore, and Richard B Reilly. 2010. Acoustic and temporal analysis of speech: A potential biomarker for schizophrenia. Medical engineering & physics 32, 9 (2010), 1074–1079.Google Scholar
- Ali Siam, Naglaa Soliman, Abeer Algarni, Fathi Abd El-Samie, and Ahmed Sedik. 2022. Deploying Machine Learning Techniques for Human Emotion Detection. Computational Intelligence and Neuroscience 2022 (02 2022). https://doi.org/10.1155/2022/8032673Google Scholar
- GM Simpson and JWS Angus. 1970. A rating scale for extrapyramidal side effects. Acta Psychiatrica Scandinavica 45, S212 (1970), 11–19.Google ScholarCross Ref
- Yashish M Siriwardena, Carol Espy-Wilson, Chris Kitchen, and Deanna L Kelly. 2021. Multimodal Approach for Assessing Neuromotor Coordination in Schizophrenia Using Convolutional Neural Networks. In Proceedings of the 2021 International Conference on Multimodal Interaction. 768–772.Google ScholarDigital Library
- David Suendermann-Oeft, Amanda Robinson, Andrew Cornish, Doug Habberstad, David Pautler, Dirk Schnelle-Walka, Franziska Haller, Jackson Liscombe, Michael Neumann, Mike Merrill, Oliver Roesler, and Renko Geffarth. 2019. NEMSI: A Multimodal Dialog System for Screening of Neurological or Mental Conditions. In Proceedings of ACM International Conference on Intelligent Virtual Agents (IVA). Paris, France.Google ScholarDigital Library
- Eric J Tan, Denny Meyer, Erica Neill, and Susan L Rossell. 2021. Investigating the diagnostic utility of speech patterns in schizophrenia and their symptom associations. Schizophrenia research 238 (2021), 91–98.Google Scholar
Index Terms
- Towards Multimodal Dialog-Based Speech & Facial Biomarkers of Schizophrenia
Recommendations
Speech dialogue with facial displays: multimodal human-computer conversation
ACL '94: Proceedings of the 32nd annual meeting on Association for Computational LinguisticsHuman face-to-face conversation is an ideal model for human-computer dialogue. One of the major features of face-to-face communication is its multiplicity of communication channels that act on multiple modalities. To realize a natural multimodal ...
Biomarkers Selection based on FS-TNNR in Schizophrenia
ICBBT '21: Proceedings of the 2021 13th International Conference on Bioinformatics and Biomedical TechnologySchizophrenia (SZ) is a chronic mental illness that severely affects people's thoughts, feelings and behaviors. As time goes by, the symptoms will become more and more serious, clinically manifested as the confusion of thinking and speech, delusions, ...
Regional thinning of cerebral cortical thickness in first-episode and chronic schizophrenia
First-episode schizophrenia and chronic schizophrenia have different patterns of cortical gray matter loss, due to differences in the period of illness. Differences in the reduction of cortical thickness between first-episode and chronic schizophrenia ...
Comments