skip to main content
10.1145/3536220.3558071acmconferencesArticle/Chapter ViewAbstractPublication Pagesicmi-mlmiConference Proceedingsconference-collections
short-paper

Exploring Facial Metric Normalization For Within- and Between-Subject Comparisons in a Multimodal Health Monitoring Agent

Published: 07 November 2022 Publication History

Abstract

The use of facial metrics obtained through remote web-based platforms has shown promising results for at-home assessment of facial function in multiple neurological and mental disorders. However, an important factor influencing the utility of the obtained metrics is the variability within and across participant sessions due to position and movement of the head relative to the camera. In this paper, we investigate two different facial landmark predictors in combination with four different normalization methods with respect to their effect on the utility of facial metrics obtained through a multimodal assessment platform. We analyzed 38 people with Parkinson’s disease (pPD) and 22 healthy controls who were asked to complete four interactive sessions, a week apart from each other. We find that metrics extracted through MediaPipe clearly outperform metrics extracted through OpenCV and Dlib in terms of test-retest reliability and patient-control discriminability. Furthermore, our results suggest that using the inter-caruncular distance to normalize all raw visual measurements prior to metric computation is optimal for between-subject analyses, while raw measurements (without normalization) can also be used for within-subject comparisons.

References

[1]
A. Bandini, J. R. Green, B. D. Richburg, and Y. Yunusova. 2018. Automatic detection of orofacial impairment in stroke. In Interspeech. Hyderabad, India.
[2]
M. Baumeister, E. Terzi, Y. Ekici, and T. Kohnen. 2004. Comparison of manual and automated methods to determine horizontal corneal diameter. Journal of Cataract & Refractive Surgery 30, 2 (2004), 374–380. https://doi.org/10.1016/j.jcrs.2003.06.004
[3]
Carolyn Baylor, Kathryn Yorkston, Tanya Eadie, Jiseon Kim, Hyewon Chung, and Dagmar Amtmann. 2013. The Communicative Participation Item Bank (CPIB): Item bank calibration and development of a disorder-generic short form. (2013).
[4]
V. Bazarevsky, Y. Kartynnik, A. Vakunov, K. Raveendran, and M. Grundmann. 2019. BlazeFace: Sub-millisecond Neural Face Detection on Mobile GPUs. CoRR abs/1907.05047(2019). arXiv:1907.05047http://arxiv.org/abs/1907.05047
[5]
J.P.G. Bergmanson and J.G. Martinez. 2017. Size does matter: what is the corneo-limbal diameter?Clinical and Experimental Optometry 100, 5 (2017), 522–528. https://doi.org/10.1111/cxo.12583
[6]
Donald M Bushnell and Mona L Martin. 1999. Quality of life and Parkinson’s disease: translation and validation of the US Parkinson’s Disease Questionnaire (PDQ-39). Quality of Life Research 8, 4 (1999), 345–350.
[7]
Luis F. Gomez, Aythami Morales, Juan R. Orozco-Arroyave, Roberto Daza, and Julian Fierrez. 2021. Improving Parkinson Detection using Dynamic Features from Evoked Expressions in Video. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). 1562–1570. https://doi.org/10.1109/CVPRW53098.2021.00172
[8]
I. Grishchenko, A. Ablavatski, Y. Kartynnik, K. Raveendran, and M. Grundmann. 2020. Attention Mesh: High-fidelity Face Mesh Prediction in Real-time. CoRR abs/2006.10962(2020). arXiv:2006.10962https://arxiv.org/abs/2006.10962
[9]
R. Gross, I. Matthews, J. Cohn, T. Kanade, and S. Baker. 2010. Multi-PIE. In Proceedings of the International Conference on Automatic Face and Gesture Recognition (FG), Vol. 28. 807–813.
[10]
Diego L. Guarin, Aidan Dempster, Andrea Bandini, Yana Yunusova, and Babak Taati. 2020. Estimation of Orofacial Kinematics in Parkinson’s Disease: Comparison of 2D and 3D Markerless Systems for Motion Tracking. In 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020). 540–543. https://doi.org/10.1109/FG47880.2020.00112
[11]
H. Hashemi, M. Khabazkhoob, M. H. Emamian, M. Shariati, A. Yekta, and A. Fotouhi. 2015. White-to-white corneal diameter distribution in an adult population. Journal of Current Ophthalmology 27, 1 (2015), 21–24. https://doi.org/10.1016/j.joco.2015.09.001
[12]
Y. Kartynnik, A. Ablavatski, I. Grishchenko, and M. Grundmann. 2019. Real-time Facial Surface Geometry from Monocular Video on Mobile GPUs. CoRR abs/1907.06724(2019). arXiv:1907.06724http://arxiv.org/abs/1907.06724
[13]
V. Kazemi and J. Sullivan. 2014. One Millisecond Face Alignment with an Ensemble of Regression Trees. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Columbus, USA.
[14]
H. Kothare, O. Roesler, W. Burke, M. Neumann, J. Liscombe, A. Exner, S. Snyder, A. Cornish, D. Habberstad, D. Pautler, D. Suendermann-Oeft, J. Huber, and V. Ramanarayanan. 2022. Speech, Facial and Fine Motor Features for Conversation-Based Remote Assessment and Monitoring of Parkinson’s Disease. In Proceedings of the 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC). Glasgow, Scotland.
[15]
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C. Berg. 2016. SSD: Single Shot MultiBox Detector. In Proceedings of the European Conference on Computer Vision (ECCV). 21–37.
[16]
Lakmini P Malasinghe, Naeem Ramzan, and Keshav Dahal. 2019. Remote patient monitoring: a comprehensive study. Journal of Ambient Intelligence and Humanized Computing 10, 1 (2019), 57–76.
[17]
Francesco Motolese, Alessandro Magliozzi, Fiorella Puttini, Mariagrazia Rossi, Fioravante Capone, Keren Karlinski, Alit Stark-Inbar, Ziv Yekutieli, Vincenzo Di Lazzaro, and Massimo Marano. 2020. Parkinson’s disease remote patient monitoring during the COVID-19 lockdown. Frontiers in neurology 11 (2020).
[18]
M. Nasir, A. Jati, P. G. Shivakumar, S. N. Chakravarthula, and P. Georgiou. 2016. Multimodal and Multiresolution Depression Detection from Speech and Facial Landmark Features. In Proceedings of the 6th International Workshop on Audio/Visual Emotion Challenge (AVEC). Amsterdam, The Netherlands, 43–50.
[19]
Ziad S Nasreddine, Natalie A Phillips, Valérie Bédirian, Simon Charbonneau, Victor Whitehead, Isabelle Collin, Jeffrey L Cummings, and Howard Chertkow. 2005. The Montreal Cognitive Assessment, MoCA: a brief screening tool for mild cognitive impairment. Journal of the American Geriatrics Society 53, 4 (2005), 695–699.
[20]
Spyros Papapetropoulos, Georgia Mitsi, and Alberto J Espay. 2015. Digital health revolution: is it time for affordable remote monitoring for Parkinson’s disease?Frontiers in neurology 6(2015), 34.
[21]
Vikram Ramanarayanan, Oliver Roesler, Michael Neumann, David Pautler, Doug Habberstad, Andrew Cornish, Hardik Kothare, Vignesh Murali, Jackson Liscombe, Dirk Schnelle-Walka, 2020. Toward Remote Patient Monitoring of Speech, Video, Cognitive and Respiratory Biomarkers Using Multimodal Dialog Technology. In INTERSPEECH. 492–493.
[22]
F. Ruefer, A. Schroeder, and C. Erb. 2005. White-to-White Corneal Diameter. Cornea 24, 3 (2005), 259–261. https://doi.org/10.1097/01.ico.0000148312.01805.53
[23]
D. Suendermann-Oeft, A. Robinson, A. Cornish, D. Habberstad, D. Pautler, D. Schnelle-Walka, F. Haller, J. Liscombe, M. Neumann, M. Merrill, O. Roesler, and R. Geffarth. 2019. NEMSI: A Multimodal Dialog System for Screening of Neurological or Mental Conditions. In Proceedings of ACM International Conference on Intelligent Virtual Agents (IVA). Paris, France.

Cited By

View all
  • (2025)Vocal and Facial Behavior During Affect Production in Autism Spectrum DisorderJournal of Speech, Language, and Hearing Research10.1044/2024_JSLHR-23-0008068:2(419-434)Online publication date: 4-Feb-2025
  • (2024)Best low-cost methods for real-time detection of the eye and gaze trackingi-com10.1515/icom-2023-002623:1(79-94)Online publication date: 8-Jan-2024
  • (2024)Multimodal speech biomarkers for remote monitoring of ALS disease progressionComputers in Biology and Medicine10.1016/j.compbiomed.2024.108949180:COnline publication date: 18-Nov-2024

Index Terms

  1. Exploring Facial Metric Normalization For Within- and Between-Subject Comparisons in a Multimodal Health Monitoring Agent

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Conferences
      ICMI '22 Companion: Companion Publication of the 2022 International Conference on Multimodal Interaction
      November 2022
      225 pages
      ISBN:9781450393898
      DOI:10.1145/3536220
      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

      Sponsors

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 07 November 2022

      Permissions

      Request permissions for this article.

      Check for updates

      Author Tags

      1. Multimodal dialogue system
      2. Normalization
      3. Parkinson’s disease
      4. Remote patient monitoring

      Qualifiers

      • Short-paper
      • Research
      • Refereed limited

      Conference

      ICMI '22
      Sponsor:

      Acceptance Rates

      Overall Acceptance Rate 453 of 1,080 submissions, 42%

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)14
      • Downloads (Last 6 weeks)2
      Reflects downloads up to 18 Feb 2025

      Other Metrics

      Citations

      Cited By

      View all
      • (2025)Vocal and Facial Behavior During Affect Production in Autism Spectrum DisorderJournal of Speech, Language, and Hearing Research10.1044/2024_JSLHR-23-0008068:2(419-434)Online publication date: 4-Feb-2025
      • (2024)Best low-cost methods for real-time detection of the eye and gaze trackingi-com10.1515/icom-2023-002623:1(79-94)Online publication date: 8-Jan-2024
      • (2024)Multimodal speech biomarkers for remote monitoring of ALS disease progressionComputers in Biology and Medicine10.1016/j.compbiomed.2024.108949180:COnline publication date: 18-Nov-2024

      View Options

      Login options

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      HTML Format

      View this article in HTML Format.

      HTML Format

      Figures

      Tables

      Media

      Share

      Share

      Share this Publication link

      Share on social media