skip to main content
10.1145/3581791.3596848acmconferencesArticle/Chapter ViewAbstractPublication PagesmobisysConference Proceedingsconference-collections
research-article
Public Access

Passive Vital Sign Monitoring via Facial Vibrations Leveraging AR/VR Headsets

Published: 18 June 2023 Publication History

Abstract

Vital signs (e.g., breathing and heart rates) and personal identities are essential information for personalized medicine and healthcare. The popularity of augmented reality/virtual reality (AR/VR) provides an excellent opportunity for enabling long-term health monitoring in a broad range of scenarios, including virtual entertainment, education, and telemedicine. However, commercial-off-the-shelf AR/VR devices do not have dedicated biosensors for providing vital signs and personal identities. In this work, we propose a novel framework that can generate fine-grained vital sign signals and other personalized health information of an AR/VR user through passive sensing on AR/VR devices. In particular, we find that the user's minute facial vibrations induced by breathing and heart beating can impact the readily available motion sensors on AR/VR headsets, which encode rich vital sign patterns and unique biometrics. The proposed framework further estimates the breathing and heartbeat rates, detects the gender and identity, and derives the body fat percentage of the user. To mitigate the impacts of body movement, we design an adaptive filtering scheme to cancel the spontaneous and non-spontaneous motion artifacts. We also develop unique facial vibration features and deep learning techniques to facilitate vital sign signal reconstruction and user identification. Extensive experiments demonstrate that our framework can achieve a low error of vital sign signal reconstruction and rate measurement, along with 95.51% and 93.33% accuracy on identity and gender recognition.

References

[1]
2019. Vital Signs (Body Temperature, Pulse Rate, Respiration Rate, Blood Pressure). https://www.hopkinsmedicine.org/health/conditions-and-diseases/vital-signs-body-temperature-pulse-rate-respiration-rate-blood-pressure.
[2]
2020. What's a normal resting heart rate? - Mayo Clinic. https://www.mayoclinic.org/healthy-lifestyle/fitness/expert-answers/heart-rate/.
[3]
2021. Gender and health. https://www.who.int/news-room/questions-and-answers/item/gender-and-health.
[4]
2022. Metaverse. https://en.wikipedia.org/wiki/Metaverse.
[5]
2022. Virtual Reality Market Size Worth USD 43.01 Billion in 2028. https://www.emergenresearch.com/press-release/global-virtual-reality-market.
[6]
2022. Virtual reality sickness. https://en.wikipedia.org/wiki/Virtual_reality_sickness.
[7]
M. Akman, M.K. Uçar, Z. Uçar, K. Uçar, B. Baraklı, and M.R. Bozkurt. 2022. Determination of Body Fat Percentage by Gender Based with Photoplethysmography Signal Using Machine Learning Algorithm. IRBM 43, 3 (2022), 169--186.
[8]
Musaed Alhussein and Ghulam Muhammad. 2018. Voice Pathology Detection Using Deep Learning on Mobile Healthcare Framework. IEEE Access 6 (2018), 41034--41041.
[9]
S Abhishek Anand and Nitesh Saxena. 2018. Speechless: Analyzing the Threat to Speech Privacy from Smartphone Motion Sensors. In 2018 IEEE Symposium on Security and Privacy (SP). 1000--1017.
[10]
Anatomy and Physiology. 2019. Cardiac Cycle. http://library.open.oregonstate.edu/aandp/chapter/19-3-cardiac-cycle/.
[11]
Jeffrey Bloom, Michael J. Lopez, and Appaji Rayi. 2021. Anatomy, Head and Neck, Eye Levator Labii Superioris Muscle. StatPearls Publishing, Treasure Island (FL). http://europepmc.org/books/NBK541031
[12]
Google Cardboard. 2014. Cardboard - Google VR. https://arvr.google.com/cardboard/.
[13]
Eunhee Chang, Hyun Taek Kim, and Byounghyun Yoo. 2022. Identifying physiological correlates of cybersickness using heartbeat-evoked potential analysis. Virtual Reality 26, 3 (01 Sep 2022), 1193--1205.
[14]
ClassVR. 2022. ClassVR: virtual reality for schools. https://www.classvr.com/us/.
[15]
Hao Dang, Muyi Sun, Guanhong Zhang, Xingqun Qi, Xiaoguang Zhou, and Qing Chang. 2019. A Novel Deep Arrhythmia-Diagnosis Network for Atrial Fibrillation Classification Using Electrocardiogram Signals. IEEE Access 7 (2019), 75577--75590.
[16]
Anne E Dixon and Ubong Peters. 2018. The effect of obesity on lung function. Expert Rev Respir Med 12, 9 (Aug. 2018), 755--767.
[17]
VR Doctors. 2022. VR Doctors - change reality, change healthcare. http://vrdoctors.net/.
[18]
Biyi Fang, Nicholas D. Lane, Mi Zhang, Aidan Boran, and Fahim Kawsar. 2016. BodyScan: Enabling Radio-Based Sensing on Wearable Devices for Contactless Activity and Vital Sign Monitoring (MobiSys '16). Association for Computing Machinery, New York, NY, USA, 97--110.
[19]
Recovery for Athletes. 2022. InBody 570 Body Composition Analyzer. https://www.recoveryforathletes.com/products/inbody-570-body-composition-analyzer.
[20]
Jann Philipp Freiwald, Yvonne Göbel, Fariba Mostajeran, and Frank Steinicke. 2020. The Cybersickness Susceptibility Questionnaire: Predicting Virtual Reality Tolerance. In Proceedings of Mensch Und Computer 2020 (Magdeburg, Germany) (MuC '20). Association for Computing Machinery, New York, NY, USA, 115--118.
[21]
Markus Funk, Karola Marky, Iori Mizutani, Mareike Kritzler, Simon Mayer, and Florian Michahelles. 2019. Lookunlock: Using spatial-targets for user-authentication on hmds. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems. 1--6.
[22]
Ceenu George, Mohamed Khamis, Emanuel von Zezschwitz, Marinus Burger, Henri Schmidt, Florian Alt, and Heinrich Hussmann. 2017. Seamless and secure vr: Adapting and evaluating established authentication systems for virtual reality. NDSS.
[23]
Azadeh Hadadi, Christophe Guillet, Jean-Rémy Chardonnet, Mikhail Langovoy, Yuyang Wang, and Jivka Ovtcharova. 2022. Prediction of cybersickness in virtual environments using topological data analysis and machine learning. Frontiers in Virtual Reality 3 (2022).
[24]
HealthData.gov. 2022. HealthData.gov. https://healthdata.gov//.
[25]
Andrii Horiachko. 2022. AR and VR for Video Conferencing. https://www.softermii.com/blog/ar-and-vr-for-video-conferencing.
[26]
Lennart Leicht, Pascal Vetter, Steffen Leonhardt, and Daniel Teichmann. 2017. The PhysioBelt: A safety belt integrated sensor system for heart activity and respiration. In 2017 IEEE International Conference on Vehicular Electronics and Safety (ICVES). 191--195.
[27]
Boning Li and Akane Sano. 2020. Extraction and Interpretation of Deep Autoencoder-Based Temporal Features from Wearables for Forecasting Personalized Mood, Health, and Stress. 4, 2, Article 49 (jun 2020), 26 pages.
[28]
Jian Liu, Yan Wang, Yingying Chen, Jie Yang, Xu Chen, and Jerry Cheng. 2015. Tracking Vital Signs During Sleep Leveraging Off-the-Shelf WiFi. In Proceedings of the 16th ACM International Symposium on Mobile Ad Hoc Networking and Computing (Hangzhou, China) (MobiHoc '15). Association for Computing Machinery, New York, NY, USA, 267--276.
[29]
Martin László, Zoltán Istenes, and Adam Tarcsi. 2020. Extracting Physiological Signals from Smartphone Sensors.
[30]
Francisco Martín-Rodríguez, José Luis Martín Conty, Verónica Casado Vicente, Pedro Arnillas Gómez, Alicia Mohedano-Moriano, and Miguel Ángel Castro Villamor. 2018. Does gender influence physiological tolerance in resuscitators when using personal protection equipment against biological hazards? Emergency Medicine International 2018 (2018), 1--7.
[31]
Carey R. Merritt, H. Troy Nagle, and Edward Grant. 2009. Textile-Based Capacitive Sensors for Respiration Monitoring. IEEE Sensors Journal 9, 1 (2009), 71--78.
[32]
Meta. 2021. How AR is making video calling more collaborative. https://tech.facebook.com/reality-labs/2021/12/how-ar-is-making-video-calling-more-collaborative/.
[33]
Meta. 2021. Unleash your creativity through VR art, paintings sculptures. https://www.oculus.com/blog/unleash-your-creativity-through-vr-art//.
[34]
Abdul Momin, Saheli Bhattacharya, Sudip Sanyal, and Pavan Chakraborty. 2020. Visual Attention, Mental Stress and Gender: A Study Using Physiological Signals. IEEE Access 8 (2020), 165973--165988.
[35]
Neulog. 2011. Heart Rate Pulse logger sensor NUL-208. https://neulog.com/heart-rate-pulse/.
[36]
Neulog. 2011. Respiration Monitor Belt logger sensor NUL-236. https://neulog.com/respiration-monitor-belt/.
[37]
Keith Nolan, Aidan Mooney, and Susan Bergin. 2019. An Investigation of Gender Differences in Computer Science Using Physiological, Psychological and Behavioural Metrics. In Proceedings of the Twenty-First Australasian Computing Education Conference (Sydney, NSW, Australia) (ACE '19). Association for Computing Machinery, New York, NY, USA, 47--55.
[38]
Oculus. 2020. Oculus Quest Store: VR Games, Apps, and More. https://www.oculus.com/experiences/quest/.
[39]
Heeseok Oh and Wookho Son. 2022. Cybersickness and Its Severity Arising from Virtual Reality Content: A Comprehensive Study. Sensors (Basel) 22, 4 (Feb. 2022).
[40]
Donald B. Percival and Andrew T. Walden. 1993. Spectral Analysis for Physical Applications. Cambridge University Press.
[41]
Richard A. Davis Peter J. Brockwell. 1991. Time Series: Theory and Methods. Springer New York, NY.
[42]
Caitlin Polley, Titus Jayarathna, Upul Gunawardana, Ganesh Naik, Tara Hamilton, Emilio Andreozzi, Paolo Bifulco, Daniele Esposito, Jessica Centracchio, and Gaetano Gargiulo. 2021. Wearable Bluetooth Triage Healthcare Monitoring System. Sensors 21, 22 (2021).
[43]
Possible. 2022. Can Excess Weight Have Major Impact On Your Blood Circulation? http://https://possible.in/can-excess-weight-have-major-impact-on-your-blood-circulation.html/.
[44]
Chenxin Qu, Xiaoping Che, Siqi Ma, and Shuqin Zhu. 2022. Bio-physiological-signals-based VR cybersickness detection. CCF Transactions on Pervasive Computing and Interaction 4, 3 (01 Sep 2022), 268--284.
[45]
RENPHO. 2022. Best Body Fat and Body Composition Analyzer Scale, Renpho. https://renpho.com/collections/renpho-scales.
[46]
S. Sabeeha and C. Shiny. 2017. ECG-based heartbeat classification for disease diagnosis. In 2017 International Conference on Computing Methodologies and Communication (ICCMC). 1113--1117.
[47]
E. Sardini and M. Serpelloni. 2010. Instrumented wearable belt for wireless health monitoring. Procedia Engineering 5 (2010), 580--583. Eurosensor XXIV Conference.
[48]
Ali H. Sayed. 2003. Fundamentals Of Adaptive Filtering.
[49]
M. Scarpetta, M. Spadavecchia, G. Andria, M.A. Ragolia, and N. Giaquinto. 2022. Accurate simultaneous measurement of heartbeat and respiratory intervals using a smartphone. 17, 07 (jul 2022), P07020.
[50]
Volkan Sevinc and Mehmet Ilker Berkman. 2020. Psychometric evaluation of Simulator Sickness Questionnaire and its variants as a measure of cybersickness in consumer virtual environments. Applied Ergonomics 82 (2020), 102958.
[51]
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018. Self-Attention with Relative Position Representations.
[52]
Cong Shi, Xiangyu Xu, Tianfang Zhang, Payton Walker, Yi Wu, Jian Liu, Nitesh Saxena, Yingying Chen, and Jiadi Yu. 2021. Face-Mic: Inferring Live Speech and Speaker Identity via Subtle Facial Dynamics Captured by AR/VR Motion Sensors (MobiCom '21). Association for Computing Machinery, New York, NY, USA, 478--490.
[53]
Abdulhamit Subasi, Mariam Radhwan, Rabea Kurdi, and Kholoud Khateeb. 2018. IoT based mobile healthcare system for human activity recognition. In 2018 15th Learning and Technology Conference (LT). 29--34.
[54]
Tim J.T. Sutherland, Christene R. McLachlan, Malcolm R. Sears, Richie Poulton, and Robert J. Hancox. 2016. The relationship between body fat and respiratory function in young adults. European Respiratory Journal 48, 3 (2016), 734--747. arXiv:https://erj.ersjournals.com/content/48/3/734.full.pdf
[55]
H. Emrah Tasli, Amogh Gudi, and Marten den Uyl. 2014. Remote PPG based vital sign measurement using adaptive facial regions. In 2014 IEEE International Conference on Image Processing (ICIP).
[56]
Osamu Tochikubo, Eiji Miyajima, Tomohiko Shigemasa, and Masao Ishii. 1999. Relation Between Body Fat-Corrected ECG Voltage and Ambulatory Blood Pressure in Patients With Essential Hypertension. Hypertension 33, 5 (May 1999), 1159--1163.
[57]
UltraLeap. 2022. What is VR training? https://www.ultraleap.com/company/news/blog/vr-training/.
[58]
Muhammed Kürşad Uçar, Zeliha Uçar, Kübra Uçar, Mehmet Akman, and Mehmet Recep Bozkurt. 2021. Determination of body fat percentage by electrocardiography signal with gender based artificial intelligence. Biomedical Signal Processing and Control 68 (2021), 102650.
[59]
Michael Vallance and Phillip A. Towndrow. 2022. Perspective: Narrative Storyliving in Virtual Reality Design. Frontiers in Virtual Reality 3 (2022).
[60]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. arXiv:1706.03762 [cs.CL]
[61]
HTC Vive. 2019. VIVE Pro Eye Overview, VIVE United States. https://www.vive.com/us/product/vive-pro-eye/overview/.
[62]
Danling Wang, Qifeng Zhang, Md. Razuan Hossain, and Michael Johnson. 2018. High Sensitive Breath Sensor Based on Nanostructured K2W7O22 for Detection of Type 1 Diabetes. IEEE Sensors Journal 18, 11 (2018), 4399--4404.
[63]
Xuyu Wang, Chao Yang, and Shiwen Mao. 2017. PhaseBeat: Exploiting CSI Phase Data for Vital Sign Monitoring with Commodity WiFi Devices. In 2017 IEEE 37th International Conference on Distributed Computing Systems (ICDCS). 1230--1239.
[64]
Xue Wang and Yang Zhang. 2021. Nod to Auth: Fluent AR/VR Authentication with User Head-Neck Modeling. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems. 1--7.
[65]
Waqas Wazir, Hasan Ali Khattak, Ahmad Almogren, Mudassar Ali Khan, and Ikram Ud Din. 2020. Doodle-based authentication technique using augmented reality. IEEE Access 8 (2020), 4022--4034.
[66]
Dhruv Kumar Yadav, Beatrice Ionascu, Sai Vamsi Krishna Ongole, Aditi Roy, and Nasir Memon. 2015. Design and analysis of shoulder surfing resistant pin based authentication mechanisms on google glass. In International conference on financial cryptography and data security. Springer, 281--297.
[67]
Tianyue Zheng, Zhe Chen, Chao Cai, Jun Luo, and Xu Zhang. 2020. V2iFi: In-Vehicle Vital Sign Monitoring via Compact RF Sensing. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 4, 2, Article 70 (jun 2020), 27 pages.

Cited By

View all
  • (2024)Artificial Intelligence of Things: A SurveyACM Transactions on Sensor Networks10.1145/369063921:1(1-75)Online publication date: 30-Aug-2024
  • (2024)EchoPFLProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36435608:1(1-22)Online publication date: 6-Mar-2024
  • (2024)Virtual Keymysteries Unveiled: Detecting Keystrokes in VR with External Side-Channels2024 IEEE Security and Privacy Workshops (SPW)10.1109/SPW63631.2024.00031(260-266)Online publication date: 23-May-2024
  • Show More Cited By

Index Terms

  1. Passive Vital Sign Monitoring via Facial Vibrations Leveraging AR/VR Headsets

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    MobiSys '23: Proceedings of the 21st Annual International Conference on Mobile Systems, Applications and Services
    June 2023
    651 pages
    ISBN:9798400701108
    DOI:10.1145/3581791
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Sponsors

    In-Cooperation

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 18 June 2023

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. health monitoring
    2. facial vibrations
    3. AR/VR headsets

    Qualifiers

    • Research-article

    Funding Sources

    Conference

    MobiSys '23
    Sponsor:

    Acceptance Rates

    MobiSys '23 Paper Acceptance Rate 41 of 198 submissions, 21%;
    Overall Acceptance Rate 274 of 1,679 submissions, 16%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)419
    • Downloads (Last 6 weeks)29
    Reflects downloads up to 30 Jan 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Artificial Intelligence of Things: A SurveyACM Transactions on Sensor Networks10.1145/369063921:1(1-75)Online publication date: 30-Aug-2024
    • (2024)EchoPFLProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36435608:1(1-22)Online publication date: 6-Mar-2024
    • (2024)Virtual Keymysteries Unveiled: Detecting Keystrokes in VR with External Side-Channels2024 IEEE Security and Privacy Workshops (SPW)10.1109/SPW63631.2024.00031(260-266)Online publication date: 23-May-2024
    • (2024)Reversing the Virtual Maze: An Overview of the Technical and Methodological Challenges for Metaverse App Analysis2024 IEEE International Conference on Metaverse Computing, Networking, and Applications (MetaCom)10.1109/MetaCom62920.2024.00038(173-181)Online publication date: 12-Aug-2024

    View Options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Login options

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media