skip to main content
10.1145/3577190.3614123acmconferencesArticle/Chapter ViewAbstractPublication Pagesicmi-mlmiConference Proceedingsconference-collections
research-article

Towards Autonomous Physiological Signal Extraction From Thermal Videos Using Deep Learning

Published: 09 October 2023 Publication History

Abstract

Using the thermal modality in order to extract physiological signals as a noncontact means of remote monitoring is gaining traction in applications, such as healthcare monitoring. However, existing methods rely heavily on traditional tracking and mostly unsupervised signal processing methods, which could be affected significantly by noise and subjects’ movements. Using a novel deep learning architecture based on convolutional long short-term memory networks on a diverse dataset of 36 subjects, we present a personalized approach to extract multimodal signals, including the heart rate, respiration rate, and body temperature from thermal videos. We perform multimodal signal extraction for subjects in states of both active speaking and silence, requiring no parameter tuning in an end-to-end deep learning approach with automatic feature extraction. We experiment with different data sampling methods for training our deep learning models, as well as different network designs. Our results indicate the effectiveness and improved efficiency of the proposed models reaching more than 90% accuracy based on the availability of proper training data for each subject.

References

[1]
[n. d.]. Thermal Imaging for Detecting Elevated Body Temperature. https://www.flir.com/discover/public-safety/thermal-imaging-for-detecting-elevated-body-temperature/. Accessed: 2022-03-22.
[2]
Abbas K Abbas, Konrad Heimann, Katrin Jergus, Thorsten Orlikowsky, and Steffen Leonhardt. 2011. Neonatal non-contact respiratory monitoring based on real-time infrared thermography. Biomedical engineering online 10, 1 (2011), 1–17.
[3]
Farhad Ahamed and Farnaz Farid. 2018. Applying Internet of Things and Machine-Learning for Personalized Healthcare: Issues and Challenges. In 2018 International Conference on Machine Learning and Data Engineering (iCMLDE). 19–21. https://doi.org/10.1109/iCMLDE.2018.00014
[4]
Farah Al-Kalidi, Heather Elphick, Reza Saatchi, and Derek Burke. 2015. Respiratory rate measurement in children using a thermal camera. International Journal of Scientific and Engineering Research 6, 4 (2015), 1748–1756.
[5]
Andrea Aliverti. 2017. Wearable technology: role in respiratory health and disease. Breathe 13, 2 (2017), e27–e36. arXiv:https://breathe.ersjournals.com/content/13/2/e27.full.pdfhttps://breathe.ersjournals.com/content/13/2/e27
[6]
Carina Barbosa Pereira, Michael Czaplik, Vladimir Blazek, Steffen Leonhardt, and Daniel Teichmann. 2018. Monitoring of Cardiorespiratory Signals Using Thermal Imaging: A Pilot Study on Healthy Human Subjects. Sensors (Basel, Switzerland) 18, 5 (13 May 2018), 1541. https://doi.org/10.3390/s18051541 29757248[pmid].
[7]
Shahina Begum. 2013. Intelligent driver monitoring systems based on physiological sensor signals: A review. In 16th International IEEE Conference on Intelligent Transportation Systems (ITSC 2013). 282–289. https://doi.org/10.1109/ITSC.2013.6728246
[8]
Clark Bowman, Yitong Huang, Olivia J. Walch, Yu Fang, Elena Frank, Jonathan Tyler, Caleb Mayer, Christopher Stockbridge, Cathy Goldstein, Srijan Sen, and Daniel B. Forger. 2021. A method for characterizing daily physiology from widely used wearables. Cell Reports Methods 1, 4 (2021), 100058. https://doi.org/10.1016/j.crmeth.2021.100058
[9]
Eli M Cahan, Tina Hernandez-Boussard, Sonoo Thadaney-Israni, and Daniel L Rubin. 2019. Putting the data before the algorithm in big data addressing personalized healthcare. NPJ Digital Medicine 2, 1 (2019), 1–6.
[10]
Daniela Cardone, Paola Pinti, and Arcangelo Merla. 2015. Thermal infrared imaging-based computational psychophysiology for psychometrics. Computational and mathematical methods in medicine 2015 (2015).
[11]
Duan-Yu Chen, Huei-Siang Zou, and An-Ting Hsieh. 2020. Thermal Image based Remote Heart Rate Measurement on Dynamic Subjects Using Deep Learning. In 2020 IEEE International Conference on Consumer Electronics - Taiwan (ICCE-Taiwan). 1–2. https://doi.org/10.1109/ICCE-Taiwan49838.2020.9258129
[12]
Irving A. Cruz-Albarran, Juan P. Benitez-Rangel, Roque A. Osornio-Rios, and Luis A. Morales-Hernandez. 2017. Human emotions detection based on a smart-thermal system of thermographic images. Infrared Physics & Technology 81 (2017), 250–261. https://doi.org/10.1016/j.infrared.2017.01.002
[13]
Robert David, Jared Duke, Advait Jain, Vijay Janapa Reddi, Nat Jeffries, Jian Li, Nick Kreeger, Ian Nappier, Meghna Natraj, Shlomi Regev, Rocky Rhodes, Tiezhen Wang, and Pete Warden. 2020. TensorFlow Lite Micro: Embedded Machine Learning on TinyML Systems. https://doi.org/10.48550/ARXIV.2010.08678
[14]
H. E. Elphick, A. H. Alkali, R. K. Kingshott, D. Burke, and R. Saatchi. 2019. Exploratory Study to Evaluate Respiratory Rate Using a Thermal Imaging Camera. Respiration 97, 3 (2019), 205–212. https://doi.org/10.1159/000490546
[15]
Travis Hall, Donald Y. C. Lie, Tam Q. Nguyen, Jill C. Mayeda, Paul E. Lie, Jerry Lopez, and Ron E. Banister. 2017. Non-Contact Sensor for Long-Term Continuous Vital Signs Monitoring: A Review on Intelligent Phased-Array Doppler Sensor Design. Sensors 17, 11 (2017). https://doi.org/10.3390/s17112632
[16]
Jun Han and Claudio Moraga. 1995. The influence of the sigmoid function parameters on the speed of backpropagation learning. In From Natural to Artificial Neural Computation, José Mira and Francisco Sandoval (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 195–201.
[17]
Kirk Havens. 2015. Thermal Imaging Techniques to Survey and Monitor Animals in the Wild : a Methodology. Academic Press Imprint,Elsevier Science & Technology Books, San Diego.
[18]
Christian Hessler, Mohamed Abouelenien, and Mihai Burzo. 2020. A Non-contact Method for Extracting Heart and Respiration Rates. In 2020 17th Conference on Computer and Robot Vision (CRV). 1–8. https://doi.org/10.1109/CRV50864.2020.00009
[19]
Menghan Hu, Guangtao Zhai, Duo Li, Yezhao Fan, Huiyu Duan, Wenhan Zhu, and Xiaokang Yang. 2018. Combination of near-infrared and thermal imaging techniques for the remote and simultaneous measurements of breathing and heart rates under sleep situation. PloS one 13, 1 (2018), e0190466.
[20]
Prasara Jakkaew and Takao Onoye. 2020. Non-Contact Respiration Monitoring and Body Movements Detection for Sleep Using Thermal Imaging. Sensors (Basel, Switzerland) 20, 21 (05 Nov 2020), 6307. https://doi.org/10.3390/s20216307 33167556[pmid].
[21]
Kevin B. Johnson, Wei-Qi Wei, Dilhan Weeraratne, Mark E. Frisse, Karl Misulis, Kyu Rhee, Juan Zhao, and Jane L. Snowdon. 2021. Precision Medicine, AI, and the Future of Personalized Health Care. Clinical and Translational Science 14, 1 (2021), 86–93. https://doi.org/10.1111/cts.12884 arXiv:https://ascpt.onlinelibrary.wiley.com/doi/pdf/10.1111/cts.12884
[22]
Yoonkyoung Kim, Yosep Park, and Eui Chul Lee. 2018. Remote Heart Rate Monitoring Method Using Infrared Thermal Camera.
[23]
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, Yoshua Bengio and Yann LeCun (Eds.). http://arxiv.org/abs/1412.6980
[24]
Alicja Kwasniewska, Jacek Ruminski, and Maciej Szankin. 2019. Improving Accuracy of Contactless Respiratory Rate Estimation by Enhancing Thermal Sequences with Deep Neural Networks. Applied Sciences 9, 20 (2019). https://doi.org/10.3390/app9204405
[25]
B.B. Lahiri, S. Bagavathiappan, T. Jayakumar, and John Philip. 2012. Medical applications of infrared thermography: A review. Infrared Physics & Technology 55, 4 (2012), 221–235. https://doi.org/10.1016/j.infrared.2012.03.007
[26]
Steffen Leonhardt, Lennart Leicht, and Daniel Teichmann. 2018. Unobtrusive Vital Sign Monitoring in Automotive Environments—A Review. Sensors 18, 9 (2018). https://doi.org/10.3390/s18093080
[27]
Magdalena Lewandowska, Jacek Rumiński, Tomasz Kocejko, and Jędrzej Nowak. 2011. Measuring pulse rate with a webcam — A non-contact method for evaluating cardiac activity. In 2011 Federated Conference on Computer Science and Information Systems (FedCSIS). 405–410.
[28]
Miguel Bordallo Lopez, Carlos R. del Blanco, and Narciso Garcia. 2017. Detecting exercise-induced fatigue using thermal imaging and deep learning. In 2017 Seventh International Conference on Image Processing Theory, Tools and Applications (IPTA). 1–6. https://doi.org/10.1109/IPTA.2017.8310151
[29]
Martin Clinton Tosima Manullang, Yuan-Hsiang Lin, Sheng-Jie Lai, and Nai-Kuan Chou. 2021. Implementation of Thermal Camera for Non-Contact Physiological Measurement: A Systematic Review. Sensors 21, 23 (2021). https://doi.org/10.3390/s21237777
[30]
Jiři Mekyska, Virginia Espinosa-Duró, and Marcos Faundez-Zanuy. 2010. Mekyska. In 44th Annual 2010 IEEE International Carnahan Conference on Security Technology. 185–189. https://doi.org/10.1109/CCST.2010.5678709
[31]
Masahiro Miyaji, Haruki Kawanaka, and Koji Oguri. 2009. Driver’s cognitive distraction detection using physiological features by the adaboost. In 2009 12th International IEEE Conference on Intelligent Transportation Systems. 1–6. https://doi.org/10.1109/ITSC.2009.5309881
[32]
Jermana L. Moraes, Matheus X. Rocha, Glauber G. Vasconcelos, José E. Vasconcelos Filho, Victor Hugo C. De Albuquerque, and Auzuir R. Alexandria. 2018. Advances in Photopletysmography Signal Analysis for Biomedical Applications. Sensors 18, 6 (2018). https://doi.org/10.3390/s18061894
[33]
Subhas Chandra Mukhopadhyay. 2015. Wearable Sensors for Human Activity Monitoring: A Review. IEEE Sensors Journal 15, 3 (2015), 1321–1330. https://doi.org/10.1109/JSEN.2014.2370945
[34]
Yosuke Nakayama, Guanghao Sun, Shigeto Abe, and Takemi Matsui. 2015. Non-contact measurement of respiratory and heart rates using a CMOS camera-equipped infrared camera for prompt infection screening at airport quarantine stations. In 2015 IEEE International Conference on Computational Intelligence and Virtual Environments for Measurement Systems and Applications (CIVEMSA). 1–4. https://doi.org/10.1109/CIVEMSA.2015.7158595
[35]
S Navaneeth, S Sarath, B Amba Nair, K Harikrishnan, and P Prajal. 2020. A Deep-Learning Approach to Find Respiratory Syndromes in Infants using Thermal Imaging. In 2020 International Conference on Communication and Signal Processing (ICCSP). 0498–0501. https://doi.org/10.1109/ICCSP48568.2020.9182231
[36]
Andrea Nicolò, Carlo Massaroni, Emiliano Schena, and Massimo Sacchetti. 2020. The Importance of Respiratory Rate Monitoring: From Healthcare to Sport and Exercise. Sensors 20, 21 (2020). https://doi.org/10.3390/s20216396
[37]
Keiron O’Shea and Ryan Nash. 2015. An Introduction to Convolutional Neural Networks. CoRR abs/1511.08458 (2015). arXiv:1511.08458http://arxiv.org/abs/1511.08458
[38]
David Perpetuini, Andrea Di Credico, Chiara Filippini, Pascal Izzicupo, Daniela Cardone, Piero Chiacchiaretta, Barbara Ghinassi, Angela Di Baldassarre, and Arcangelo Merla. 2021. Is It Possible to Estimate Average Heart Rate from Facial Thermal Imaging?Engineering Proceedings 8, 1 (2021). https://doi.org/10.3390/engproc2021008010
[39]
Alex Sherstinsky. 2018. Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network. CoRR abs/1808.03314 (2018). arXiv:1808.03314http://arxiv.org/abs/1808.03314
[40]
Xingjian Shi, Zhourong Chen, Hao Wang, Dit-Yan Yeung, Wai-kin Wong, and Wang-chun Woo. 2015. Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting. In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1 (Montreal, Canada) (NIPS’15). MIT Press, Cambridge, MA, USA, 802–810.
[41]
Shuangbao Shu, Huajun Liang, Yu Zhang, Yuzhong Zhang, and Ziqiang Yang. 2022. Non-contact measurement of human respiration using an infrared thermal camera and the deep learning method. Measurement Science and Technology (2022). http://iopscience.iop.org/article/10.1088/1361-6501/ac5ed9
[42]
Riccardo Sioni and Luca Chittaro. 2015. Stress Detection Using Physiological Sensors. Computer 48, 10 (2015), 26–33. https://doi.org/10.1109/MC.2015.316
[43]
Muhammad Usman, Ruth Evans, Reza Saatchi, Ruth Kingshott, and Heather Elphick. 2019. Non-invasive respiration monitoring by thermal imaging to detect sleep apnoea. (2019).
[44]
Mauricio Villarroel, Sitthichok Chaichulee, João Jorge, Sara Davis, Gabrielle Green, Carlos Arteta, Andrew Zisserman, Kenny McCormick, Peter Watkinson, and Lionel Tarassenko. 2019. Non-contact physiological monitoring of preterm infants in the Neonatal Intensive Care Unit. npj Digital Medicine (2019).

Index Terms

  1. Towards Autonomous Physiological Signal Extraction From Thermal Videos Using Deep Learning
                  Index terms have been assigned to the content through auto-classification.

                  Recommendations

                  Comments

                  Information & Contributors

                  Information

                  Published In

                  cover image ACM Conferences
                  ICMI '23: Proceedings of the 25th International Conference on Multimodal Interaction
                  October 2023
                  858 pages
                  ISBN:9798400700552
                  DOI:10.1145/3577190
                  Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

                  Sponsors

                  Publisher

                  Association for Computing Machinery

                  New York, NY, United States

                  Publication History

                  Published: 09 October 2023

                  Permissions

                  Request permissions for this article.

                  Check for updates

                  Author Tags

                  1. deep learning
                  2. machine learning
                  3. multimodal dataset
                  4. physiological signals
                  5. realtime prediction
                  6. thermal imaging

                  Qualifiers

                  • Research-article
                  • Research
                  • Refereed limited

                  Conference

                  ICMI '23
                  Sponsor:

                  Acceptance Rates

                  Overall Acceptance Rate 453 of 1,080 submissions, 42%

                  Contributors

                  Other Metrics

                  Bibliometrics & Citations

                  Bibliometrics

                  Article Metrics

                  • 0
                    Total Citations
                  • 134
                    Total Downloads
                  • Downloads (Last 12 months)55
                  • Downloads (Last 6 weeks)3
                  Reflects downloads up to 17 Feb 2025

                  Other Metrics

                  Citations

                  View Options

                  Login options

                  View options

                  PDF

                  View or Download as a PDF file.

                  PDF

                  eReader

                  View online with eReader.

                  eReader

                  HTML Format

                  View this article in HTML Format.

                  HTML Format

                  Figures

                  Tables

                  Media

                  Share

                  Share

                  Share this Publication link

                  Share on social media