skip to main content
10.1145/3197768.3197792acmotherconferencesArticle/Chapter ViewAbstractPublication PagespetraConference Proceedingsconference-collections
research-article

A Survey on Extracting Physiological Measurements from Thermal Images

Published: 26 June 2018 Publication History

Abstract

Multiple techniques are used to extract physiological signals from the human body. These signals provide a reliable method to identify the physical and mental state of a person at any given point in time. However, these techniques require contact and cooperation of the individual as well as human effort for connecting the devices and collecting the needed measurement. Moreover, these methods can be invasive, time-consuming, and infeasible in many cases. Recent efforts have been made in order to find alternatives to extract these measurements using non-contact and efficient techniques. In this paper we provide a survey that explores different approaches for extracting vital signs from thermal images as well as review applications that could potentially leverage these techniques.

References

[1]
Mohamed Abouelenien, Mihai Burzo, and Rada Mihalcea. 2015. Cascaded Multimodal Analysis of Alertness Related Features for Drivers Safety Applications. In Proceedings of the 8th ACM International Conference on PErvasive Technologies Related to Assistive Environments (PETRA '15). ACM, New York, NY, USA, Article 59, 8 pages.
[2]
Mohamed Abouelenien, Mihai Burzo, and Rada Mihalcea. 2016. Human acute stress detection via integration of physiological signals and thermal imaging. In Proceedings of the 9th ACM International Conference on PErvasive Technologies Related to Assistive Environments. ACM, 32.
[3]
Mohamed Abouelenien, Rada Mihalcea, and Mihai Burzo. 2016. Analyzing Thermal and Visual Clues of Deception for a Non-Contact Deception Detection Approach. In Proceedings of the 9th ACM International Conference on PErvasive Technologies Related to Assistive Environments. ACM, 35.
[4]
Mohamed Abouelenien, Veronica Pérez-Rosas, Rada Mihalcea, and Mihai Burzo. 2014. Deception Detection Using a Multimodal Approach. In Proceedings of the 16th International Conference on Multimodal Interaction (ICMI '14). ACM, New York, NY, USA, 58--65.
[5]
Mohamed Abouelenien, Verónica Pérez-Rosas, Rada Mihalcea, and Mihai Burzo. 2017. Multimodal Gender Detection. In Proceedings of the 19th ACM International Conference on Multimodal Interaction (ICMI 2017). ACM, New York, NY, USA, 302--311.
[6]
Mohamed Abouelenien, Verónica Pérez-Rosas, Bohan Zhao, Rada Mihalcea, and Mihai Burzo. 2017. Gender-based Multimodal Deception Detection. In Proceedings of the Symposium on Applied Computing (SAC '17). ACM, New York, NY, USA, 137--144.
[7]
M. Abouelenien, V. PÃI'rez-Rosas, R. Mihalcea, and M. Burzo. 2017. Detecting Deceptive Behavior via Integration of Discriminative Features From Multiple Modalities. IEEE Transactions on Information Forensics and Security 12, 5 (May 2017), 1042--1055.
[8]
M. Abouelenien and X. Yuan. 2012. SampleBoost: Improving boosting performance by destabilizing weak learners based on weighted error analysis. In Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012). 585--588.
[9]
Abdulkadir Hamidu Alkali, Reza Saatchi, Heather Elphick, and Derek Burke. 2013. Facial tracking in thermal images for real-time noncontact respiration rate monitoring. In Modelling Symposium (EMS), 2013 European. IEEE, 265--270.
[10]
Stephanie L Bennett, Rafik Goubran, and Frank Knoefel. 2016. Adaptive eulerian video magnification methods to extract heart rate from thermal video. In Medical Measurements and Applications (MeMeA), 2016 IEEE International Symposium on. IEEE, 1--5.
[11]
Stephanie L Bennett, Rafik Goubran, and Frank Knoefel. 2017. Comparison of motion-based analysis to thermal-based analysis of thermal video in the extraction of respiration patterns. In Engineering in Medicine and Biology Society (EMBC), 2017 39th Annual International Conference of the IEEE. IEEE, 3835--3839.
[12]
Richard A Bryant, Mark Creamer, Meaghan O'Donnell, Derrick Silove, and Alexander C McFarlane. 2008. A multisite study of initial respiration rate and heart rate as predictors of posttraumatic stress disorder. The Journal of Clinical Psychiatry (2008).
[13]
Pradeep Buddharaju, Ioannis T Pavlidis, Panagiotis Tsiamyrtzis, and Mike Bazakos. 2007. Physiology-based face recognition in the thermal infrared spectrum. IEEE transactions on pattern analysis and machine intelligence 29, 4 (2007), 613--626.
[14]
Mihai Burzo, Mohamed Abouelenien, Veronica Perez-Rosas, Cakra Wicaksono, Yong Tao, and Rada Mihalcea. 2014. Using Infrared Thermography and Biosensors to Detect Thermal Discomfort in a BuildingâĂŹs Inhabitants. In ASME 2014 International Mechanical Engineering Congress and Exposition. American Society of Mechanical Engineers, V06BT07A015-V06BT07A015.
[15]
Nisha Charkoudian. 2003. Skin blood flow in adult human thermoregulation: how it works, when it does not, and why. In Mayo Clinic Proceedings, Vol. 78, no. 5. Elsevier, 603--612.
[16]
Sergey Y Chekmenev, Aly A Farag, William M Miller, Edward A Essock, and Aruni Bhatnagar. 2009. Multiresolution approach for noncontact measurements of arterial pulse using thermal imaging. In Augmented vision perception in infrared. Springer, 87--112.
[17]
Michelle A Cretikos, Rinaldo Bellomo, Ken Hillman, Jack Chen, Simon Finfer, and Arthas Flabouris. 2008. Respiratory rate: the neglected vital sign. Medical Journal of Australia 188, 11 (2008), 657.
[18]
Michael Dawson, Anne Schell, and Diane Filion. 2007. The electrodermal system. Handbook of psychophysiology 2 (2007), 200--223.
[19]
Maarten Derksen. 2012. Control and resistance in the psychology of lying. Theory and Psychology 22, 2 (2012), 196--212.
[20]
Jin Fei and Ioannis Pavlidis. 2010. Thermistor at a distance: unobtrusive measurement of breathing. IEEE Transactions on Biomedical Engineering 57, 4 (2010), 988--998.
[21]
Bernd Figner, Ryan Murphy, et al. 2011. Using skin conductance in judgment and decision making research. A handbook of process tracing methods for decision research (2011), 163--184.
[22]
Theresa Gannon, Anthony Beech, and Tony Ward. 2009. Risk Assessment and the Polygraph. Nova Science Publishers, 129--154.
[23]
Tia Gao, Dan Greenspan, Matt Welsh, Radford R Juang, and Alex Alm. 2006. Vital signs monitoring and patient tracking over a wireless network. In Engineering in Medicine and Biology Society, 2005. IEEE-EMBS 2005. 27th Annual International Conference of the. IEEE, 102--105.
[24]
Marc Garbey, Nanfei Sun, Arcangelo Merla, and Ioannis Pavlidis. 2007. Contact-free measurement of cardiac pulse based on the analysis of thermal imagery. IEEE Transactions on Biomedical Engineering 54, 8 (2007), 1418--1426.
[25]
Travis Gault and Aly Farag. 2013. A fully automatic method to extract the heart rate from thermal video. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 336--341.
[26]
Travis R Gault, Nicholas Blumenthal, Aly A Farag, and Tom Starr. 2010. Extraction of the superficial facial vasculature, vital signs waveforms and rates using thermal imaging. In Computer Vision and Pattern Recognition Workshops (CVPRW), 2010 IEEE Computer Society Conference on. IEEE, 1--8.
[27]
Travis R Gault and Aly A Farag. 2012. Computationally light forehead segmentation from thermal images. In Image Processing(ICIP), 2012 19th IEEE International Conference on. IEEE, 169--172.
[28]
Charles R Honts and John C Kircher. 1994. Mental and physical countermeasures reduce the accuracy of polygraph tests. Journal of Applied Psychology 79, 2 (1994), 252.
[29]
Charlie Huizenga, Hui Zhang, Edward Arens, and Danni Wang. 2004. Skin and core temperature response to partial-and whole-body heating and cooling. Journal of Thermal Biology 29, 7--8 (2004), 549--558.
[30]
Bryan F Jones and Peter Plassmann. 2002. Digital infrared thermal imaging of human skin. IEEE Engineering in Medicine and Biology Magazine 21, 6 (2002), 41--48.
[31]
MM Khan, RD Ward, and M Ingleby. 2005. Distinguishing facial expressions by thermal imaging using facial thermal feature points. In Proceedings of HCI. 5--9.
[32]
M. Khan, R. Ward, and M. Ingleby. 2017. Toward Use of Facial Thermal Features in Dynamic Assessment of Affect and Arousal Level. IEEE Transactions on Affective Computing 8, 3 (July 2017), 412--425.
[33]
Samar Khoury, Guy A Rouleau, Pierre H Rompré, Pierre Mayer, Jacques Y Montplaisir, and Gilles J Lavigne. 2008. A significant increase in breathing amplitude precedes sleep bruxism. CHEST Journal 134, 2 (2008), 332--337.
[34]
Alan T Krzywicki, Gary G Berntson, and Barbara L O'Kane. 2014. A non-contact technique for measuring eccrine sweat gland activity using passive thermal imaging. International Journal of Psychophysiology 94, 1 (2014), 25--34.
[35]
Gregory F Lewis, Rodolfo G Gatto, and Stephen W Porges. 2011. A novel method for extracting respiration rate and relative tidal volume from infrared thermography. Psychophysiology 48, 7 (2011), 877--887.
[36]
Chin Leong Lim, Chris Byrne, and Jason KW Lee. 2008. Human thermoregulation and measurement of body temperature in exercise and clinical settings. Annals Academy of Medicine Singapore 37, 4 (2008), 347.
[37]
David T. Lykken. 1984. Polygraphic interrogation. Nature 307, 5953 (1984), 681--684.
[38]
Katherine A McGonagle and Ronald C Kessler. 1990. Chronic stress, acute stress, and depressive symptoms. American journal of community psychology 18, 5 (1990), 681--706.
[39]
James B Mercer and E Francis J Ring. 2009. Fever screening and infrared thermal imaging: concerns and guidelines. Thermology International 19, 3 (2009), 67--69.
[40]
A Merla, P Iodice, A Tangherlini, G De Michele, S Di Romualdo, R Saggini, and GL Romani. 2006. Monitoring skin temperature in trained and untrained subjects throughout thermal video. In Engineering in Medicine and Biology Society, 2005. IEEE-EMBS 2005. 27th Annual International Conference of the. IEEE, 1684--1686.
[41]
Yosuke Nakayama, Guanghao Sun, Shigeto Abe, and Takemi Matsui. 2015. Non-contact measurement of respiratory and heart rates using a CMOS camera-equipped infrared camera for prompt infection screening at airport quarantine stations. In Computational Intelligence and Virtual Environments for Measurement Systems and Applications (CIVEMSA), 2015 IEEE International Conference on. IEEE, 1--4.
[42]
Eddie YK Ng, GJL Kawb, and WM Chang. 2004. Analysis of IR thermal imager for mass blind fever screening. Microvascular research 68, 2 (2004), 104--109.
[43]
Brian R Nhan and Tom Chau. 2010. Classifying affective states using thermal infrared imaging of the human face. IEEE Transactions on Biomedical Engineering 57, 4 (2010), 979--987.
[44]
Laszlo A Papp, Jose M Martinez, Donald F Klein, Jeremy D Coplan, Robert G Norman, Randolph Cole, Marybeth J de Jesus, Donald Ross, Raymond Goetz, and Jack M Gorman. 1997. Respiratory psychophysiology of panic disorder: three respiratory challenges in 98 subjects. American Journal of Psychiatry 154, 11 (1997), 1557--1565.
[45]
I. Pavlidis. 2005. System and method using thermal image analysis for polygraph testing. (Feb. 15 2005). https://www.google.com/patents/US6854879 US Patent 6,854,879.
[46]
Ioannis Pavlidis and James Levine. 2002. Thermal image analysis for polygraph testing. IEEE Engineering in Medicine and Biology Magazine 21, 6 (2002), 56--64.
[47]
Ioannis Pavlidis, James Levine, and Paulette Baukol. 2001. Thermal image analysis for anxiety detection. In Image Processing, 2001. Proceedings. 2001 International Conference on, Vol. 2. IEEE, 315--318.
[48]
Verónica Pérez-Rosas, Mohamed Abouelenien, Rada Mihalcea, and Mihai Burzo. 2015. Deception Detection Using Real-life Trial Data. In Proceedings of the 2015 ACM on International Conference on Multimodal Interaction (ICMI '15). ACM, New York, NY, USA, 59--66.
[49]
Verónica Pérez-Rosas, Mohamed Abouelenien, Rada Mihalcea, Yao Xiao, CJ Linton, and Mihai Burzo. 2015. Verbal and Nonverbal Clues for Real-life Deception Detection. In 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015. Association for Computational Linguistics, 2336--2346.
[50]
Jonathan Posner, James A Russell, and Bradley S Peterson. 2005. The circumplex model of affect: An integrative approach to affective neuroscience, cognitive development, and psychopathology. Development and psychopathology 17, 3 (2005), 715--734.
[51]
Colin Puri, Leslie Olson, Ioannis Pavlidis, James Levine, and Justin Starren. 2005. StressCam: non-contact measurement of users' emotional states through thermal imaging. In extended abstracts on Human factors in computing systems. 1725--1728.
[52]
Bashar A Rajoub and Reyer Zwiggelaar. 2014. Thermal facial analysis for deception detection. IEEE transactions on information forensics and security 9, 6 (2014), 1015--1023.
[53]
Neil Schneiderman, Gail Ironson, and Scott D Siegel. 2005. Stress and health: psychological, behavioral, and biological determinants. Annu. Rev. Clin. Psychol. 1 (2005), 607--628.
[54]
Nandita Sharma, Abhinav Dhall, Tom Gedeon, and Roland Goecke. 2013. Modeling stress using thermal facial patterns: A spatio-temporal approach. In Affective Computing and Intelligent Interaction (ACII), 2013 Humaine Association Conference on. IEEE, 387--392.
[55]
Dvijesh Shastri, Arcangelo Merla, Panagiotis Tsiamyrtzis, and Ioannis Pavlidis. 2009. Imaging facial signs of neurophysiological responses. IEEE Transactions on Biomedical Engineering 56, 2 (2009), 477--484.
[56]
Guanghao Sun, Tadafumi Saga, Takao Shimizu, Yukiya Hakozaki, and Takemi Matsui. 2014. Fever screening of seasonal influenza patients using a cost-effective thermopile array with small pixels for close-range thermometry. International Journal of Infectious Diseases 25 (2014), 56--58.
[57]
Nanfei Sun, Marc Garbey, Arcangelo Merla, and Ioannis Pavlidis. 2005. Imaging the cardiovascular pulse. In Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on, Vol. 2. IEEE, 416--421.
[58]
Nanfei Sun and Ioannis Pavlidis. 2006. Counting heartbeats at a distance. In Engineering in Medicine and Biology Society, 2006. EMBS'06. 28th Annual International Conference of the IEEE. IEEE, 228--231.
[59]
G Tanda. 2015. The use of infrared thermography to detect the skin temperature response to physical activity. In Journal of Physics: Conference Series, Vol. 655, no. 1. IOP Publishing, 012062.
[60]
Julian F Thayer, Shelby S Yamamoto, and Jos F Brosschot. 2010. The relationship of autonomic imbalance, heart rate variability and cardiovascular disease risk factors. International journal of cardiology 141, 2 (2010), 122--131.
[61]
Marieke van Dooren, Joris H Janssen, et al. 2012. Emotional sweating across the body: Comparing 16 different skin conductance measurement locations. Physiology & behavior 106, 2 (2012), 298--304.
[62]
Peter Walla, Keith Nesbitt, Karen Blackmore, Geoffrey Hookham, and Frances Kay-Lambkin. 2015. Using the Startle Eye-Blink to Measure Affect in Players. In Serious Games Analytics. 401--434.
[63]
Lara Warmelink, Aldert Vrij, Samantha Mann, Sharon Leal, Dave Forrester, and Ronald P Fisher. 2011. Thermal imaging as a lie detection tool at airports. Law and human behavior 35, 1 (2011), 40--48.
[64]
Avinash Wesley, Pradeep Buddharaju, Robert Pienta, and Ioannis Pavlidis. 2012. A comparative analysis of thermal and visual modalities for automated facial expression recognition. In International Symposium on Visual Computing. Springer, 51--60.
[65]
Hao-Yu Wu, Michael Rubinstein, Eugene Shih, John Guttag, Frédo Durand, and William Freeman. 2012. Eulerian video magnification for revealing subtle changes in the world. ACM Transactions on Graphics 31, 4 (2012), 1--8.
[66]
Ming Yang, Qiong Liu, Thea Turner, and Ying Wu. 2008. Vital sign estimation from passive thermal video. In Computer Vision and Pattern Recognition, 2008. CVPR 2008. IEEE Conference on. IEEE, 1--8.
[67]
Tuba Yilmaz, Robert Foster, and Yang Hao. 2010. Detecting vital signs with wearable wireless sensors. Sensors 10, 12 (2010), 10837--10862.
[68]
Peter Yuen, Kan Hong, Tong Chen, Aristeidis Tsitiridis, F Kam, James Jackman, David James, Mark Richardson, L Williams, William Oxford, et al. 2009. Emotional & physical stress detection and classification using thermal imaging technique. (2009).

Cited By

View all
  • (2020)Towards detecting levels of alertness in drivers using multiple modalitiesProceedings of the 13th ACM International Conference on PErvasive Technologies Related to Assistive Environments10.1145/3389189.3389192(1-9)Online publication date: 30-Jun-2020
  • (2020)Analysis of Multimodal Information for Multi-robot SystemHCI International 2020 – Late Breaking Posters10.1007/978-3-030-60700-5_20(148-155)Online publication date: 8-Nov-2020

Index Terms

  1. A Survey on Extracting Physiological Measurements from Thermal Images

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Other conferences
    PETRA '18: Proceedings of the 11th PErvasive Technologies Related to Assistive Environments Conference
    June 2018
    591 pages
    ISBN:9781450363907
    DOI:10.1145/3197768
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    In-Cooperation

    • NSF: National Science Foundation

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 26 June 2018

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. Physiological
    2. Thermal
    3. human behavior
    4. non-contact

    Qualifiers

    • Research-article
    • Research
    • Refereed limited

    Conference

    PETRA '18

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)19
    • Downloads (Last 6 weeks)0
    Reflects downloads up to 10 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2020)Towards detecting levels of alertness in drivers using multiple modalitiesProceedings of the 13th ACM International Conference on PErvasive Technologies Related to Assistive Environments10.1145/3389189.3389192(1-9)Online publication date: 30-Jun-2020
    • (2020)Analysis of Multimodal Information for Multi-robot SystemHCI International 2020 – Late Breaking Posters10.1007/978-3-030-60700-5_20(148-155)Online publication date: 8-Nov-2020

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media