Skip to main content
SpringerLink
Log in

Stress levels estimation from facial video based on non-contact measurement of pulse wave

  • Original Article
  • Published:
Artificial Life and Robotics Aims and scope Submit manuscript

Abstract

In this paper, we propose a method to measure multiple stress levels using pulse wave extracted from a facial video. Stress is a crucial factor which triggers mental illnesses. For the prevention, it is necessary to measure and handle with stress. Many researchers have attempted to measure stress using with a contact sensor. However, contact sensors can be uncomfortable or unsafe depending on the person. Recently, imaging photoplethysmography (iPPG), which measures pulse wave without contact sensors has been gathering interest. However, the estimatability of pulse rate variability (PRV) on stress condition, which include information of stress has not been investigated in most papers. In addition, the stress measurement based on PRV only detect relaxation and stress. Therefore, in this study, the following contributions is made. First, we investigate the effectiveness of the PRV of the several iPPG methods on stress condition. It is noted that comparing several iPPG methods on the stress condition is not achieved in the previous works. Second, we propose a method to estimate 4 stress levels based on PRV features based on a facial video with the iPPG methods. It is noted that “multiple” stress levels estimation based on ‘a facial video’ has not been accomplished in the previous works in the iPPG method.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Fernandez JRM, Lesya A (2018) Mental stress detection using bioradar respiratory signals. Biomed Signal Process Control 43:244–249

    Article  Google Scholar 

  2. Moriguchi A, Otsuka A, Kohara K et al (1992) Spectral change in heart rate variability in response to mentalarithmetic before and after the beta-adrenoceptor blocker, carteolol. Clin Auton Res 2(4):267–270

    Article  Google Scholar 

  3. Hjortskov N, Rissen D, Blangsted AK et al (2004) The effect of mental stress on heart rate variability and blood pressure during computer work. Eur J Appl Physiol 92(1–2):84–89

    Article  Google Scholar 

  4. Z. B. Moses, L. J. Luecken, and J. C. Eason (2007), Measuring taskrelated changes in heart rate variability. in Engineering in Medicine and Biology Society, 2007. EMBS 2007. In: 29th annual international conference of the IEEE. IEEE, pp. 644–647.

  5. Macor F, Fagard R, Amery A (1996) Power spectral analysis of RR interval and blood pressure short-term variability at rest and during dynamic exercise: comparison between cyclists and controls. Int J Sports Med 17:175–181

    Article  Google Scholar 

  6. Pagani M, Lombardi F, Guzzetti S et al (1986) Power spectral analysis of heart rate and arterial pressure variabilities as a marker of sympatho- vagal interaction in man and conscious dog. Circ Res 1986(59):178–193

    Article  Google Scholar 

  7. Malik M, Camm AJ (1990) Heart rate variability. Clin Cardiol 13:570–576

    Article  Google Scholar 

  8. Kranjec J, Begus S, Gersak G et al (2014) Non-contact heart rate and heart rate variability measurements: a review. Biomed Signal Process Control 13:102–112

    Article  Google Scholar 

  9. Sun Y, Thankor N (2016) Photoplethysmography revisited : from contact to noncontact, from point to imaging. IEEE Trans Biomed Eng 63(3):463–477

    Article  Google Scholar 

  10. C. Wang, T. Pun, and G. Chanel. (2018) A comparative survey of methods for remote heart rate detection from frontal face videos. J Front Bioeng Biotechnol pp. 6–33

  11. Mitsuhashi R, Okada G, Kurita K et al (2018) Non-contact pulse wave detection by two-band infrared video-based measurement on face without visible lighting. J Artificial Life Robotics 23:345–352

    Article  Google Scholar 

  12. McDuff D, Kaliouby R, Cohn J et al (2015) Predicting Ad liking and purchase intent: large-scale analysis of facial responses to Ads. IEEE Trans Affective Comput 6(3):223–235

    Article  Google Scholar 

  13. Wang W, denBrinker AC, Stuijk S et al (2017) Robust heartrate from fitness videos. Physiol Meas 38(6):1023

    Article  Google Scholar 

  14. Mitsuhashi R, Iuchi K, Goto T, (2019) Video-based stress levels measurement using imaging photoplethysmography. In: Proceedings of the 2019 IEEE international conference on multimedia & expo workshops (ICMEW), pp 90–95

  15. Li F, Xu P, Zheng S et al. (2018) Photoplethysmography based psychological stress detection with pulse rate variability feature differences and elastic net. Int J Distributed Sensor Netw 14(9)

  16. Wang W, den Brinker AC, de Haan G (2018) Single element remote PPG. IEEE Trans Biomed Eng 66(7):2032–2043

    Article  Google Scholar 

  17. Wang W, Stuijk S, Haan G (2015) Exploiting spatial-redundancy of image sensor for motion robust rPPG. IEEE Trans Biomed Eng 62(2):415–425

    Article  Google Scholar 

  18. Wang W, Stuijk S, Haan G (2017) Living-skin classification via remote PPG. IEEE Trans Biomed Eng 64(12):2781–2792

    Article  Google Scholar 

  19. Chen W, McDuff D (2018) DeepPhys: video-based physiological measurement using convolutional attention networks. In: The European conference on computer vision (ECCV), pp 349–365

  20. Weenk M, Alken A, Engelen L et al (2018) Stress measurement in surgeons and residents using a smart patch. Am J Surg 216:361–368

    Article  Google Scholar 

  21. Abdi R, Alhitary A, Hay E et al (2018) Objective detection of chronic stress using physiological parameters. Med Biol Eng Compu 56(12):2273–2286

    Article  Google Scholar 

  22. Verkruysse W, Svaasand LO, Nelson JS (2008) Remote plethysmographic imaging using ambient light. Opt Express 16(26):21434–21445

    Article  Google Scholar 

  23. Alghoul K, Alharthi S, Osman Al et al (2017) Heart rate variability extraction from videos signals: ICA vs. EVM comparison IEEE Access 5:4711–4719

    Article  Google Scholar 

  24. Yang Y, Liu C, Yu H et al (2016) Motion robust remote photoplethysmography in CIELab color space. J Biomed Opt 21(11):117001

    Article  Google Scholar 

  25. Haan G, Jeanne V (2013) Robust pulse rate from chrominance-based rPPG. IEEE Trans Biomed Eng 60(10):2878–2886

    Article  Google Scholar 

  26. Fukunishi M, Kurita K, Yamamoto S et al (2017) Non-contact video-based estimation of heart rate variability spectrogram from hemoglobin composition. Artificial Life Robotics 22(4):457–463

    Article  Google Scholar 

  27. Fukunishi M, Mcduff D, Tsumura N (2018) Improvements in remote video based estimation of heart rate variability using the Welch FFT method. J Artificial Life Robotics 23(1):15–22

    Article  Google Scholar 

  28. Tsumura N, Ojima N, Sato K et al (2003) Image-based skin color and texture analysis/synthesis by extracting hemoglobin and melanin information in the skin. ACM Trans Graphics 22(3):770–779

    Article  Google Scholar 

  29. Poh M, Mcduff D, Picard R (2010) Non-contact, automated cardiac pulse measurements using video imaging and blind separation. Opt Express 18(10):10762–10774

    Article  Google Scholar 

  30. Hurter C, Mcduff D (2017) Cardiolens: remote physiological monitoring in a mixed reality environment. SIGGRAPH 2017. Emerging Technol 6:1–2

    Google Scholar 

  31. Wang W, Stuijk S, Haan G (2016) A novel algorithm for remote photoplethysmography: spatial subspace rotation. IEEE Trans Biomed Eng 63(9):1974–1984

    Article  Google Scholar 

  32. Tarvainen M, Ranta-aho P, Karjalainen P (2002) An advanced detrending method with application to HRV analysis. IEEE Trans Biomed Eng 49(2):172–175

    Article  Google Scholar 

  33. Lomb NR (1976) Least-squares frequency analysis of unequally spaced data. Astrophys Space Sci 39:447–462

    Article  Google Scholar 

  34. Scargle JD (1982) Studies in astronomical time series analysis. II. Statistical aspects of spectral analysis of unevenly spaced data. Astrophys J 263:835–853

    Article  Google Scholar 

  35. Gil E, Orini M, Bailon R et al (2010) Photoplethysmography pulse rate variability as a surrogate measurement of heart rate variability during non-stationary conditions. Physiol Meas 31:1271–1290

    Article  Google Scholar 

  36. Task Force of the European Society of Cardiology and the North American Society of Pacing and Electrophysiology (1996) Heart rate variability standards of measurement, physiological interpretation, and clinical use. Circulation 93:1043–1065

    Article  Google Scholar 

  37. Kononenko I (1994) Estimation attributes: analysis and extensions of RELIEF. In: european conference on machine learning, pp 171–182

  38. Spielberger CD (1983) Manual for the state-trait anxiety inventory (STAI). Consulting Psychologists Press, Palo Alto

    Google Scholar 

  39. Marteau T, Bekker H (1992) The development of a six-item short-form of the state scale of the Spielberger State-Trait Anxiety Inventory (STAI). Br J Clin Psychol Soc 31:301–306

    Article  Google Scholar 

  40. Mcduff D, Gontarek S, Picard R (2014) Improvements in remote cardiopulmonary measurement using five band digital camera. IEEE Trans Biomed Engs 61(10):2593–2601

    Article  Google Scholar 

  41. Wang W, denBrinker AC, Stuijk S et al (2017) Amplitude-selective filtering for remote-PPG. Biomed Opt Express 8(3):1965–1980

    Article  Google Scholar 

  42. Wang W, den Brinker AC, Stuijk S et al (2017) Robust heart rate from fitness videos. Physiol Meas 38(6):1023–1044

    Article  Google Scholar 

  43. Lang P. J., Bradley M. M., and Cuthbert B. N. (1997) International Affective Picture System (IAPS): technical manual and affective ratings. NIMH Center for the Study of Emotion and Attention, University of Florida, Gainesville, FL

Download references

Author information

Authors and Affiliations

  1. Department of Imaging Science, Graduate School of Science and Engineering, Chiba University, Chiba, Japan

    Kaito Iuchi & Ryota Mitsuhashi

  2. Business Development Sales Department, Defense Systems Division, Daikin Industries, LTD., Osaka, Japan

    Takashi Goto

  3. Research and Development Department, Defense Systems Division, Daikin Industries, LTD., Osaka, Japan

    Akira Matsubara, Takahiro Hirayama & Hideki Hashizume

  4. Department of Imaging Science, Graduate School of Engineering, Chiba University, Chiba, Japan

    Norimichi Tsumura

Authors
  1. Kaito Iuchi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ryota Mitsuhashi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Takashi Goto
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Akira Matsubara
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Takahiro Hirayama
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Hideki Hashizume
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Norimichi Tsumura
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kaito Iuchi.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Iuchi, K., Mitsuhashi, R., Goto, T. et al. Stress levels estimation from facial video based on non-contact measurement of pulse wave. Artif Life Robotics 25, 335–342 (2020). https://doi.org/10.1007/s10015-020-00624-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10015-020-00624-4

Keywords

Search

Navigation