Skip to main content
SpringerLink
Log in

Heart rate estimation based on face video under unstable illumination

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Remote photoplethysmography (rPPG) is a noncontact heart rate (HR) measurement technique. The current heart rate measurement methods based on rPPG all require ideal lighting conditions, but the lighting in real scenes is complicated, Therefore, this article proposes a robust heart rate measurement method when unstable light (time-varying light and uneven spatial illumination) exists. First, the method locates the ROI(Region of Interest) area of ​​the face, divides it into blocks, and uses the color signals of different sub-blocks to establish a three-dimensional rPPG model. Second, the method performs logarithmic operations on each frame of the image to convert the relationship between the illumination component and the reflection component from a product to a sum so that the reflected component and noise can be separated in the frequency domain. Then, the ensemble empirical mode decomposition (EEMD) is used to decompose the reflected component, and the obtained intrinsic mode function (IMF) is applied to obtain the waveform reflecting the change in the heart rate. Finally, the signal quality (SQ) of each ROI sub-block is calculated, and the high-quality signals are combined to reconstruct the heart rate signal.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  1. Hernandez-Matamoros A, Fujita H, Escamilla-Hernandez E, Perez-Meana H, Nakano-Miyatake M (2020) Recognition of ECG signals using wavelet based on atomic functions. Biocyb Biomed Eng 40(2):803–814

  2. Hernandez-Matamoros A, Fujita H, Perez-Meana H (2020) A novel approach to create synthetic biomedical signals using BiRNN. Inform Sci 541:218–241

  3. Verkruysse W, Svaasand LO, and Nelson JS (2008) Remote plethysmographic imaging using ambient light. Opt Exp 16(26):21434–21445

  4. Ming-Zher P, McDuffDaniel J, PicardRosalind W (2010) Non-contact, automated cardiac pulse measurements using video imaging and blind source separation. Opt Exp 18(10):10762–10774

  5. Poh MZ, Mcduff DJ, Picard RW (2010) Advancements in noncontact, multiparameter physiological measurements using a webcam. IEEE Trans Biomed Eng 58(1):7–11

  6. Balakrishnan G, Durand F, Guttag J (2013) Detecting pulse from head motions in video. In Proceedings of the IEEE Conference on Computer Vision and Pattern (CVPR), Portland, OR, USA, 3430–3437

  7. De Haan G, Jeanne V (2013) Robust pulse rate From chrominance-based rPPG. IEEE Trans Biomed Eng 60(10):2878–2886

  8. Asthana A, Zafeiriou S, Cheng S., Pantic M (2013) Robust discriminative response map fitting with constrained local models. In Proceedings of the IEEE Conference on Computer Vision and Pattern (CVPR), Portland, OR, 3444–3451. https://doi.org/10.1109/CVPR.2013.442

  9. Tulyakov S, Sebe N (2015) regressing a 3D face shape from a single image. In IEEE International Conference on Computer Vision (ICCV), Santiago, Chile, 3748–3755. https://doi.org/10.1109/ICCV.2015.427

  10. Xu Z-F, Jia R-S, Liu Y-B, Zhao C-Y, Sun H-M (2020) Fast method of detecting tomatoes in a complex scene for picking robots. IEEE Access 8:55289–55299

  11. Chwyl B, Chung AG, Amelara R (2016) SAPPHIRE: Stochastically acquired photoplethysmogram for heart rate inference in realistic environments. In IEEE International Conference on Image Processing(ICIP), Phoenix, AZ, 1230–1234. https://doi.org/10.1109/ICIP.2016.7532554

  12. Li X, Chen J, Zhao G (2014) Remote heart rate measurement from face videos under realistic situations. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Columbus, OH, 4264–4271. https://doi.org/10.1109/CVPR.2014.543

  13. Lee D, Kim J, Kwon S (2015) Heart rate estimation from facial photoplethysmography during dynamic illuminance changes. In 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Milan, 2758–2761. https://doi.org/10.1109/EMBC.2015.7318963

  14. Cheng J, Chen X, Xu L (2017) Illumination variation-resistant video-based heart rate measurement using joint blind source separation and ensemble empirical mode decomposition. IEEE J Biomed Health Infor 21(5):1422–1433.

  15. Goldberg AB, Zhu X, Recht BH (2010) Transduction with matrix completion: three birds with one stone. Adv Neur Infor Proc Syst 23:757–765

  16. Tulyakov S, Alameda-Pineda X, Ricci E, Yin L, Cohn JF and Sebe N (2016) Self-adaptive matrix completion for heart rate estimation from face videos under realistic conditions. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, 2396–2404. https://doi.org/10.1109/CVPR.2016.263

  17. Nowara EM, Marks TK, Mansour H, Veeraraghavany A (2018) SparsePPG: towards driver monitoring using camera-based vital signs estimation in near-infrared. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Salt Lake City, UT, 1353–135309. https://doi.org/10.1109/CVPRW.2018.00174

  18. Wang W, Brinker ACd, Stuijk S (2017) Algorithmic principles of remote PPG. IEEE Trans Biomed Eng 64(7):1479–1491. https://doi.org/10.1109/TBME.2016.2609282

  19. Acharya UR, Fujita H, Oh SL (2017) Application of deep convolutional neural network for automated detection of myocardial infarction using ECG signals. Inform Sci 415:190–198. https://doi.org/10.1016/j.ins.2017.06.027

  20. Acharya UR, Fujita H, Oh SL (2017) Automated identification of shockable and non-shockable life-threatening ventricular arrhythmias using convolutional neural network. Future Gen Comput Syst 79(3):952–959

  21. Fujita H, Cimr D (2019) Decision support system for arrhythmia prediction using convolutional neural network structure without preprocessing. Appl Intell 49:3383–3391. https://doi.org/10.1007/s10489-019-01461-0

  22. Chen X, Cheng J, Song R, Liu Y, Ward R, Wang ZJ (2018) Video-based heart rate measurement: recent advances and future prospects. IEEE Trans Inst Meas 68(10):3600–3615. https://doi.org/10.1109/TIM.2018.2879706

  23. Martinez N, Bertran M, Sapiro G (2019) Non-contact photoplethysmogram and instantaneous heart rate estimation from infrared face video. In IEEE International Conference on Image Processing (ICIP), Taipei, Taiwan, 2020–2024. https://doi.org/10.1109/ICIP.2019.8803109

  24. Kumar M, Veeraraghavan A, Sabharwal A (2015) Distanceppg: robust non-contact vital signs monitoring using a camera. Biomed Opt Exp 6(5):1565–1588. https://doi.org/10.1364/BOE.6.001565

  25. Chen DY, Wang JJ, Lin KY (2015) Image sensor-based heart rate evaluation from face reflectance using hilbert–huang transform. IEEE Sens J 15(1):618–627

  26. King DE (2009) Dlib-ml: a machine learning toolkit. J Mach Learn Res 10(3):1755–1758

  27. Tomas C, Kanade T (1991) Detection and tracking of point features. Int J Comput Vis 9(3):137–154

  28. Candès EJ, Recht B (2009) Exact matrix completion via convex optimization. Found Comput Math 9(6):717. https://doi.org/10.1145/2184319.2184343

  29. WU ZH, Huang NE (2009) Ensemble empirical mode decomposition: a noise-assisted data analysis method. Adv Adapt Data Anal 1(1):1-41

  30. Sun Y, Thakor N (2016) Photoplethysmography revisited: from contact to noncontact, from point to imaging. IEEE Trans Biomed Eng 63(3):463–477

  31. Avram R, Tison GH, Aschbacher K (2019) Real-world heart rate norms in the Health eHeart study. NPJ Digit Med 2:58. https://doi.org/10.1038/s41746-019-0134-9

  32. Robert A, Clausi DA, Alexander W (2016) Spectral-spatial fusion model for robust blood pulse waveform extraction in photoplethysmographic imaging. Biomed Opt Exp 7(12):4874–4885. https://doi.org/10.1364/BOE.7.004874

  33. Favilla R, Zuccalà VC, Coppini G (2019) Heart rate and heart rate variability from Single-Channel video and ICA integration of multiple signals. in IEEE J Biomed Health Inf 23(6):2398–2408. https://doi.org/10.1109/JBHI.2018.2880097

  34. Glenn WH (1989) Noise in interferometric optical systems: an optical Nyquist theorem. IEEE J Quant Electron 25(6):1218–1224

  35. Chen W, Thierry P, Guillaume C (2018) A comparative survey of methods for remote heart rate detection from frontal face videos. Front Bioeng Biotechnol 6:33. https://doi.org/10.3389/fbioe.2018.00033

Download references

Acknowledgments

The authors are grateful for collaborative funding support from the Natural Science Foundation of Shandong Province, China (ZR2018 MEE008).

Author information

Authors and Affiliations

  1. College of Computer Science and Engineering, Shandong University of Science and Technology, Qingdao, 266590, China

    Ruo-Nan Yin, Rui-Sheng Jia, Zhe Cui, Jin-Tao Yu, Yan-Bin Du, Li Gao & Hong-Mei Sun

  2. Shandong Province Key Laboratory of Wisdom Mine Information Technology, Shandong University of Science and Technology, Qingdao, 266590, China

    Rui-Sheng Jia & Hong-Mei Sun

Authors
  1. Ruo-Nan Yin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rui-Sheng Jia
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zhe Cui
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jin-Tao Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yan-Bin Du
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Li Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Hong-Mei Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Rui-Sheng Jia or Hong-Mei Sun.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yin, RN., Jia, RS., Cui, Z. et al. Heart rate estimation based on face video under unstable illumination. Appl Intell 51, 5388–5404 (2021). https://doi.org/10.1007/s10489-020-02167-4

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-020-02167-4

Keywords

Search

Navigation