Skip to main content
SpringerLink
Log in

Motion-resistant heart rate measurement from face videos using patch-based fusion

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

The ability to measure heart rate (HR) from face videos is useful in applications such as neonatal monitoring, telemedicine and affective computing. In the realistic environments, subjects often have spontaneous head movements and facial expressions which severely degrade the performances of the current methods. We propose a novel patch-based fusion framework for estimating accurate HR from face videos in the presence of subjects’ motions. The wavelet time–frequency analysis is applied on the raw blood volume pulse (BVP) signals for selecting less contaminated patches. Furthermore, a weighted fusion formula is constructed to obtain the final precise BVP signal, which is based on frequency and gradient information. Our method is validated on both our self-collected dataset and public dataset MAHNOB-HCI. Compared with the state of the art, experimental results show that the proposed method has an obvious superiority in the accuracy and robustness.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Kannel, W.B., Kannel, C., Paffenbarger, R.S., Cupples, L.A.: Heart rate and cardiovascular mortality: the Framingham study. Am. Heart J. 113(6), 1489–1494 (1953). https://doi.org/10.1016/0002-8703(87)90666-1

    Article  Google Scholar 

  2. Temko, A.: Accurate heart rate monitoring during physical exercises using PPG. IEEE Trans. Biomed. Eng. 64(9), 2016–2024 (2017). https://doi.org/10.1109/TBME.2017.2676243

    Article  Google Scholar 

  3. Guven, G., Gurkan, H., Guz, U.: Biometric identification using fingertip electrocardiogram signals. Signal Image Video Process. 12, 1–8 (2018). https://doi.org/10.1007/s11760-018-1238-4

    Article  Google Scholar 

  4. Allen, J.: Photoplethysmography and its application in clinical physiological measurement. Physiol. Meas. 28(3), R1-39 (2007). https://doi.org/10.1088/0967-3334/28/3/R01

    Article  Google Scholar 

  5. Wu, T., Blazek, V., Schmitt, H.: Photoplethysmography imaging: a new noninvasive and non-contact method for mapping of the dermal perfusion changes. Proc. SPIE 4163, 62–70 (2000). https://doi.org/10.1117/12.407646

    Article  Google Scholar 

  6. Verkruysse, W., Svaasand, L.O., Nelson, J.S.: Remote plethysmographic imaging using ambient light. Opt. Express 16(26), 21434–21445 (2008). https://doi.org/10.1364/OE.16.021434

    Article  Google Scholar 

  7. Poh, M.Z., McDuff, D.J., Picard, R.W.: Non-contact, automated cardiac pulse measurements using video imaging and blind source separation. Opt. Express 18(10), 10762–10774 (2010). https://doi.org/10.1364/OE.18.010762

    Article  Google Scholar 

  8. Kwon, S., Kim, H., Park K.S.: Validation of heart rate extraction using video imaging on a built-in camera system of a smartphone. In: IEEE Engineering in Medicine and Biology Society (EMBC). San Diego, CA, USA, pp. 2174–2177 (2012). https://doi.org/10.1109/EMBC.2012.6346392

  9. Poh, M.Z., McDuff, D.J., Picard, R.W.: Advancements in non-contact, multiparameter physiological measurements using a webcam. IEEE Trans. Biomed. Eng. 58(1), 7–11 (2011). https://doi.org/10.1109/TBME.2010.2086456

    Article  Google Scholar 

  10. Li, X.B., Chen, J., Zhao, G., Pietikainen, M.: Remote heart rate measurement from face videos under realistic situation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Columbus, OH, USA, pp. 4264–4271 (2014). https://doi.org/10.1109/CVPR.2014.543

  11. De Haan, G., Jeanne, V.: Robust pulse rate from chrominance-based rPPG. IEEE Trans. Biomed. Eng. 60(10), 2878–2886 (2013). https://doi.org/10.1109/TBME.2013.2266196

    Article  Google Scholar 

  12. Wang, W., den Brinker, A.C., Stuijk, S., de Haan, G.: Algorithmic principles of remote-PPG. IEEE Trans. Biomed. Eng. 64(7), 1479–1491 (2017). https://doi.org/10.1109/TBME.2016.2609282

    Article  Google Scholar 

  13. Wang, W., den Brinker, A.C., Stuijk, S., de Haan, G.: Amplitude-selective filtering for remote-PPG. Biomed. Opt. Express 8(3), 1965–1980 (2017). https://doi.org/10.1364/BOE.8.001965

    Article  Google Scholar 

  14. Lam, A., Kuno, Y.: Robust heart rate measurement from video using select random patches. In: IEEE International Conference on Computer Vision (ICCV). Washington, DC, USA, pp. 3640–3648 (2015). https://doi.org/10.1109/ICCV.2015.415

  15. Kumar, M., Veeraraghavan, A., Sabharwal, A.: DistancePPG: robust non-contact vital signs monitoring using a camera. Biomed. Opt. Express 6(5), 1565–1588 (2015). https://doi.org/10.1364/BOE.6.001565

    Article  Google Scholar 

  16. Tulyakov, S., Alameda-Pineda, X., Ricci, E., Yin, L., Cohn, J.F., Sebe, N.: Self-adaptive matrix completion for heart rate estimation from face videos under realistic conditions. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, NV, USA, pp. 2396–2404 (2016). https://doi.org/10.1109/CVPR.2016.263

  17. Ha, R.Y., Nojima, K., Adams, W.J., Brown, S.A.: Analysis of facial skin thickness: defining the relative thickness index. Plast. Reconstr. Surg. 115(6), 1769–1773 (2005). https://doi.org/10.1097/01.PRS.0000161682.63535.9B

    Article  Google Scholar 

  18. Viola, P., Jones, M.: Rapid object detection using a boosted cascade of simple features. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR). Kauai, HI, USA, pp. 511–518 (2001). https://doi.org/10.1109/CVPR.2001.990517

  19. Asthana, A., Zafeiriou, S., Cheng, S., Pantic, M.: Robust discriminative response map fitting with constrained local models. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Portland, OR, USA, pp. 3444–3451 (2013). https://doi.org/10.1109/CVPR.2013.442

  20. Tomasi, C., Kanade, T.: Detection and tracking of point features. Technical Report CMU-CS-91-132, Carnegie Mellon University (1991)

  21. Daubechies, I., Heil, C.: Ten Lectures on Wavelets. Capital City Press, Vermont (1992)

    Book  MATH  Google Scholar 

  22. Tarvainen, M.P., Ranta-aho, P.O., Karjalainen, P.A.: An advanced detrending method with application to HRV analysis. IEEE Trans. Biomed. Eng. 49(2), 172–175 (2002). https://doi.org/10.1109/10.979357

    Article  Google Scholar 

  23. Soleymani, M., Lichtenauer, J., Pun, T., Pantic, M.: A multimodal database for affect recognition and implicit tagging. IEEE Trans. Affect. Comput. 3(1), 42–55 (2012). https://doi.org/10.1109/T-AFFC.2011.25

    Article  Google Scholar 

  24. Huang, S.J., Hsieh, C.T., Huang, C.L.: Application of Morlet wavelets to supervise power system disturbances. IEEE Trans. Power Deliv. 14(1), 235–243 (1999). https://doi.org/10.1109/61.736728

    Article  Google Scholar 

Download references

Acknowledgements

We acknowledge funding support from: Training Programme Foundation for Application of Scientific and Technological Achievements of Hefei University of Technology (JZ2018YYPY0289) and the Specialized Research Fund for the Doctoral Program of Higher Education of China (JZ2018HGBZ0186).

Author information

Authors and Affiliations

  1. School of Computer and Information, Hefei University of Technology, Hefei, 230009, China

    Zhao Yang, Xuezhi Yang, Jing Jin & Xiu Wu

  2. Anhui Province Key Laboratory of Industry Safety and Emergency Technology, Hefei, 230009, China

    Zhao Yang, Xuezhi Yang, Jing Jin & Xiu Wu

Authors
  1. Zhao Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xuezhi Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jing Jin
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xiu Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xuezhi Yang.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yang, Z., Yang, X., Jin, J. et al. Motion-resistant heart rate measurement from face videos using patch-based fusion. SIViP 13, 423–430 (2019). https://doi.org/10.1007/s11760-018-01409-w

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11760-018-01409-w

Keywords

Search

Navigation