Skip to main content
SpringerLink
Log in
Book cover

International Symposium on Visual Computing

ISVC 2019: Advances in Visual Computing pp 540–551Cite as

Heart Rate Based Face Synthesis for Pulse Estimation

  • Conference paper
  • First Online:

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 11844))

Abstract

With the technological advancements in non-invasive heart rate (HR) detection, it becomes more feasible to estimate heart rate using commodity digital cameras. However, achieving high accuracy in HR estimation still remains a challenge. One of the bottlenecks is the lack of sufficient facial videos annotated with corresponding HR signals. In order to prevent this bottleneck, we propose to create videos enriched with different HR values from existing data sets with an attempt to increase the data size in a controllable manner. This paper presents a new method to generate facial videos with various heart rate values through a video synthesis procedure. Our method involves the synthesis of heart beat effects from skin colors of a face. New face video is generated with various heart rate values while taking identity information into account. The quality of the synthetic videos is evaluated by comparing to the original ground truth videos at the pixel level as well as by computing their differentiability across the synthetic face videos. Furthermore, the usability of the new data is assessed through the application of HR estimation from remote video approaches.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

References

  1. Aarts, L.A., Jeanne, V., Cleary, J.P., Lieber, C., Nelson, J.S., Oetomo, S.B., Verkruysse, W.: Non-contact heart rate monitoring utilizing camera photoplethysmography in the neonatal intensive care unit-a pilot study. Early Human Dev. 89(12), 943–948 (2013)

    Article  Google Scholar 

  2. Al-Ali, A., Diab, M.K., Coverston, R., Maurer, G., Schmidt, J., Schulz, C.: Pulse oximetry sensor, 9 October 2007, patent 7,280,858

    Google Scholar 

  3. Baltrusaitis, T., Robinson, P., Morency, L.P.: Openface: an open source facial behavior analysis toolkit. In: WACV, pp. 1–10. IEEE Computer Society (2016)

    Google Scholar 

  4. Bao, S.D., Zhang, Y.T., Shen, L.F.: Physiological signal based entity authentication for body area sensor networks and mobile healthcare systems. In: IEEE Engineering in Medicine and Biology (2006)

    Google Scholar 

  5. Biopack. https://www.biopac.com

  6. Bobbia, S., Benezeth, Y., Dubois, J.: Remote photoplethysmography based on implicit living skin tissue segmentation. In: IEEE ICPR, pp. 361–365 (2016)

    Google Scholar 

  7. Chekmenev, S.Y., Rara, H., Farag, A.A.: Non-contact, wavelet-based measurement of vital signs using thermal imaging. In: The 1st International Conference on Graphics, Vision, and Image Processing (GVIP), pp. 107–112 (2005)

    Google Scholar 

  8. Chollet, F.: Xception: deep learning with depthwise separable convolutions. In: IEEE CVPR (2017)

    Google Scholar 

  9. De Haan, G., Jeanne, V.: Robust pulse rate from chrominance-based rPPG. IEEE Trans. Biomed. Eng. 60(10), 2878–2886 (2013)

    Article  Google Scholar 

  10. Feng, L., Po, L., Xu, X., Li, Y., Ma, R.: Motion-resistant remote imaging photoplethysmography based on the optical properties of skin. IEEE Trans. Circuits Syst. Video Technol. 25(5), 879–891 (2015)

    Article  Google Scholar 

  11. Gibert, G., D’Alessandro, D., Lance, F.: Face detection method based on photoplethysmography. In: IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS) (2013)

    Google Scholar 

  12. Gordon, J.W.: Certain molar movements of the human body produced by the circulation of the blood. J. Anat. Physiol. 11(Pt 3), 533–536 (1877)

    Google Scholar 

  13. Hassan, M., et al.: Heart rate estimation using facial video: a review. Biomed. Signal Process. Control 38, 346–360 (2017)

    Article  Google Scholar 

  14. Hsu, G., Ambikapathi, A., Chen, M.: Deep learning with time-frequency representation for pulse estimation from facial videos. In: 2017 IEEE International Joint Conference on Biometrics (IJCB), pp. 383–389, October 2017

    Google Scholar 

  15. Hsu, Y., Lin, Y., Hsu, W.: Learning-based heart rate detection from remote photoplethysmography features. In: 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4433–4437 (2014)

    Google Scholar 

  16. Inan, O.T., et al.: Ballistocardiography and seismocardiography: a review of recent advances. IEEE J. Biomed. Health Inf. 19(4), 1414–1427 (2015)

    Article  Google Scholar 

  17. Jose, A.D., Collison, D.: The normal range and determinants of the intrinsic heart rate in man. Cardiovasc. Res. 4(2), 160–167 (1970)

    Article  Google Scholar 

  18. Kakria, P., Tripathi, N.K., Kitipawang, P.: A real-time health monitoring system for remote cardiac patients using smartphone and wearable sensors. Int. J. Telemed. Appl. 2015, 373474 (2015)

    Google Scholar 

  19. Kranjec, J., Beguš, S., Geršak, G., Drnovšek, J.: Non-contact heart rate and heart rate variability measurements: a review. Biomed. Sig. Process. Control 13, 102–112 (2014)

    Article  Google Scholar 

  20. Kumar, M., Veeraraghavan, A., Sabharwal, A.: DistancePPG: robust non-contact vital signs monitoring using a camera. Biomed. Optics Express. 6(5), 1565–1588 (2015)

    Article  Google Scholar 

  21. Lam, A., Kuno, Y.: Robust heart rate measurement from video using select random patches. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3640–3648 (2015)

    Google Scholar 

  22. Lawson, G., Belcher, R., Dawes, G., Redman, C.: A comparison of ultrasound (with autocorrelation) and direct electrocardiogram fetal heart rate detector systems. Am. J. Obstet. Gynecol. 147(6), 721–722 (1983)

    Article  Google Scholar 

  23. Lee, A., Kim, Y.: Photoplethysmography as a form of biometric authentication. In: IEEE SENSORS, pp. 1–2 (2015)

    Google Scholar 

  24. Lee, T.W.: Independent component analysis. In: Lee, T.W. (ed.) Independent Component Analysis, pp. 27–66. Springer, Boston (1998). https://doi.org/10.1007/978-1-4757-2851-4_2

    Chapter  Google Scholar 

  25. Lewandowska, M., Rumiński, J., Kocejko, T., Nowak, J.: Measuring pulse rate with a webcam a non-contact method for evaluating cardiac activity. In: IEEE Federated Conference on Computer Science and Information Systems (FedCSIS) (2011)

    Google Scholar 

  26. Li, X., Chen, J., Zhao, G., Pietikäinen, M.: Remote heart rate measurement from face videos under realistic situations. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp. 4264–4271, June 2014

    Google Scholar 

  27. Lu, G., Yang, F., Taylor, J.A., Stein, J.F.: A comparison of photoplethysmography and ECG recording to analyse heart rate variability in healthy subjects. J. Med. Eng. Technol. 33(8), 634–641 (2009)

    Article  Google Scholar 

  28. Mestha, L.K., Kyal, S., Xu, B., Lewis, L.E., Kumar, V.: Towards continuous monitoring of pulse rate in neonatal intensive care unit with a webcam. In: IEEE Engineering in Medicine and Biology Society (EMBC) (2014)

    Google Scholar 

  29. Niu, X., Han, H., Shan, S., Chen, X.: SynRhythm: learning a deep heart rate estimator from general to specific. In: 2018 24th International Conference on Pattern Recognition (ICPR), pp. 3580–3585 (2018)

    Google Scholar 

  30. Poh, M.Z., McDuff, D.J., Picard, R.W.: Non-contact, automated cardiac pulse measurements using video imaging and blind source separation. Opt. Express 18(10), 10762–10774 (2010)

    Article  Google Scholar 

  31. Poh, M.Z., McDuff, D.J., Picard, R.W.: Advancements in noncontact, multiparameter physiological measurements using a webcam. IEEE Trans. Biomed. Eng. 58(1), 7–11 (2011)

    Article  Google Scholar 

  32. Rouast, P.V., Adam, M.T., Chiong, R., Cornforth, D., Lux, E.: Remote heart rate measurement using low-cost RGB face video: a technical literature review. Front. Comput. Sci. 12(5), 858–872 (2018)

    Article  Google Scholar 

  33. Rouast, P.V., Adam, M.T.P., Cornforth, D.J., Lux, E., Weinhardt, C.: Using contactless heart rate measurements for real-time assessment of affective states. In: Davis, F.D., Riedl, R., vom Brocke, J., Léger, P.-M., Randolph, A.B. (eds.) Information Systems and Neuroscience. LNISO, vol. 16, pp. 157–163. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-41402-7_20

    Chapter  Google Scholar 

  34. Sameni, R., Clifford, G.D.: A review of fetal ECG signal processing; issues and promising directions. Open Pacing Electrophysiol. Ther. J. 3, 4 (2010)

    Google Scholar 

  35. Scalise, L., Bernacchia, N., Ercoli, I., Marchionni, P.: Heart rate measurement in neonatal patients using a webcamera. In: IEEE International Symposium on Medical Measurements and Applications (MeMeA) (2012)

    Google Scholar 

  36. Sodi-Pallares, D.: Electrophysiology of the heart. Am. J. Cardiol. (1961)

    Google Scholar 

  37. Soleymani, M., Lichtenauer, J., Pun, T., Pantic, M.: A multimodal database for affect recognition and implicit tagging. IEEE Trans. Affect. Comput. 3(1), 42–55 (2012)

    Article  Google Scholar 

  38. Tulyakov, S., Alameda-Pineda, X., Ricci, E., Yin, L., Cohn, J.F., Sebe, N.: Self-adaptive matrix completion for heart rate estimation from face videos under realistic conditions. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)

    Google Scholar 

  39. Verkruysse, W., Svaasand, L.O., Nelson, J.S.: Remote plethysmographic imaging using ambient light. Opt. Express 16(26), 21434–21445 (2008)

    Article  Google Scholar 

  40. Wang, W., den Brinker, A.C., Stuijk, S., de Haan, G.: Algorithmic principles of remote PPG. IEEE Trans. Biomed. Eng. 64(7), 1479–1491 (2017)

    Article  Google Scholar 

  41. Wang, W., Stuijk, S., De Haan, G.: Unsupervised subject detection via remote PPG. IEEE Trans. Biomed. Eng. 62(11), 2629–2637 (2015)

    Article  Google Scholar 

  42. Wang, W., Stuijk, S., De Haan, G.: A novel algorithm for remote photoplethysmography: spatial subspace rotation. IEEE Trans. Biomed. Eng. 63(9), 1974–1984 (2016)

    Article  Google Scholar 

  43. Wang, W., Stuijk, S., de Haan, G.: Living-skin classification via remote-PPG. IEEE Trans. Biomed. Eng. 64(12), 2781–2792 (2017)

    Article  Google Scholar 

  44. Yan, Y., Ma, X., Yao, L., Ouyang, J.: Noncontact measurement of heart rate using facial video illuminated under natural light and signal weighted analysis. Biomed. Mater. Eng. 26(s1), S903–S909 (2015)

    Google Scholar 

  45. Yu, Y.P., Kwan, B.H., Lim, C.L., Wong, S.L., Raveendran, P.: Video-based heart rate measurement using short-time Fourier transform. In: Intelligent Signal Processing and Communications Systems (ISPACS), pp. 704–707. IEEE (2013)

    Google Scholar 

  46. Zhang, Z., et al.: Multimodal spontaneous emotion corpus for human behavior analysis. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3438–3446 (2016)

    Google Scholar 

Download references

Acknowledgement

The material is based upon the work supported in part by the National Science Foundation under grants CNS-1629898 and CNS-1205664.

Author information

Authors and Affiliations

  1. State University of New York at Binghamton, Binghamton NY, USA

    Umur Aybars Ciftci & Lijun Yin

Authors
  1. Umur Aybars Ciftci
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lijun Yin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Umur Aybars Ciftci .

Editor information

Editors and Affiliations

  1. University of Nevada, Reno, NV, USA

    George Bebis

  2. NASA Ames Research Center, Moffett Field, CA, USA

    Richard Boyle

  3. University of Nevada, Reno, NV, USA

    Bahram Parvin

  4. Desert Research Institute, Reno, NV, USA

    Darko Koracin

  5. Lawrence Berkeley National Laboratory, Berkeley, CA, USA

    Daniela Ushizima

  6. Latent AI, Palo Alto, CA, USA

    Sek Chai

  7. Texas A&M University, College Station, TX, USA

    Shinjiro Sueda

  8. Louisiana State University, Baton Rouge, LA, USA

    Xin Lin

  9. University of North Carolina at Charlotte, Charlotte, NC, USA

    Aidong Lu

  10. École Polytechnique Fédérale de Lausanne, Lausanne, Switzerland

    Daniel Thalmann

  11. Notre Dame University, Notre Dame, IN, USA

    Chaoli Wang

  12. Bosch Research North America, Palo Alto, CA, USA

    Panpan Xu

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Ciftci, U.A., Yin, L. (2019). Heart Rate Based Face Synthesis for Pulse Estimation. In: Bebis, G., et al. Advances in Visual Computing. ISVC 2019. Lecture Notes in Computer Science(), vol 11844. Springer, Cham. https://doi.org/10.1007/978-3-030-33720-9_42

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-33720-9_42

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-33719-3

  • Online ISBN: 978-3-030-33720-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation