Skip to main content
Springer Nature Link
Log in

Spatial-Channel Mixed Attention Based Network for Remote Heart Rate Estimation

  • Conference paper
  • First Online:
Pattern Recognition and Computer Vision (PRCV 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13535))

Included in the following conference series:

  • 1659 Accesses

Abstract

Recently, remote heart rate (HR) estimation has attracted increasing attention. Some previous methods employ the Spatial-Temporal Map (STMap) of facial sequences to estimate HR. However, STMap ignores that each facial regions play a different role in HR estimation task. Moreover, how to focus on key facial regions to improve the performance of HR estimation is also a challenging problem.To overcome this issue, this paper proposes a novel Spatial-Channel Mixed Attention Module (SCAM) to select the facial regions with high correlation for HR estimation adaptively. To our best knowledge, we are the first to design an attention module for STMap in the HR estimation task. Furthermore, the whole HR estimation framework SCANet is proposed, including feature extraction, signal generation, and HR regression.The experiments performed on three benchmark datasets show that the proposed method achieves better performance over state-of-the-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    The position of ECG sensor is upper left corner of chest and under clavicle bone.

  2. 2.

    https://github.com/mne-tools/mne-python.

  3. 3.

    https://pytorch.org/.

References

  1. Balakrishnan, G., Durand, F., Guttag, J.: Detecting pulse from head motions in video. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3430–3437 (2013)

    Google Scholar 

  2. Chen, W., McDuff, D.: DeepPhys: video-based physiological measurement using convolutional attention networks. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11206, pp. 356–373. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01216-8_22

    Chapter  Google Scholar 

  3. De Haan, G., Jeanne, V.: Robust pulse rate from chrominance-based RPPG. IEEE Trans. Biomed. Eng. 60(10), 2878–2886 (2013)

    Article  Google Scholar 

  4. De Haan, G., Van Leest, A.: Improved motion robustness of remote-PPG by using the blood volume pulse signature. Physiol. Meas. 35(9), 1913 (2014)

    Article  Google Scholar 

  5. Dosovitskiy, A., et al.: An image is worth 16x16 words: transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)

  6. Garbey, M., Sun, N., Merla, A., Pavlidis, I.: Contact-free measurement of cardiac pulse based on the analysis of thermal imagery. IEEE Trans. Biomed. Eng. 54(8), 1418–1426 (2007)

    Article  Google Scholar 

  7. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

    Google Scholar 

  8. Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7132–7141 (2018)

    Google Scholar 

  9. Kwon, S., Kim, J., Lee, D., Park, K.: Roi analysis for remote photoplethysmography on facial video. In: 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pp. 4938–4941. IEEE (2015)

    Google Scholar 

  10. Lam, A., Kuno, Y.: Robust heart rate measurement from video using select random patches. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3640–3648 (2015)

    Google Scholar 

  11. Lee, E., Chen, E., Lee, C.-Y.: Meta-rPPG: remote heart rate estimation using a transductive meta-learner. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12372, pp. 392–409. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58583-9_24

    Chapter  Google Scholar 

  12. Lempe, G., Zaunseder, S., Wirthgen, T., Zipser, S., Malberg, H.: Roi selection for remote photoplethysmography. In: Bildverarbeitung für die Medizin 2013, pp. 99–103. Springer, Heidelberg (2013)

    Google Scholar 

  13. Li, X., Chen, J., Zhao, G., Pietikainen, M.: Remote heart rate measurement from face videos under realistic situations. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4264–4271 (2014)

    Google Scholar 

  14. Lu, H., Han, H., Zhou, S.K.: Dual-gan: Joint bvp and noise modeling for remote physiological measurement. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12404–12413 (2021)

    Google Scholar 

  15. McDuff, D.: Deep super resolution for recovering physiological information from videos. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 1367–1374 (2018)

    Google Scholar 

  16. McDuff, D., Gontarek, S., Picard, R.W.: Improvements in remote cardiopulmonary measurement using a five band digital camera. IEEE Trans. Biomed. Eng. 61(10), 2593–2601 (2014)

    Article  Google Scholar 

  17. Niu, X., Han, H., Shan, S., Chen, X.: VIPL-HR: a multi-modal database for pulse estimation from less-constrained face video. In: Jawahar, C.V., Li, H., Mori, G., Schindler, K. (eds.) ACCV 2018. LNCS, vol. 11365, pp. 562–576. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-20873-8_36

    Chapter  Google Scholar 

  18. Niu, X., Shan, S., Han, H., Chen, X.: Rhythmnet: end-to-end heart rate estimation from face via spatial-temporal representation. TIP 29, 2409–2423 (2019)

    MATH  Google Scholar 

  19. Niu, X., Yu, Z., Han, H., Li, X., Shan, S., Zhao, G.: Video-based remote physiological measurement via cross-verified feature disentangling. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12347, pp. 295–310. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58536-5_18

    Chapter  Google Scholar 

  20. Niu, X., Zhao, X., Han, H., Das, A., Dantcheva, A., Shan, S., Chen, X.: Robust remote heart rate estimation from face utilizing spatial-temporal attention. In: 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition, pp. 1–8. IEEE (2019)

    Google Scholar 

  21. Perepelkina, O., Artemyev, M., Churikova, M., Grinenko, M.: Hearttrack: Convolutional neural network for remote video-based heart rate monitoring. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 288–289 (2020)

    Google Scholar 

  22. Poh, M.Z., McDuff, D.J., Picard, R.W.: Advancements in noncontact, multiparameter physiological measurements using a webcam. IEEE Trans. Biomed. Eng. 58(1), 7–11 (2010)

    Article  Google Scholar 

  23. Poh, M.Z., McDuff, D.J., Picard, R.W.: Non-contact, automated cardiac pulse measurements using video imaging and blind source separation. Opt. Express 18(10), 10762–10774 (2010)

    Article  Google Scholar 

  24. Sabokrou, M., Pourreza, M., Li, X., Fathy, M., Zhao, G.: Deep-hr: Fast heart rate estimation from face video under realistic conditions. arXiv preprint arXiv:2002.04821 (2020)

  25. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)

  26. Soleymani, M., Lichtenauer, J., Pun, T., Pantic, M.: A multimodal database for affect recognition and implicit tagging. IEEE Trans. Affect. Comput. 3(1), 42–55 (2011)

    Article  Google Scholar 

  27. Špetlík, R., Franc, V., Matas, J.: Visual heart rate estimation with convolutional neural network. In: Proceedings of the british machine vision conference, Newcastle, UK, pp. 3–6 (2018)

    Google Scholar 

  28. Stricker, R., Müller, S., Gross, H.M.: Non-contact video-based pulse rate measurement on a mobile service robot. In: The 23rd IEEE International Symposium on Robot and Human Interactive Communication, pp. 1056–1062. IEEE (2014)

    Google Scholar 

  29. Ulyanov, S.S., Tuchin, V.V.: Pulse-wave monitoring by means of focused laser beams scattered by skin surface and membranes. In: Static and Dynamic Light Scattering in Medicine and Biology, vol. 1884, pp. 160–167. International Society for Optics and Photonics (1993)

    Google Scholar 

  30. Verkruysse, W., Svaasand, L.O., Nelson, J.S.: Remote plethysmographic imaging using ambient light. Opt. Express 16(26), 21434–21445 (2008)

    Article  Google Scholar 

  31. Wang, W., den Brinker, A.C., Stuijk, S., de Haan, G.: Algorithmic principles of remote ppg. IEEE Trans. Biomed. Eng. 64(7), 1479–1491 (2016)

    Article  Google Scholar 

  32. Woo, S., Park, J., Lee, J.-Y., Kweon, I.S.: CBAM: convolutional block attention module. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11211, pp. 3–19. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01234-2_1

    Chapter  Google Scholar 

  33. Yu, Z., Peng, W., Li, X., Hong, X., Zhao, G.: Remote heart rate measurement from highly compressed facial videos: an end-to-end deep learning solution with video enhancement. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 151–160 (2019)

    Google Scholar 

Download references

Acknowledgments

This work was supported by the National Science Fund of China under Grant Nos. 61876083,62176124.

Author information

Authors and Affiliations

  1. PCA Lab, Key Lab of Intelligent Perception and Systems for High-Dimensional Information of Ministry of Education, and Jiangsu Key Lab of Image and Video Understanding for Social Security, School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing, China

    Bixiao Ling, Pengfei Zhang, Jianjun Qian & Jian Yang

Authors
  1. Bixiao Ling
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Pengfei Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jianjun Qian
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jian Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jianjun Qian .

Editor information

Editors and Affiliations

  1. Southern University of Science and Technology, Shenzhen, China

    Shiqi Yu

  2. Institute of Automation, Chinese Academy of Sciences, Beijing, China

    Zhaoxiang Zhang

  3. Hong Kong Baptist University, Hong Kong, China

    Pong C. Yuen

  4. Northwestern Polytechnical University, Xi'an, China

    Junwei Han

  5. Institute of Automation, Chinese Academy of Sciences, Beijing, China

    Tieniu Tan

  6. Hong Kong Baptist University, Hong Kong, China

    Yike Guo

  7. Sun Yat-sen University, Guangzhou, China

    Jianhuang Lai

  8. Southern University of Science and Technology, Shenzhen, China

    Jianguo Zhang

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Ling, B., Zhang, P., Qian, J., Yang, J. (2022). Spatial-Channel Mixed Attention Based Network for Remote Heart Rate Estimation. In: Yu, S., et al. Pattern Recognition and Computer Vision. PRCV 2022. Lecture Notes in Computer Science, vol 13535. Springer, Cham. https://doi.org/10.1007/978-3-031-18910-4_37

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-18910-4_37

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-18909-8

  • Online ISBN: 978-3-031-18910-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics