Yun-Yun Tsou
ABSTRACT
Remote photoplethysmography (rPPG) is a contactless method for heart rate (HR) estimation from face videos. In this paper, we propose to estimate rPPG signals directly from input video sequences in an end-to-end manner. We propose a novel Siamese-rPPG network to simultaneously learn the heterogeneous and homogeneous features from two facial regions. Furthermore, to analyze the temporal periodicity of rPPG signals, we construct the network with 3D CNNs and jointly train the two-branch model under the negative Pearson loss function. Experimental results on three benchmark datasets: COHFACE, UBFC, and PURE, show that our method significantly outperforms existing methods with a large margin.
- John Allen. 2007. Photoplethysmography and Its Application in Clinical Physiological Measurement. Physiological Measurement 28, 3 (Feb 2007), R1--R39. Google Scholar
Cross Ref
- Yannick Benezeth, Serge Bobbia, Keisuke Nakamura, Randy Gomez, and Julien Dubois. 2019. Probabilistic Signal Quality Metric for Reduced Complexity Unsupervised Remote Photoplethysinography. 1--5. Google ScholarCross Ref
- Serge Bobbia, Richard Macwan, Yannick Benezeth, Alamin Mansouri, and Julien Dubois. 2017. Unsupervised skin tissue segmentation for remote photoplethysmography. Pattern Recognition Letters (10 2017). Google ScholarDigital Library
- Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger, and Roopak Shah. 1993. Signature Verification Using a "Siamese" Time Delay Neural Network. In Proceedings of the 6th International Conference on Neural Information Processing Systems (NIPS'93). Morgan Kaufmann Publishers Inc., San Francisco, CA, USA. 737--744. http://dl.acm.org/citation.cfm?id=2987189.2987282Google ScholarDigital Library
- Weixuan Chen and Daniel McDuff. 2018. DeepPhys: Video-Based Physiological Measurement Using Convolutional Attention Networks. In Computer Vision - ECCV 2018, Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss (Eds.). Springer International Publishing, Cham, 356--373.Google ScholarDigital Library
- Weixuan Chen and Daniel J. McDuff. 2018. DeepMag: Source Specific Motion Magnification Using Gradient Ascent. CoRR abs/1808.03338 (2018). arXiv:1808.03338 http://arxiv.org/abs/1808.03338Google Scholar
- Gerard de Haan and Vincent Jeanne. 2013. Robust Pulse Rate From Chrominance-Based rPPG. IEEE Transactions on Biomedical Engineering 60, 10 (Oct 2013). 2878--2886. Google ScholarCross Ref
- Javier Hernandez-Ortega, Julian Fierrez, Aythami Morales, and Pedro Tome. 2018. Time Analysis of Pulse-Based Face Anti-Spoofing in Visible and NIR. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Google Scholar
- A. B. Hertzmann. 1937. Observations on the Finger Volume Pulse Recorded Photo-electrically. American Journal of Physiology 119 (1937), 334--335.Google Scholar
- Guillaume Heusch, André Anjos, and Sébastien Marcel. 2017. A Reproducible Study on Remote Heart Rate Measurement. CoRR abs/1709.00962 (2017). arXiv:1709.00962 http://arxiv.org/abs/1709.00962Google Scholar
- Branislav HollÃd'nder. 2018. Siamese Networks: Algorithm, Applications And PyTorch Implementation. https://becominghuman.ai/Siamese-networks-algorithm-applications-and-pytorch-implementation-4ffa3304c18 Last accessed on Aug 29, 2019.Google Scholar
- Davis E. King. 2009. Dlib-ml: A Machine Learning Toolkit. Journal of Machine Learning Research 10 (2009), 1755--1758.Google ScholarDigital Library
- Georg Lempe, Sebastian Zaunseder, Tom Wirthgen, Stephan Zipser, and Hagen Malberg. 2013. ROI Selection for Remote Photo plethysmography. In Bildverarbeitung für die Medizin 2013. Springer Berlin Heidelberg, Berlin, Heidelberg, 99--103.Google Scholar
- Peixi Li, Keisuke Nakamura Yannick Benezeth, Randy Gomez, and Fan Yang. 2019. Model-based Region of Interest Segmentation for Remote Photoplethysinography. In 14th International Conference on Computer Vision Theory and Applications. 383--388.Google Scholar
- Xiaobai Li, Iman Alikhani, Jingang Shi, Tapio Seppanen, Juhani Junttila, Kirsi Majamaa-Voltti, Mikko Tulppo, and Guoying Zhao. 2018. The OBF Database: A Large Face Video Database for Remote Physiological Signal Measurement and Atrial Fibrillation Detection. In 2018 13th IEEE International Conference on Automatic Face Gesture Recognition (FG 2018). 242--249.Google Scholar
- Xiaobai Li, Jie Chen, Guoying Zhao, and Matti PietikÃd'inen. 2014. Remote Heart Rate Measurement from Face Videos under Realistic Situations. In 2014 IEEE Conference on Computer Vision and Pattern Recognition. 4264--4271. Google ScholarDigital Library
- Yaojie Liu, Amin Jourabloo, and Xiaoming Liu. 2018. Learning Deep Models for Face Anti-Spoofing: Binary or Auxiliary Supervision. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 389--398.Google ScholarCross Ref
- Richard Macwan, Yannick Benezeth, and Alamin Mansouri. 2019. Heart rate estimation using remote photoplethysmography with multi-objective optimization. Biomedical Signal Processing and Control 49 (03 2019), 24--33. Google ScholarCross Ref
- Richard Macwan, Serge Bobbia, Yannick Benezeth, Julien Dubois, and Alamin Mansouri. 2018. Periodic Variance Maximization Using Generalized Eigenvalue Decomposition Applied to Remote Photoplethysmography Estimation. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). 1413--14138. Google ScholarCross Ref
- Xuesong Niu, Hu Han, Shiguang Shan, and Xilin Chen. 2018. SynRhythm: Learning a Deep Heart Rate Estimator from General to Specific. In 2018 24th International Conference on Pattern Recognition (ICPR). 3580--3585.Google ScholarCross Ref
- Ming-Zher Poh, Daniel J. McDuff, and Rosalind W. Picard. 2010. Non-contact Automated Cardiac Pulse Measurements Using Video Imaging and Blind Source Separation. Optics Express 18, 10 (2010), 10762--10774.Google ScholarCross Ref
- Ying Qiu, Yang Liu, Juan Arteaga-Falconi, Haiwei Dong, and Abdulmotaleb E. Saddik. 2019. EVM-CNN: Real-Time Contactless Heart Rate Estimation From Facial Video. IEEE Transactions on Multimedia 21, 7 (July 2019), 1778--1787. Google ScholarCross Ref
- Ronny Stricker, Steffen MÃijller, and Horst-Michael Gross. 2014. Non-contact video-based pulse rate measurement on a mobile service robot. Proceedings - IEEE International Workshop on Robot and Human Interactive Communication 2014, 1056--1062. Google ScholarCross Ref
- Sergey Tulyakov, Xavier Alameda-Pineda, Elisa Ricci, Lijun Yin, Jeffrey F. Cohn, and Nicu Sebe. 2016. Self-Adaptive Matrix Completion for Heart Rate Estimation from Face Videos under Realistic Conditions. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2396--2404. Google ScholarCross Ref
- Radim Špetlík, Vojtěch Franc, Jan Čech, and Jiří Matas. 2018. Visual Heart Rate Estimation with Convolutional Neural Network. In Proceedings of British Machine Vision Conference.Google Scholar
- Wim Verkruysse, Lars O. Svaasand, and J S. Nelson. 2008. Remote Plethysmo-graphic Imaging Using Ambient Light. Optics Express 16, 26 (2008), 21434--21445.Google ScholarCross Ref
- Wenjin Wang, Albertus C. den Brinker, Sander Stuijk, and Gerard de Haan. 2017. Algorithmic Principles of Remote PPG. IEEE Transactions on Biomedical Engineering 64, 7 (July 2017), 1479--1491. Google ScholarCross Ref
- Wenjin Wang, Sander Stuijk, and Gerard de Haan. 2016. A Novel Algorithm for Remote Photoplethysmography: Spatial Subspace Rotation. IEEE Transactions on Biomedical Engineering 63, 9 (Sep. 2016), 1974--1984. Google ScholarCross Ref
- Zhi-Kuan Wang, Ying Kao, and Chiou-Ting Hsu. 2019. Vision-based Heart Rate Estimation via a Two-stream CNN. In 2019 IEEE International Conference on Image Processing (ICIP). 3327--3331. Google ScholarCross Ref
- Sun Yu, Sijung Hu, Vicente Azorin-Peris, Jonathon A. Chambers, Yisheng Zhu, and Stephen E. Greenwald. 2011. Motion-compensated noncontact imaging photoplethysmography to monitor cardiorespiratory status during exercise. Journal of biomedical optics 16 (07 2011), 077010. Google ScholarCross Ref
- Zitong Yu, Xiaobai Li, and Guoying Zhao. 2019. Recovering remote Photoplethysmograph Signal from Facial videos Using Spatio-Temporal Convolutional Networks. CoRR abs/1905.02419 (2019). arXiv:1905.02419 http://arxiv.org/abs/1905.02419Google Scholar
Index Terms
- Siamese-rPPG network: remote photoplethysmography signal estimation from face videos
Recommendations
Multi-scale Multi-structure Siamese Network (MMSNet) for Primary Open-Angle Glaucoma Prediction
Machine Learning in Medical ImagingAbstractPrimary open-angle glaucoma (POAG) is one of the leading causes of irreversible blindness in the United States and worldwide. POAG prediction before onset plays an important role in early treatment. Although deep learning methods have been ...
Prediction of cardiovascular risk by measuring carotid intima media thickness from an ultrasound image for type II diabetic mellitus subjects using machine learning and transfer learning techniques
AbstractCardiovascular disease (CVD) is a fatal disease that causes increased death in developing and developed nations. Among the various reasons, the increase in carotid intima media thickness (CIMT) is also a significant reason for CVD. It is expected ...
Augmentation of rPPG Benchmark Datasets: Learning to Remove and Embed rPPG Signals via Double Cycle Consistent Learning from Unpaired Facial Videos
Computer Vision – ECCV 2022AbstractRemote estimation of human physiological condition has attracted urgent attention during the pandemic of COVID-19. In this paper, we focus on the estimation of remote photoplethysmography (rPPG) from facial videos and address the deficiency issues ...
Comments