Abstract
The remote detection of liveness is critical for senior and baby care, disaster response, the military, and law enforcement. Existing solutions are mostly based on special sensor hardware or the spectral signature of living skin. This paper uses commercial electro-optical and infrared (EO/IR) sensors to capture a very short video for low cost and fast liveness detection. The key components of our system include: tiny human body and face detection from long range and low-resolution video, and remote liveness detection based on micro-motion from a short human body and face video. These micro-motions are caused by breathing and heartbeat. A deep learning architecture is designed for remote body and face detection. A novel algorithm is proposed for adaptive sensor and background noise cancellation. An air platform motion compensation algorithm is tested on video data collected on a drone. The key advantages are: low cost, requires very short video, works with many parts of a human body even when skin is not visible, works on any motion caused by eyes, mouth, heartbeat, breathing, or body parts, and works in all lighting conditions. To the author’s best knowledge, this is the first work on video micro-motion based liveness detection on a moving platform and from a long standoff range of 100 m. Once a subject is deemed alive, video-based remote heart rate detection is applied to assess the physiological and psychological state of the subject. This is also the first work on outdoor remote heart rate detection from a long standoff range of 100 m. On a public available indoor COHFACE data evaluation, our heart rate estimation algorithm outperforms all published work on the same dataset.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Balakrishnan, G., Durand, F., Guttag, J.: Detecting pulse from head motions in video. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3430–3437 (2013)
Hu, P., Ramanan, D.: Finding tiny faces. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 951–959 (2017)
Neumann, L., Vedaldi, A.: Tiny people pose. In: Jawahar, C. V., Li, Hongdong, Mori, Greg, Schindler, Konrad (eds.) ACCV 2018. LNCS, vol. 11363, pp. 558–574. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-20893-6_35
Polonskiy, L., et al.: U.S. Patent No. 7,417,727. U.S. Patent and Trademark Office, Washington, DC (2008)
Rice, R.R., Zediker, M.S.: U.S. Patent No. 5,867,257. U.S. Patent and Trademark Office, Washington, DC (1999)
Yazdi, M., Bouwmans, T.: New trends on moving object detection in video images captured by a moving camera: a survey. Comput. Sci. Rev. 28, 157–177 (2018)
Yuen, A., Droitcour, A., Madsen, A.H., Park, B.K., El Hourani, C., Shing, T.: U.S. Patent No. 8,454,528. U.S. Patent and Trademark Office, Washington, DC (2013)
Deng, Y., Kumar, A.: Standoff Heart Reate Estimation from Video –A Review, SPIE Defense + Commercial Sensing Expo, April 2020, Online, CA
Verkruysse, W., Svaasand, L.O., Nelson, J.S.: Remote plethysmographic imaging using ambient light. Opt. Express 16(26), 21434–21445 (2008). https://doi.org/10.1364/OE.16.021434
Procházka, A., Schätz, M., Vyšata, O., Vališ, M.: Microsoft kinect visual and depth sensors for breathing and heart rate analysis. Sensors 16(7), 996 (2016)
Rouast, P.V., Adam, M.T.P., Chiong, R., Cornforth, D., Lux, E.: Remote heart rate measurement using low-cost RGB face video: a technical literature review. Front. Comput. Sci. 12(5), 858–872 (2018). https://doi.org/10.1007/s11704-016-6243-6
Viola, P., Jones, M.J.: Robust real-time face detection. Int. J. Comput. Vision 57(2), 137–154 (2004). https://doi.org/10.1023/B:VISI.0000013087.49260.fb
Lin, Y.C., Lin, Y.H.: A study of color illumination effect on the SNR of rPPG signals. In: 2017 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 4301–4304. IEEE, July 2017
Martinez, N., Bertran, M., Sapiro, G., Wu, H.: Non-contact photoplethysmogram and instantaneous heart rate estimation from infrared face video. In: 2019 IEEE International Conference on Image Processing (ICIP), Taipei, Taiwan, pp. 2020–2024 (2019)
The COHFACE Dataset made available by the Idiap Research Institute, Martigny, Switzerland. https://www.idiap.ch/dataset/cohface
Heusch, G., Anjos, A., Marcel, S.: A reproducible study on remote heart rate measurement. arXiv preprint arXiv:1709.00962 (2017)
Špetlík, R., Franc, V., Matas, J.: Visual heart rate estimation with convolutional neural network. In: Proceedings of the British Machine Vision Conference, Newcastle, UK, pp. 3–6, September 2018
Alahi, A., Ortiz, R., Vandergheynst, P.: Freak: fast retina keypoint. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 510–517. IEEE, June 2012
Torr, P.H., Murray, D.W.: The development and comparison of robust methods for estimating the fundamental matrix. Int. J. Comput. Vision 24(3), 271–300 (1997). https://doi.org/10.1023/A:1007927408552
Seguin, C., Blaquière, G., Loundou, A., Michelet, P., Markarian, T.: Unmanned aerial vehicles (drones) to prevent drowning. Resuscitation 127, 63–67 (2018)
Pensieri, M.G., Garau, M., Barone, P.M.: Drones as an integral part of remote sensing technologies to help missing people. Drones 4(2), 15 (2020)
FLIR ONE PRO User Guide, Third Gneration for Android and iOS
Carreiras, C., Alves, A.P., Lourenço, A., Canento, F., Silva, H., Fred, A., et al.: BioSPPy - Biosignal Processing in Python (2015)
De Haan, G., Jeanne, V.: Robust pulse rate from chrominance-based rPPG. IEEE Trans. Biomed. Eng. 60(10), 2878–2886 (2013)
Li, X., Chen, J., Zhao, G., Pietikainen, M.: Remote heart rate measurement from face videos under realistic situations. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2014)
Wang, W., Stuijk, S., De Haan, G.: A novel algorithm for remote photoplethysmography: spatial subspace rotation. IEEE Trans. Biomed. Eng. 63(9), 1974–1984 (2015)
Zhong, Y., Deng, Y., Meltzner, G.: Pace independent mobile gait biometrics. In: 2015 IEEE 7th International Conference on Biometrics Theory, Applications and Systems (BTAS), September 2015
Acknowledgment
The author of this paper would like to thank the Empower team of BAE Systems, Inc. for funding this project. Volunteers from the Empower team helped with data collection for this research. The author would also like to thank Dr. Stephen DelMarco, Mr. Derek Baker, ICPR and WAAMI reviewers for reviewing this paper.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Deng, Y. (2021). Remote Liveness and Heart Rate Detection from Video. In: Del Bimbo, A., et al. Pattern Recognition. ICPR International Workshops and Challenges. ICPR 2021. Lecture Notes in Computer Science(), vol 12668. Springer, Cham. https://doi.org/10.1007/978-3-030-68793-9_7
Download citation
DOI: https://doi.org/10.1007/978-3-030-68793-9_7
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-68792-2
Online ISBN: 978-3-030-68793-9
eBook Packages: Computer ScienceComputer Science (R0)
