Skip to main content

Advertisement

SpringerLink
Log in

Exploring the feasibility of seamless remote heart rate measurement using multiple synchronized cameras

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Heart rate (HR) measurement and monitoring is of great importance to determine the physiological and mental status of individuals. Recently, it has been demonstrated that HR can be remotely retrieved from facial video-based photoplethysmographic signals captured using consumer-grade cameras. However, in existing studies, subjects are mostly required to keep their facial regions of interest (ROIs) within one single camera. To make this technique usable in a daily life situation where subjects move around freely, we launch a preliminary simulated study of seamless remote HR measurement using multiple synchronized cameras by combining ensemble empirical mode decomposition (EEMD) with time-delay canonical correlation analysis (TDCCA), termed as EEMD-TDCCA. At each time point, a target ROI with the largest area is first determined from all the ROIs provided by all the cameras. Then, the RGB time sequence is formed by taking average of all pixels within each target ROI. Afterwards, the green channel time sequence is decomposed into several intrinsic mode functions (IMFs) and only the IMF candidates, whose frequency corresponding to the maximum amplitude falling into the interested HR range will be further processed by TDCCA. Finally, the first pair of the canonical variables having the largest correlation coefficient is the HR source and the corresponding HR is derived by peak detection or frequency analysis. Thirty subjects were recruited and four state-of-the-art methods were employed for comparison. The best performance was achieved by using the proposed EEMD-TDCCA followed by frequency analysis, with the mean absolute error 4.11 bpm, mean percentage error 5.26%, root mean square error 5.37 bpm, the Pearson’s correlation coefficient 0.90 and the intra-class correlation coefficient 0.89, demonstrating the feasibility of our proposed seamless remote HR measurement framework. This study will provide a promising solution for practical and robust non-contact HR measurement applications.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Aarts LA, Jeanne V, Cleary JP, Lieber C, Nelson JS, Oetomo SB, Verkruysse W (2013) Non-contact heart rate monitoring utilizing camera photoplethysmography in the neonatal intensive care unit a pilot study. Early Hum Dev 89(12):943–948

    Google Scholar 

  2. Al-Naji A, Perera AG, Chahl J (2017) Remote monitoring of cardiorespiratory signals from a hovering unmanned aerial vehicle. Biomed Eng Online 16(1):101

    Google Scholar 

  3. Amelard R, Scharfenberger C, Kazemzadeh F, Pfisterer KJ, Lin BS, Clausi DA, Wong A (2015) Feasibility of long-distance heart rate monitoring using transmittance photoplethysmographic imaging (ppgi). Sci Rep 5:14637

    Google Scholar 

  4. Blackford EB, Estepp JR (2017) Using consumer-grade devices for multi-imager non-contact imaging photoplethysmography. In: Optical Diagnostics and Sensing XVII: Toward Point-of-Care Diagnostics, vol. 10072. International Society for Optics and Photonics, pp 100720P

  5. Caruccio L, Polese G, Tortora G, Iannone D (2019) Edcar: a knowledge representation framework to enhance automatic video surveillance. Expert Syst Appl 131:190–207

    Google Scholar 

  6. Chen X, Wang ZJ, McKeown M (2013) A three-step multimodal analysis framework for modeling corticomuscular activity with application to parkinson’s disease. IEEE J Biomed Health Inf 18(4):1232–1241

    Google Scholar 

  7. Chen X, Liu A, Peng H, Ward R (2014) A preliminary study of muscular artifact cancellation in single-channel eeg. Sensors 14(10):18 370–18 389

    Google Scholar 

  8. Chen D-Y, Wang J-J, Lin K-Y, Chang H-H, Wu H-K, Chen Y-S, Lee S-Y (2015) Image sensor-based heart rate evaluation from face reflectance using hilbert–huang transform. IEEE Sens J 15(1):618–627

    Google Scholar 

  9. Chen X, Liu A, Chiang J, Wang ZJ, McKeown M, Ward RK (2015) Removing muscle artifacts from eeg data: Multichannel or single-channel techniques? IEEE Sens J 16(7):1986–1997

    Google Scholar 

  10. Chen X, Wang ZJ, McKeown M (2016) Joint blind source separation for neurophysiological data analysis: Multiset and multimodal methods. IEEE Signal Proc Mag 33(3):86–107

    Google Scholar 

  11. Chen X, Chen Q, Zhang Y, Wang ZJ (2018) A novel eemd-cca approach to removing muscle artifacts for pervasive eeg. IEEE Sens J 19(19):8420–8431

    Google Scholar 

  12. Chen X, Cheng J, Song R, Liu Y, Ward R, Wang ZJ (2018) Video-based heart rate measurement: Recent advances and future prospects. IEEE Transactions on Instrumentation and Measurement

  13. Chen X., Xu X., Liu A., McKeown M. J., Wang Z. J. (2018) The use of multivariate emd and cca for denoising muscle artifacts from few-channel EEG recordings. IEEE Trans Instrum Measur 67(2):359–370

    Google Scholar 

  14. Cheng J, Chen X, Xu L, Wang ZJ (2017) Illumination variation-resistant video-based heart rate measurement using joint blind source separation and ensemble empirical mode decomposition. IEEE J Biomed Health Inf 21(5):1422–1433

    Google Scholar 

  15. Correa NM, Adali T, Li Y-O, Calhoun VD (2010) Canonical correlation analysis for data fusion and group inferences. IEEE Signal Process Mag 27(4):39–50

    Google Scholar 

  16. De Haan G, Jeanne V (2013) Robust pulse rate from chrominance-based rppg. IEEE Trans Biomed Eng 60(10):2878–2886

    Google Scholar 

  17. Estepp JR, Blackford EB, Meier CM (2014) Recovering pulse rate during motion artifact with a multi-imager array for non-contact imaging photoplethysmography. In: 2014 IEEE International Conference on Systems, Man, and Cybernetics (SMC). IEEE, pp 1462–1469

  18. Feng L, Po L-M, Xu X, Li Y, Ma R (2015) Motion-resistant remote imaging photoplethysmography based on the optical properties of skin. IEEE Trans Circ Syst Video Technol 25(5):879–891

    Google Scholar 

  19. Guo Z, Wang ZJ, Shen Z (2014) Physiological parameter monitoring of drivers based on video data and independent vector analysis. In: 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, pp 4374–4378

  20. Haque MA, Irani R, Nasrollahi K, Moeslund TB (2016) Heartbeat rate measurement from facial video. IEEE Intell Syst 31(3):40–48

    Google Scholar 

  21. Hassan M, Malik A, Fofi D, Saad N, Meriaudeau F (2017) Novel health monitoring method using an rgb camera. Biomed Opt Express 8(11):4838–4854

    Google Scholar 

  22. Hills M (1987) The design and analysis of clinical experiments. J R Stat Soc Ser A (Gen) 150(4):400–400

    Google Scholar 

  23. Holton BD, Mannapperuma K, Lesniewski PJ, Thomas JC (2013) Signal recovery in imaging photoplethysmography. Physiol Measur 34(11):1499

    Google Scholar 

  24. Hotelling H (1935) Relations between 2 sets of variants. Biometrika 28(3–4):312–377

    Google Scholar 

  25. Huang NE, Shen Z, Long SR, Wu MC, Shih HH, Zheng Q, Yen N-C, Tung CC, Liu H (1998) The empirical mode decomposition and the hilbert spectrum for nonlinear and non-stationary time series analysis. Proc R Soc Lond Ser A: Math Phys Eng Sci 454(1971):903–995

    MathSciNet  MATH  Google Scholar 

  26. Kalal Z, Mikolajczyk K, Matas J (2012) Tracking-learning-detection. IEEE Trans Pattern Anal Mach Intell 34(7):1409–1422

    Google Scholar 

  27. Lee D, Kim J, Kwon S, Park K (2015) Heart rate estimation from facial photoplethysmography during dynamic illuminance changes. In: 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). IEEE, pp 2758–2761

  28. Leonhardt S, Leicht L, Teichmann D (2018) Unobtrusive vital sign monitoring in automotive environments a review. Sensors 18(9):3080

    Google Scholar 

  29. Lewandowska M, Rumiński J, Kocejko T, Nowak J (2011) Measuring pulse rate with a webcam a non-contact method for evaluating cardiac activity. In: 2011 federated conference on computer science and information systems (FedCSIS). IEEE, pp 405–410

  30. Li X, Chen J, Zhao G, Pietikainen M (2014) Remote heart rate measurement from face videos under realistic situations. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4264–4271

  31. Liu H, Sun X (2016) Linear canonical correlation analysis based ranking approach for facial age estimation. In: 2016 IEEE International Conference on Image Processing (ICIP). IEEE, pp 3249–3253

  32. McDuff DJ, Blackford EB, Estepp JR (2018) Fusing partial camera signals for noncontact pulse rate variability measurement. IEEE Trans Biomed Eng 65(8):1725–1739

    Google Scholar 

  33. Monkaresi H, Calvo RA, Yan H (2014) A machine learning approach to improve contactless heart rate monitoring using a webcam. IEEE J Biomed Health Inf 18(4):1153–1160

    Google Scholar 

  34. Po L-M, Feng L, Li Y, Xu X, Cheung TC-H, Cheung K-W (2018) Block-based adaptive roi for remote photoplethysmography. Multimed Tools Appl 77 (6):6503–6529

    Google Scholar 

  35. Poh M-Z, McDuff DJ, Picard RW (2010) Non-contact, automated cardiac pulse measurements using video imaging and blind source separation. Opt Express 18 (10):10 762–10 774

    Google Scholar 

  36. Poh M-Z, McDuff DJ, Picard RW (2011) Advancements in noncontact, multiparameter physiological measurements using a webcam. IEEE Trans Biomed Eng 58(1):7–11

    Google Scholar 

  37. Qi H, Guo Z, Chen X, Shen Z, Wang ZJ (2017) Video-based human heart rate measurement using joint blind source separation. Biomed Signal Process Control 31:309–320

    Google Scholar 

  38. Rasche S, Trumpp A, Waldow T, Gaetjen F, Plötze K, Wedekind D, Schmidt M, Malberg H, Matschke K, Zaunseder S (2016) Camera-based photoplethysmography in critical care patients. Clin Hemorheol Microcirc 64(1):77–90

    Google Scholar 

  39. Rouast PV, Adam MT, Chiong R, Cornforth D, Lux E (2018) Remote heart rate measurement using low-cost rgb face video: a technical literature review. Front Comput Sci 12(5):858–872

    Google Scholar 

  40. Soneson C, Lilljebjörn H, Fioretos T, Fontes M (2010) Integrative analysis of gene expression and copy number alterations using canonical correlation analysis. BMC Bioinform 11(1):191

    Google Scholar 

  41. Sun Y, Azorin-Peris V, Kalawsky R, Hu S, Papin C, Greenwald SE (2012) Use of ambient light in remote photoplethysmographic systems: comparison between a high-performance camera and a low-cost webcam. J Biomed Opt 17(3):037005

  42. Sun Y, Thakor N (2016) Photoplethysmography revisited: from contact to noncontact, from point to imaging. IEEE Trans Biomed Eng 63(3):463–477

    Google Scholar 

  43. Tarassenko L, Villarroel M, Guazzi A, Jorge J, Clifton D, Pugh C (2014) Non-contact video-based vital sign monitoring using ambient light and auto-regressive models. Physiol Measur 35(5):807

    Google Scholar 

  44. Tarvainen MP, Ranta-Aho PO, Karjalainen PA (2002) An advanced detrending method with application to hrv analysis. IEEE Trans Biomed Eng 49(2):172–175

    Google Scholar 

  45. Verkruysse W, Svaasand LO, Nelson JS (2008) Remote plethysmographic imaging using ambient light. Opt Express 16(26):21 434–21 445

    Google Scholar 

  46. Wang W, Stuijk S, De Haan G (2016) A novel algorithm for remote photoplethysmography: Spatial subspace rotation. IEEE Trans Biomed Eng 63 (9):1974–1984

    Google Scholar 

  47. Wang W, den Brinker AC, Stuijk S, de Haan G (2017) Algorithmic principles of remote ppg. IEEE Trans Biomed Eng 64(7):1479–1491

    Google Scholar 

  48. Wu Z, Huang NE (2009) Ensemble empirical mode decomposition: a noise-assisted data analysis method. Adv Adapt Data Anal 1(01):1–41

    Google Scholar 

  49. Xu L, Cheng J, Chen X (2017) Illumination variation interference suppression in remote ppg using pls and memd. Electron Lett 53(4):216–218

    Google Scholar 

  50. Yang Z, Yang X, Wu X (2019) Motion-tolerant heart rate estimation from face videos using derivative filter. Multimed Tools Appl 78:26747–26757

    Google Scholar 

  51. Yu S, Hu S, Azorin-Peris V, Chambers JA, Zhu Y, Greenwald SE (2011) Motion-compensated noncontact imaging photoplethysmography to monitor cardiorespiratory status during exercise. J Biomed Opt 16(7):077010

  52. Yu Y-P, Raveendran P, Lim C-L, Kwan B-H (2015) Dynamic heart rate estimation using principal component analysis. Biomed Opt Express 6(11):4610–4618

    Google Scholar 

  53. Zhao F, Li M, Jiang Z, Tsien JZ, Lu Z (2016) Camera-based, non-contact, vital-signs monitoring technology may provide a way for the early prevention of sids in infants. Front Neurol 7:236

    Google Scholar 

  54. Zheng S, Sturgess P, Torr PH (2013) Approximate structured output learning for constrained local models with application to real-time facial feature detection and tracking on low-power devices. In: 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG). IEEE, pp 1–8

Download references

Acknowledgments

This work was supported by the National Key R&D Program of China (Grant 2017YFB1002802), National Natural Science Foundation of China (Grants: 61922075, 81571760, 61701160 and 41901350) and the Fundamental Research Funds for the Central Universities (Grants: JZ2019HGBZ0151 and JZ2020HGPA0111).

Author information

Authors and Affiliations

  1. Department of Biomedical Engineering, Hefei University of Technology, Hefei, 230009, China

    Juan Cheng, Xingmao Wang, Rencheng Song, Yu Liu & Chang Li

  2. Department of Electronic Engineering, Information Science, University of Science and Technology of China, Hefei, 230026, China

    Xun Chen

Authors
  1. Juan Cheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xingmao Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Rencheng Song
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yu Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Chang Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Xun Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Rencheng Song.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Cheng, J., Wang, X., Song, R. et al. Exploring the feasibility of seamless remote heart rate measurement using multiple synchronized cameras. Multimed Tools Appl 79, 23023–23043 (2020). https://doi.org/10.1007/s11042-020-09075-2

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-020-09075-2

Keywords

Search

Navigation