Skip to main content

Advertisement

SpringerLink
Log in

Doctor Robots: Design and Implementation of a Heart Rate Estimation Algorithm

  • Published:
International Journal of Social Robotics Aims and scope Submit manuscript

Abstract

Populations are ageing and the healthcare costs are increasing accordingly. Humanoid Robots (HRs) perform basic but crucial health checkups such as the heart rate can efficiently meet the healthcare demands. This paper develops a 9-stage heart rate estimation algorithm and implements it to an HR. The 9-stages cover the recognition of the face with the Viola–Jones algorithm, determination of the facial regions with the geometric-based facial distance measurement technique, extraction of the forehead and cheek regions, tracking of these facial regions with the Hierarchical Multi Resolution algorithm, decomposition of the facial regions in the Red–Green–Blue (RGB) color channels, averaging and normalization of the RGB color data, elimination of the artifacts with the Adaptive Independent Component Analysis (ICA) technique, calculating the power spectrum of the data with the Fast Fourier Transform (FFT) technique, and finally determining the peaks inside the threshold reflecting the human heart rate boundaries. One of the key contributions of this paper is building and incorporating the Hierarchical Multi Resolution technique in the heart rate estimation algorithm to eliminate the deteriorating effects of the human and camera motions. A further contribution of this paper is generating a rule-based approach to discard the effects of the sudden movements. These two contributions have noticeably improved the accuracy of the heart rate estimation algorithm in the dynamic environments. The algorithm has been assessed extensively with 5 different experimental scenarios consisting of 33 conditions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21

Similar content being viewed by others

Data availability

Not applicable.

References

  1. American Heart Association (2018) All about heart rate (pulse) | American Heart Association. Heart.Org. https://www.heart.org/en/health-topics/high-blood-pressure/the-facts-about-high-blood-pressure/all-about-heart-rate-pulse accessed 06 Oct 2021

  2. Poh M-Z, McDuff DJ, Picard RW (2010) Non-contact, automated cardiac pulse measurements using video imaging and blind source separation. Opt Express 18(10):10762. https://doi.org/10.1364/oe.18.010762

    Article  Google Scholar 

  3. Poh MZ, McDuff DJ, Picard RW (2011) Advancements in noncontact, multiparameter physiological measurements using a webcam. IEEE Trans Biomed Eng 58(1):7–11. https://doi.org/10.1109/TBME.2010.2086456

    Article  Google Scholar 

  4. Viola P, Jones M (2001) Rapid object detection using a boosted cascade of simple features. Proc IEEE Comput Soc Conf Comput Vis Pattern Recogn. https://doi.org/10.1109/cvpr.2001.990517

    Article  Google Scholar 

  5. M Lewandowska M, Rumiński J, Kocejko T, Nowak J (2011) Measuring pulse rate with a webcam—a non-contact method for evaluating cardiac activity. In 2011 federated conference on computer science and information systems, FedCSIS 2011, pp 405–410

  6. Balakrishnan G, Durand F, Guttag J (2013) Detecting pulse from head motions in video. Proc IEEE Comput Soc Conf Comput Vis Pattern Recognit. https://doi.org/10.1109/CVPR.2013.440

    Article  Google Scholar 

  7. Xu S, Sun L, Rohde GK (2014) Robust efficient estimation of heart rate pulse from video. Biomed Opt Express 5(4):1124. https://doi.org/10.1364/boe.5.001124

    Article  Google Scholar 

  8. Kumar M, Veeraraghavan A, Sabharwal A (2015) DistancePPG: robust non-contact vital signs monitoring using a camera. Biomed Opt Express 6(5):1565. https://doi.org/10.1364/boe.6.001565

    Article  Google Scholar 

  9. Li X, Chen J, Zhao G, Pietikainen M (2014) Remote heart rate measurement from face videos under realistic situations. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition, pp 4264–4271. https://doi.org/10.1109/CVPR.2014.543

  10. Soleymani M, Lichtenauer J, Pun T, Pantic M (2012) A multimodal database for affect recognition and implicit tagging. IEEE Trans Affect Comput 3(1):42–55. https://doi.org/10.1109/T-AFFC.2011.25

    Article  Google Scholar 

  11. Wang W, Stuijk S, De Haan G (2016) A novel algorithm for remote photoplethysmography: spatial subspace rotation. IEEE Trans Biomed Eng 63(9):1974–1984. https://doi.org/10.1109/TBME.2015.2508602

    Article  Google Scholar 

  12. Chen DY et al (2015) Image sensor-based heart rate evaluation from face reflectance using Hilbert–Huang transform. IEEE Sens J 15(1):618–627. https://doi.org/10.1109/JSEN.2014.2347397

    Article  Google Scholar 

  13. Farnebäck G (2003) Two-frame motion estimation based on polynomial expansion. Lect Notes Comput Sci 2749:363–370. https://doi.org/10.1007/3-540-45103-x_50

    Article  MATH  Google Scholar 

  14. De Haan G, Jeanne V (2013) Robust pulse-rate from chrominance-based rPPG. IEEE Trans Biomed Eng 60:1

    Article  Google Scholar 

  15. De Haan G, Van Leest A (2014) Improved motion robustness of remote-PPG by using the blood volume pulse signature. Physiol Meas 35(9):1913–1926. https://doi.org/10.1088/0967-3334/35/9/1913

    Article  Google Scholar 

  16. Lam A, Kuno Y (2015) Robust heart rate measurement from video using select random patches. In: Proceedings of the IEEE international conference on computer vision, vol 2015 Inter, pp 3640–3648. https://doi.org/10.1109/ICCV.2015.415

  17. Feng L, Po LM, Xu X, Li Y, Ma R (2015) Motion-resistant remote imaging photoplethysmography based on the optical properties of skin. IEEE Trans Circ Syst Video Technol 25(5):879–891. https://doi.org/10.1109/TCSVT.2014.2364415

    Article  Google Scholar 

  18. Haque MA, Irani R, Nasrollahi K, Moeslund TB (2016) Heartbeat rate measurement from facial video. IEEE Intell Syst 31(3):40–48. https://doi.org/10.1109/MIS.2016.20

    Article  Google Scholar 

  19. Abdi J, Al-Hindawi A, Ng T, Vizcaychipi MP (2018) Scoping review on the use of socially assistive robot technology in elderly care. BMJ Open. https://doi.org/10.1136/bmjopen-2017-018815

    Article  Google Scholar 

  20. Briggs P, Scheutz M, Tickle-Degnen L (2015) Are robots ready for administering health status surveys’: first results from an HRI study with subjects with Parkinson’s disease. In: ACM/IEEE international conference on human-robot interaction, Mar. 2015, vol 2015, pp 327–334. https://doi.org/10.1145/2696454.2696476

  21. Wilson JR, Tickle-Degnen L, Scheutz M (2016) Designing a social robot to assist in medication sorting. Lecture Not Comput Sci 9979:211–221. https://doi.org/10.1007/978-3-319-47437-3_21

    Article  Google Scholar 

  22. Kim GH et al (2015) Structural brain changes after traditional and robot-assisted multi-domain cognitive training in community-dwelling healthy elderly. PLoS ONE 10(4):e0123251. https://doi.org/10.1371/journal.pone.0123251

    Article  Google Scholar 

  23. Johnson DO et al (2013) Socially assistive robots: a comprehensive approach to extending independent living. Int J Soc Robot 6(2):195–211. https://doi.org/10.1007/s12369-013-0217-8

    Article  Google Scholar 

  24. Torta E et al (2014) Evaluation of a small socially-assistive humanoid robot in intelligent homes for the care of the elderly. J Intell Robot Syst Theory Appl 76(1):57–71. https://doi.org/10.1007/s10846-013-0019-0

    Article  Google Scholar 

  25. Johnson DO, Cuijpers RH, Pollmann K, Ven AAJ (2016) Exploring the entertainment value of playing games with a humanoid robot. Int J Soc Robot 8(2):247–269. https://doi.org/10.1007/S12369-015-0331-X

    Article  Google Scholar 

  26. Broadbent E et al (2014) Robots in older people’s homes to improve medication adherence and quality of life: a randomised cross-over trial. Lect Notes Comput Sci 8755:64–73. https://doi.org/10.1007/978-3-319-11973-1_7

    Article  Google Scholar 

  27. Beran TN, Ramirez-Serrano A, Vanderkooi OG, Kuhn S (2013) Reducing children’s pain and distress towards flu vaccinations: a novel and effective application of humanoid robotics. Vaccine 31(25):2772–2777. https://doi.org/10.1016/j.vaccine.2013.03.056

    Article  Google Scholar 

  28. Zhao M, Zhao Q, Liu Q, Li N, Peng H, Hu B (2013) A humanoid robot used as an assistive intervention tool for children with autism spectrum disorder: a preliminary research. Lecture Notes Comput Sci 8211:336–347. https://doi.org/10.1007/978-3-319-02753-1_34

    Article  Google Scholar 

  29. Tapus A, Ţǎpuş C, Matarić MJ (2009) The use of socially assistive robots in the design of intelligent cognitive therapies for people with dementia. In: 2009 IEEE international conference on rehabilitation robotics, ICORR 2009, pp 924–929. https://doi.org/10.1109/ICORR.2009.5209501

  30. Kidd CD (2008) Designing for long-term human–robot interaction and application to weight loss (Doctoral dissertation, Massachusetts Institute of Technology).

  31. Göngör F, Tutsoy Ö (2019) Design and implementation of a facial character analysis algorithm for humanoid robots. Robotica 37(11):1850–1866. https://doi.org/10.1017/S0263574719000304

    Article  Google Scholar 

  32. Göngör F, Tutsoy Ö, Barkana DE, Köse H (2017) An emotion analysis algorithm and implementation to NAO humanoid robot. In: Technol. Eng. Sci. https://www.researchgate.net/publication/321704164 accessed 07 Oct 2021

  33. Göngör F, Tutsoy Ö (2018) Humanoid robots learn and recognize seven facial emotions with ANN. https://www.researchgate.net/publication/329557920 accessed 07 Oct 2021

  34. Göngör F, Tutsoy Ö (2018) Eigenface based emotion analysis algorithm and implementation to humanoid robot. Int Sci Acad Congr. pp 1–17. https://www.researchgate.net/publication/329557854_Eigenface_Based_Emotion_Analysis_Algorithm_and_Implementation_to_Humanoid_Robot accessed 07 Oct 2021

  35. Werner P, Al-Hamadi A, Walter S, Gruss S, Traue HC (2014) Automatic heart rate estimation from painful faces. In: 2014 IEEE international conference on image processing, ICIP 2014, pp 1947–1951. https://doi.org/10.1109/ICIP.2014.7025390

  36. Kwon S, Kim J, Lee D, Park K (2015) ROI analysis for remote photoplethysmography on facial video. In: Proceedings of the annual international conference of the IEEE engineering in medicine and biology society, EMBS, vol 2015-Novem, pp 4938–4941. https://doi.org/10.1109/EMBC.2015.7319499

  37. Hassan MA, Malik AS, Fofi D, Saad N, Meriaudeau F (2017) Novel health monitoring method using an RGB camera. Biomed Opt Express 8(11):4838. https://doi.org/10.1364/boe.8.004838

    Article  Google Scholar 

  38. Ohnishi N, Kameda Y, Imiya A, Dorst L, Klette R (2008) Dynamic multiresolution optical flow computation. Lecture Notes Comput Sci 4931:1–15. https://doi.org/10.1007/978-3-540-78157-8_1

    Article  Google Scholar 

  39. Woo DM, Do VD (2016) Multi-resolution estimation of optical flow for vehicle tracking. Contemp Eng Sci 9(17):843–851. https://doi.org/10.12988/ces.2016.6696

    Article  Google Scholar 

  40. Schwartz GE, Weinberger DA, Singer JA (1981) Cardiovascular differentiation of happiness, sadness, anger, and fear following imagery and exercise. Psychosom Med 43(4):343–364. https://doi.org/10.1097/00006842-198108000-00007

    Article  Google Scholar 

  41. Sinha R, Lovallo WR, Parsons OA (1992) Cardiovascular differentiation of emotions. Psychosom Med 54(4):422–435. https://doi.org/10.1097/00006842-199207000-00005

    Article  Google Scholar 

Download references

Acknowledgements

This research is funded by The Turkish Council of Higher Education (YÖK) and The Scientific and Research Council of Turkey (TÜBİTAK).

Author information

Authors and Affiliations

  1. Electrical and Electronic Engineering Department, Adana Alparslan Turkes Science and Technology University, Adana, Turkey

    Fatma Gongor & Onder Tutsoy

Authors
  1. Fatma Gongor
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Onder Tutsoy
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Fatma Gongor.

Ethics declarations

Conflict of interest

Authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendıx

Appendıx

1.1 Algorithm 1: Hierarchical Multi-resolution Based Facial Region Tracking Algorithm

Goal: Let \(u\) be a feature point of the extracted facial region on the first image frame \(I_{t - 1}\). Find its corresponding updated location \(v\) on the next image frame \(I_{t}\).

figure a

Solution: The corresponding facial point is tracked and transferred to location \(v\) on the image frame \(I_{t}\) (Chart 1, Tables 6, 7, 8).

Chart 1
scheme 1

Adaptive ICA algorithm

Full size image
Table 6 Performance analysis of the proposed Hierarchical Multi Resolution based tracking algorithm for 1 subject from distance of 0.5 and 1 m. Where S: Small, M: Medium, F: Fast, D: Degree, Y: Yaw, P: Pitch, R: Roll, E: Everything is free
Full size table
Table 7 Performance analysis of the proposed heart rate estimation algorithm for 10 subjects from distance of 0.5 Meter
Full size table
Table 8 Performance analysis of the proposed heart rate estimation algorithm for 10 subjects from distance of 1 Meter
Full size table

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gongor, F., Tutsoy, O. Doctor Robots: Design and Implementation of a Heart Rate Estimation Algorithm. Int J of Soc Robotics 14, 1435–1461 (2022). https://doi.org/10.1007/s12369-022-00888-9

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12369-022-00888-9

Keywords

Search

Navigation