Abstract
In this work, we present an approach for non-contact automatic extraction of respiration in infants using infrared thermography video sequences, which were recorded in a neonatal intensive care unit. The respiratory signal was extracted in real-time on low-cost embedded GPUs by analyzing breathing-related temperature fluctuations in the nasal region. The automatic detection of the patient’s nose was performed using the Deep Learning-based YOLOv4-Tiny object detector. Additionally, the head was detected for movement tracking. A leave-one-out cross validation showed a mean intersection over union of 79% and a mean average precision of 93% for the detection algorithm. Since no clinical reference was provided, the extracted respiratory activity was validated for video sequences without motion artifacts using Farnebäck’s Optical Flow algorithm. A mean MAE of 8.5 breaths per minute and a mean \(\mathrm{F}_{1}\)-score of 80% for respiration detection were achieved. The model inference on NVIDIA Jetson modules showed a performance of 32 fps on the Xavier NX and 62 fps on the Xavier AGX. These outcomes showed promising results for the real-time extraction of respiratory activity from thermography recordings of neonates using Deep Learning-based techniques on embedded GPUs.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Abbas, A.K., Heimann, K., Jergus, K., Orlikowsky, T., Leonhardt, S.: Neonatal non-contact respiratory monitoring based on real-time infrared thermography. Biomed. Eng. Online 10(1), 1–17 (2011)
Berrar, D.: Cross-validation. In: Ranganathan, S., Gribskov, M., Nakai, K., Schönbach, C. (eds.) Encyclopedia of Bioinformatics and Computational Biology, pp. 542–545. Academic Press, Oxford (2019)
Bochkovskiy, A.: Yolo_mark - github repository (2019). https://github.com/AlexeyAB/Yolo_mark. Accessed 06 Dec 2021
Bochkovskiy, A., Wang, C.Y., Liao, H.Y.M.: Yolov4: optimal speed and accuracy of object detection. arXiv: 2004.10934 (2020)
Farnebäck, G.: Two-frame motion estimation based on polynomial expansion. In: Bigun, J., Gustavsson, T. (eds.) SCIA 2003. LNCS, vol. 2749, pp. 363–370. Springer, Heidelberg (2003). https://doi.org/10.1007/3-540-45103-X_50
Ferris, T., Shepley, M.: The design of neonatal incubators: a systems-oriented, human-centered approach. J. Perinatol. 33, S24–S31 (2013)
van Gastel, M., Stuijk, S., de Haan, G.: Robust respiration detection from remote photoplethysmography. Biomed. Opt. Express 7(12), 4941–4957 (2016)
Gleichauf, J., et al.: Automated non-contact respiratory rate monitoring of neonates based on synchronous evaluation of a 3D time-of-flight camera and a microwave interferometric radar sensor. Sensors 21(9), 2959 (2021)
Greneker, E.: Radar sensing of heartbeat and respiration at a distance with applications of the technology. In: Radar 97 (Conf. Publ. No. 449), pp. 150–154 (1997)
Heimann, K., Jergus, K., Abbas, A.K., Heussen, N., Leonhardt, S., Orlikowsky, T.: Infrared thermography for detailed registration of thermoregulation in premature infants. J. Perinat. Med. 1–8 (2013)
Hoog Antink, C., et al.: Fast body part segmentation and tracking of neonatal video data using deep learning. Med. Biol. Eng. Comput. 58(12), 3049–3061 (2020). https://doi.org/10.1007/s11517-020-02251-4
Hoog Antink, C., Lyra, S., Paul, M., Yu, X., Leonhardt, S.: A broader look: camera-based vital sign estimation across the spectrum. Yearb. Med. Inform. 28(01), 102–114 (2019)
Kinney, M., Howson, C., McDougall, L., Lawn, J.: Executive summary for born too soon: the global action report on preterm birth. March of Dimes, PMNCH (World Health Organization) (2012)
Kopaczka, M., Kolk, R., Schock, J., Burkhard, F., Merhof, D.: A thermal infrared face database with facial landmarks and emotion labels. IEEE Trans. Instrum. Meas. 1–13 (2019)
Liu, L., et al.: Global, regional, and national causes of under-5 mortality in 2000–15: an updated systematic analysis with implications for the sustainable development goals. LANCET 388 (2016)
Lorato, I., Bakkes, T., Stuijk, S., Meftah, M., De Haan, G.: Unobtrusive respiratory flow monitoring using a thermopile array: a feasibility study. Appl. Sci. 9(12), 2449 (2019)
Lorato, I., et al.: Multi-camera infrared thermography for infant respiration monitoring. Biomed. Opt. Express 11(9), 4848–4861 (2020)
Lorato, I., et al.: Towards continuous camera-based respiration monitoring in infants. Sensors 21(7), 2268 (2021)
Lund, C.H., et al.: Neonatal skin care: clinical outcomes of the AWHONN/NANN evidence-based clinical practice guideline. J. Obstet. Gynecol. Neonatal Nurs. 30(1), 41–51 (2001)
Lyra, S., et al.: A deep learning-based camera approach for vital sign monitoring using thermography images for ICU patients. Sensors 21(4) (2021)
Lyra, S., Paul, M.: Organic led panels for pulse rate measurement using photoplethysmography imaging. In: Proceedings of the International Student Scientific Conference Poster, pp. BI04 1–4 (2019)
Maier, A., Syben, C., Lasser, T., Riess, C.: A gentle introduction to deep learning in medical image processing. Z. Med. Phys. 29(2), 86–101 (2019)
Pereira, C.B., et al.: Estimation of respiratory rate from thermal videos of preterm infants. In: 2017 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 3818–3821. IEEE (2017)
Pereira, C.B., et al.: Noncontact monitoring of respiratory rate in newborn infants using thermal imaging. IEEE Trans. Biomed. Eng. 66(4), 1105–1114 (2018)
Reuter, S., Moser, C., Baack, M.: Respiratory distress in the newborn. Pediatr. Rev. 35(10), 417 (2014)
Reza, A.: Realization of the contrast limited adaptive histogram equalization (clahe) for real-time image enhancement. VLSI Sig. Process. 38, 35–44 (2004)
Shelov, S.P., Altmann, T.R. (eds.): Caring for Your Baby and Young Child, 5th edn. American Academy of Pediatrics, Itasca (2005)
Acknowledgments
We acknowledge the effort from the authors of the YOLOv4 object detection algorithm and Kopaczka et al. for providing the adult IR dataset. The authors gratefully acknowledge financial support provided by German Research Foundation [Deutsche Forschungsgemeinschaft, LE 817/32-1].
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Lyra, S., Groß-Weege, I., Leonhardt, S., Lüken, M. (2022). Real-Time Respiration Monitoring of Neonates from Thermography Images Using Deep Learning. In: Sclaroff, S., Distante, C., Leo, M., Farinella, G.M., Tombari, F. (eds) Image Analysis and Processing – ICIAP 2022. ICIAP 2022. Lecture Notes in Computer Science, vol 13231. Springer, Cham. https://doi.org/10.1007/978-3-031-06427-2_19
Download citation
DOI: https://doi.org/10.1007/978-3-031-06427-2_19
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-06426-5
Online ISBN: 978-3-031-06427-2
eBook Packages: Computer ScienceComputer Science (R0)