Abstract
Nowadays, low-resolution thermal cameras are gaining relevance in smart environments due to keeping user privacy by recording images and videos in domestic environments. Many neural networks obtain outstanding results from visible spectrum devices for human activity and event detection, such as fall detection, object detection or pose estimation. However, these state-of-the-art neural networks are trained in datasets that do not contain thermal images, so their performance on them is not good. The main objective of this work is human body recognition and segmentation from thermal cameras. For this purpose, we propose ThermalYOLO, a neural network based on the YOLO neural network and fine-tuned with thermal images. For the generation and auto-labelling of the thermal dataset, an IoT device with two cameras, a visible camera and a thermal camera, is used. Therefore, the user does not have to manually annotate the dataset. As a result, ThermalYOLO outperforms YOLO in thermal images from two different smart environments.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Adarsh, P., Rathi, P., Kumar, M.: Yolo v3-tiny: object detection and recognition using one stage improved model. In: 2020 6th International Conference on Advanced Computing and Communication Systems (ICACCS), pp. 687–694. IEEE (2020)
Agarwal, A., Jawahar, C.V., Narayanan, P.J.: A survey of planar homography estimation techniques. Tech. rep. (2005)
Ahmad, T., Ma, Y., Yahya, M., Ahmad, B., Nazir, S.: Object detection through modified YOLO neural network. Sci. Program. 2020 (2020)
Al-Sarawi, S., Anbar, M., Alieyan, K., Alzubaidi, M.S.: Internet of Things (IoT) communication protocols: review. In: 2017 8th International Conference on Information Technology (ICIT), pp. 685–690 (2017)
Beddiar, D.R., Nini, B., Sabokrou, M., Hadid, A.: Vision-based human activity recognition: a survey. Multimed. Tools Appl. 79(41), 30509–30555 (2020). https://doi.org/10.1007/s11042-020-09004-3
Chen, I.C., Wang, C.J., Wen, C.K., Tzou, S.J.: Multi-person pose estimation using thermal images. IEEE Access 8, 174964–174971 (2020)
Dang, L.M., Min, K., Wang, H., Piran, M.J., Lee, C.H., Moon, H.: Sensor-based and vision-based human activity recognition: a comprehensive survey. Pattern Recognit. 108, 107561 (2020)
Espinilla, M., Martínez, L., Medina, J., Nugent, C.: The experience of developing the UJAm I Smart lab. IEEE Access 6, 34631–34642 (2018)
Fischler, M.A., Bolles, R.C.: Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM 24(6), 381–395 (1981)
Girshick, R., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp. 580–587 (2014)
Haider, A., Shaukat, F., Mir, J.: Human detection in aerial thermal imaging using a fully convolutional regression network. Infrared Phys. Technol. 116, 103796 (2021)
Hammad, I., El-Sankary, K.: Using machine learning for person identification through physical activities. In: 2020 IEEE International Symposium on Circuits and Systems (ISCAS), pp. 1–5. IEEE (2020)
Ivašić-Kos, M., Krišto, M., Pobar, M.: Human detection in thermal imaging using YOLO. In: Proceedings of the 2019 5th International Conference on Computer and Technology Applications, pp. 20–24 (2019)
Jung, I.Y.: A review of privacy-preserving human and human activity recognition. Int. J. Smart Sens. Intell. Syst. 13(1), 1–13 (2020)
Kurmi, I., Schedl, D.C., Bimber, O.: Combined person classification with airborne optical sectioning. Sci. Rep. 12(1), 1–11 (2022)
Lin, T.-Y., et al.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10602-1_48
Lupión, M., Medina-Quero, J., Sanjuan, J.F., Ortigosa, P.M.: Dolars, a distributed on-line activity recognition system by means of heterogeneous sensors in real-life deployments—a case study in the smart lab of the University of Almería. Sensors 21(2), 405 (2021)
Malis, E., Vargas, M.: Deeper understanding of the homography decomposition for vision-based control. Research Report RR-6303, INRIA (2007). https://hal.inria.fr/inria-00174036
Otanasap, N.: Pre-impact fall detection system using real time dynamic threshold and human body bounding box by multiple kinects. SAU J. Sci. Technol. 5(1), 49–61 (2019)
Ramamurthy, S.R., Roy, N.: Recent trends in machine learning for human activity recognition—a survey. Wiley Interdiscip. Rev. Data Min. Knowl. Discov. 8(4), e1254 (2018)
Saadatnejad, S., Ju, Y.Z., Alahi, A.: Pedestrian 3D bounding box prediction. arXiv preprint arXiv:2206.14195 (2022)
Sixsmith, A., Johnson, N.: A smart sensor to detect the falls of the elderly. IEEE Pervasive Comput. 3(2), 42–47 (2004)
Tsai, P.F., Liao, C.H., Yuan, S.M.: Using deep learning with thermal imaging for human detection in heavy smoke scenarios. Sensors 22(14), 5351 (2022)
Wang, J., Chen, Y., Hao, S., Peng, X., Hu, L.: Deep learning for sensor-based activity recognition: a survey. Pattern Recognit. Lett. 119, 3–11 (2019)
Zhang, Y., Sidibé, D., Morel, O., Mériaudeau, F.: Deep multimodal fusion for semantic image segmentation: a survey. Image Vis. Comput. 105, 104042 (2021)
Acknowledgements
This research was funded by the R+D+i projects RTI2018-095993-B-I00 and PID2021-123278OB-I0, financed by Spanish Ministry of Science and Innovation - Spanish State Research Agency and ERDF “A way to make Europe”; by the Junta de Andalucía, Spain grant number P18-RT-1193; by the University of Almería grant number UAL18-TIC-A020-B and by the Department of Informatics at University of Almería. Marcos Lupión Lorente is a fellow of the Spanish ‘Formación del Profesorado Universitario’ program (FPU19/02756). Moreover, this research has received funding by EU Horizon 2020 Pharaon Project ‘Pilots for Healthy and Active Aging’, Grant agreement no. 857188. Furthermore, this research has received funding under the REMIND project Marie Sklodowska-Curie EU Framework for Research and Innovation Horizon 2020, under grant agreement no. 734355. Lastly, this contribution has been supported by the Spanish Institute of Health ISCIII by means of the project DTS21-00047.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Lupión, M., Polo-Rodríguez, A., Ortigosa, P.M., Medina-Quero, J. (2023). ThermalYOLO: A Person Detection Neural Network in Thermal Images for Smart Environments. In: Bravo, J., Ochoa, S., Favela, J. (eds) Proceedings of the International Conference on Ubiquitous Computing & Ambient Intelligence (UCAmI 2022). UCAmI 2022. Lecture Notes in Networks and Systems, vol 594. Springer, Cham. https://doi.org/10.1007/978-3-031-21333-5_76
Download citation
DOI: https://doi.org/10.1007/978-3-031-21333-5_76
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-21332-8
Online ISBN: 978-3-031-21333-5
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)