Skip to main content

Advertisement

SpringerLink
Log in

Real-time human detection in thermal infrared imaging at night using enhanced Tiny-yolov3 network

  • Original Research Paper
  • Published:
Journal of Real-Time Image Processing Aims and scope Submit manuscript

Abstract

Human detection is a technology that detects human shapes in the image and ignores everything else. However, modern person detectors have some inefficiencies in detecting pedestrians during video surveillance at night, and the accuracy rate is still insufficient. Therefore, this paper aims to increase the accuracy rate for automatic human detection at night from thermal infrared (TIR) images and real-time video sequences. For this purpose, a new architecture is proposed to enhance the backbone of the Tiny-yolov3 network. The enhanced network used the YOLOv3 algorithm’s tasks with the K-means clustering method to extract more complex features of a person. This network was pre-trained on the MS. COCO dataset to obtain the initial weights. Through the comparison with other related methods showed that the experimental results have achieved the significantly improved performance of human detection from thermal imaging in terms of accuracy, speed, and detection time. The method has achieved a high accuracy rate (90%) compared with the TF-YOLOv3 (88%) trained on the DHU Night Dataset. Although the method has achieved an accuracy rate equal to the YOLOv3-Human (90%), the detection time (4.88 ms) is less, Furthermore, the method has a higher accuracy rate (49.8%) than the YOLO (29.36%) and TF-YOLOv3 (29.8%) with lower detection time (8 ms) on the FLIR Dataset. In addition, the model has achieved a good TP detection for multiple small size of person. By improving the performance of human detection in thermal imaging at night, the method will be able to detect intruders in the night surveillance system.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  1. Kim, K., Choi, H.-I., Oh, K.: Object detection using ensemble of linear classifiers with fuzzy adaptive boosting. Eurasip J. Image Video Process. 40(2017), 2–10 (2017)

    Google Scholar 

  2. Ansari, M., Singh, D.K.: Human detection techniques for real-time surveillance: A comprehensive survey. Multimed. Tools Appl. 80(6), 8759–8808 (2021)

    Article  Google Scholar 

  3. Zhang, W., Huang, L., Wei, Z., Nie, J.: Appearance feature enhancement for person re-identification. Expert Sys. Appl. 163, 113771 (2021)

    Article  Google Scholar 

  4. Tian, X., Daigle, H.: Machine-learning-based object detection in images for reservoir characterization: A case study of fracture detection in shales. Lead. Edge 37(6), 435–442 (2018)

    Article  Google Scholar 

  5. Zou, Z., Shi, Z., Guo, Y., Ye, J.: Object detection in 20 years: A survey. arXiv preprint arXiv, 2019, https://arxiv.org/abs/1905.05055 (2019).

  6. Park, J., Chen, J., Cho, Y.K., Kang, D.Y., Son, B.J.: CNN-based person detection using infrared images for night-time intrusion warning systems. Sensors 20(1), 34 (2020)

    Article  Google Scholar 

  7. Noll, P., Henkel, M.: History and evolution of modeling in biotechnology: modeling & simulation, application, and hardware performance. Comput. Struct. Biotechnol. J. 18, 3309–3323 (2020)

    Article  Google Scholar 

  8. Zhao, Z.-Q., Zheng, P., Xu, S.-T., Wu, X.: Object detection with deep learning: A review. IEEE Trans. Neural Netw. Learn. Syst. 30(11), 3212–3232 (2019)

    Article  Google Scholar 

  9. Jiao, L., Zhang, F., Liu, F., Yang, S., Li, L., Feng, Z., et al.: A survey of deep learning-based object detection. IEEE Access 7, 128837–128868 (2019)

    Article  Google Scholar 

  10. Liwag, R. J. H., Cepria, K. J. T., Rapio, A., Cabatuan, K., Calilung, E.: Single Shot Multi-Box Detector with Multi Task Convolutional Network for Carabao Mango Detection and Classification using Tensorflow. In: Proceedings of the 5th DLSU Innovation and Technology, pp. 1–8. IEEE (2017)

  11. Han, F., Yao, J., Zhu, H., Wang, C.: Underwater image processing and object detection based on deep CNN method. J. Sens. 2020(9), 1–20 (2020)

    Google Scholar 

  12. Wu, M., Yue, H., Wang, J., Huang, Y., Liu, M., Jiang, Y., et al.: Object detection based on RGC Mask R-CNN. IET Image Proc. 14(8), 1502–1508 (2020)

    Article  Google Scholar 

  13. Xiao, Y., Wang, X., Zhang, P., Meng, F., Shao, F.: Object detection based on Faster R-CNN algorithm with skip pooling and fusion of contextual information. Sensors 20(19), 5490 (2020)

    Article  Google Scholar 

  14. Ahmad, T., Ma, Y., Yahya, M., Ahmad, B., Nazir, S.: Object detection through modified YOLO neural network. Sci. Progr. 2020(6), 1–10 (2020)

    Google Scholar 

  15. Nowosielski, A., Małecki, K., Forczmański, P., Smoliński, A., Krzywicki, K.: Embedded night-vision system for pedestrian detection. IEEE Sens. J. 20(16), 9293–9304 (2020)

    Google Scholar 

  16. Wang, D., Zhang, S.: Unsupervised person re-identification via multi-label classification. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10981–10990. IEEE (2020).

  17. Huang, R., Pedoeem, J., Chen, C.: YOLO-LITE: a real-time object detection algorithm optimized for non-GPU computers. In: 2018 IEEE International Conference on Big Data (Big Data), pp. 2503–2510. IEEE (2018)

  18. Ćorović, A., Ilić, V., Durić, S., Marijan, M., Pavković, B.: The real-time detection of traffic participants using YOLO algorithm. In: 2018 26th Telecommunications Forum (TELFOR), pp. 1–4. IEEE (2018)

  19. Ferreira, A.C., Silva, L.R., Renna, F., Brandl, H.B., Renoult, J.P., Farine, D.R., et al.: Deep learning-based methods for individual recognition in small birds. Methods Ecol. Evol. 11(9), 1072–1085 (2020)

    Article  Google Scholar 

  20. Tran, T.-H., Nguyen, D. T., Nguyen, T. P.: Human posture classification from multiple viewpoints and application for fall detection. In: 2020 IEEE Eighth International Conference on Communications and Electronics (ICCE), pp. 262–267. IEEE (2021)

  21. Anderson, V.R., Kulig, T.C.: Human trafficking. Encycl. Res. Methods Criminol. Crim. Justice 2, 710–715 (2021)

    Article  Google Scholar 

  22. Ivašić-Kos, M., Krišto, M., Pobar, M.: Human detection in thermal imaging using YOLO. In: Proceedings of the 2019 5th International Conference on Computer and Technology Applications, pp. 20–24 (2019)

  23. Juang, J.-G., Tu, G.-T., Liao, Y.-H., Huang, T.-H., Chang, S.-I.: Drone patrol using thermal imaging for object detection. In: Infrared Sensors, Devices, and Applications X, vol. 11503, pp. 115030U. International Society for Optics and Photonics (2020)

  24. Chen, Y., Shin, H.: Pedestrian detection at night in infrared images using an attention-guided encoder-decoder convolutional neural network. Appl. Sci. 10(3), 809 (2020)

    Article  Google Scholar 

  25. Manssor, S.A., Sun, S., Elhassan, M.A.: Real-time human recognition at night via integrated face and gait recognition technologies. Sensors 21(13), 4323 (2021)

    Article  Google Scholar 

  26. Hayner, C. R., Liu, E., Gupta, N., Zhou, T., Mayhew, P., Vagners, J. Real-time human detection with integration of visual and thermal data from high altitude sUAS. In: AIAA Scitech 2021 Forum, pp. 0397. 2021

  27. Wang, X., Wang, Z., Liu, W., Xu, X., Chen, J., Lin, C.-W.: Consistency-constancy bi-knowledge learning for pedestrian detection in night surveillance. In: Proceedings of the 29th ACM International Conference on Multimedia, pp. 4463–4471. (2021)

  28. Teutsch, M., Sappa, A.D., Hammoud, R.I.: Computer vision in the infrared spectrum: challenges and approaches. Synth. Lect. Comput. Vis. 10(2), 1–138 (2021)

    Article  Google Scholar 

  29. Krišto, M., Ivašić-Kos, M.: Thermal imaging dataset for person detection. In: 2019 42nd International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO), pp. 1126–1131. IEEE (2019)

  30. Munir, F., Azam, S., Rafique, M. A., Sheri, A. M., Jeon, M., Pedrycz, W.: Exploring Thermal Images for Object Detection in Underexposure Regions for Autonomous Driving. arXiv preprint arXiv, 2020, https://arxiv.org/abs/2006.00821 (2020)

  31. Khandhediya, Y., Sav, K., Gajjar, V.: Human detection for night surveillance using adaptive background subtracted image. arXiv preprint arXiv, 2017, https://arxiv.org/abs/1709.09389 (2017)

  32. Yi, Z., Yongliang, S., Jun, Z.: An improved tiny-yolov3 pedestrian detection algorithm. Optik 183, 17–23 (2019)

    Article  Google Scholar 

  33. He, W., Huang, Z., Wei, Z., Li, C., Guo, B.: TF-YOLO: An improved incremental network for real-time object detection. Appl. Sci.-Basel 9(16), 3225 (2019)

    Article  Google Scholar 

  34. Liu, P., Yang, H., Fu, J.: Marine biometric recognition algorithm based on YOLOv3-GAN network. In: International Conference on Multimedia Modeling, pp. 581–592. Springer (2020)

  35. Gong, X., Ma, L., Ouyang, H.: An improved method of Tiny YOLOV3. In: IOP Conference Series: Earth and Environmental Science, vol. 440, pp. 052025, vol. 5. IOP Publishing (2020)

  36. Redmon, J., Farhadi, A.: Yolov3: An incremental improvement. arXiv preprint arXiv, 2018, https://arxiv.org/abs/1804.02767 (2018)

  37. Dhamecha, M.: Improve K-mean clustering algorithm in large-scale data for accuracy improvement. In: Machine Intelligence and Soft Computing, pp. 61–69. Springer (2021)

  38. Jiang, D., Qi, G., Hu, G., Mazur, N., Zhu, Z., Wang, D.: A residual neural network based method for the classification of tobacco cultivation regions using near-infrared spectroscopy sensors. Infrared Phys. Technol. 111, 103494 (2020)

    Article  Google Scholar 

  39. Valova, I., Harris, C., Gueorguieva, N., Mai, T.: In-between layers modular residual neural network for the classification of images. Proc. Comput. Sci. 185, 223–230 (2021)

    Article  Google Scholar 

  40. Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You only look once: Unified, real-time object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 779–788. (2016)

  41. Redmon, J., Farhadi, A.: YOLO9000: better, faster, stronger. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7263–7271 (2017)

  42. Huang, J., Rathod, V., Sun, C., Zhu, M., Korattikara, A., Fathi, A., et al.: Speed/accuracy trade-offs for modern convolutional object detectors. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7310–7311 (2017)

  43. Xiao, D., Shan, F., Li, Z., Le, B.T., Liu, X., Li, X.: A target detection model based on improved tiny-yolov3 under the environment of mining truck. IEEE Access 7, 123757–123764 (2019)

    Article  Google Scholar 

  44. Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., et al. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740–755. Springer (2014)

  45. Huang, Z., Wang, Z., Zhang, J., Li, Q., Shi, Y.: Image enhancement with the preservation of brightness and structures by employing contrast limited dynamic quadri-histogram equalization. Optik 226, 165877 (2021)

    Article  Google Scholar 

  46. Hua, J., Gong, X.: A normalized convolutional neural network for guided sparse depth upsampling. In: IJCAI, pp. 2283–2290 (2018)

  47. Badrinarayanan, V., Kendall, A., Cipolla, R.: Segnet: A deep convolutional encoder-decoder architecture for image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 39(12), 2481–2495 (2017)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Information Science and Technology, Donghua University, Shanghai, 201620, China

    Samah A. F. Manssor & Shaoyuan Sun

  2. Faculty of Engineering and Technology, University of Gezira, 22211, Wad Madni, Sudan

    Samah A. F. Manssor

  3. School of Computer Science and Technology, Northwestern Polytechnical University, Xi’an, 710072, China

    Mohammed Abdalmajed

  4. Network Department, National Information Center, 11111, Khartoum, Sudan

    Shima Ali

Authors
  1. Samah A. F. Manssor
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shaoyuan Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mohammed Abdalmajed
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Shima Ali
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Samah A. F. Manssor.

Ethics declarations

Conflict of interest

This paper does not contain any unethical studies on humans or animals. All authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Manssor, S.A.F., Sun, S., Abdalmajed, M. et al. Real-time human detection in thermal infrared imaging at night using enhanced Tiny-yolov3 network. J Real-Time Image Proc 19, 261–274 (2022). https://doi.org/10.1007/s11554-021-01182-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11554-021-01182-z

Keywords

Search

Navigation