Skip to main content
SpringerLink
Log in

Real Time Victim Detection in Smoky Environments with Mobile Robot and Multi-sensor Unit Using Deep Learning

  • Conference paper
  • First Online:
Robot Intelligence Technology and Applications 7 (RiTA 2022)

Part of the book series: Lecture Notes in Networks and Systems ((LNNS,volume 642))

  • 764 Accesses

Abstract

Victim detection in smoky indoor environments during search and rescue missions is still challenging and a critical situation. This situation is because firefighters are on the one hand exposed to unstable building structures and on the other hand their cognitive fatigue, due to long search missions, reduces the efficient victim detection in these hazardous environments. In this paper, an approach to detect a victim in real time with an optical and low resolution thermal camera assisting firefighters in their missions is presented. Thereby, the multi-sensor unit is mounted on a remote-controlled mobile robot with a trained victim detector using deep learning and display the detection in real time to an operator outside the scene. Experiments show that this approach enables an efficient detection in smoky indoor environments. The victim detection model achieves an average detection rate above 75% in real time with a low resolution thermal camera in dense smoke.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 189.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 249.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    https://github.com/tensorflow/

  2. 2.

    https://github.com/tensorflow/tensorflow/

  3. 3.

    https://keras.io/

  4. 4.

    https://cocodataset.org/

  5. 5.

    https://mega.nz/folder/AO0AWCJB#Dgl_1pdq1B_Icz7lSXJzrw.

  6. 6.

    https://github.com/tzutalin/labelImg/

  7. 7.

    https://pypi.org/project/split-folders/

References

  1. CTIF.: World Fire Statistics (2019). https://www.ctif.org/

  2. Casper, J., Murphy, R.R.: Human-robot interactions during the robot-assisted urban search and rescue response at the world trade center. IEEE Trans. Syst. Man Cybern. 33, 367–385 (2003)

    Article  Google Scholar 

  3. Murphy, R.R., Burke, J.L.: Up from the rubble: lessons learned about HRI from search and rescue. Proc. Hum. Factors Ergon. Soc. Ann. Meet. 49, 437–441 (2005)

    Article  Google Scholar 

  4. Dadwhal, Y.S., Kumar, S., Sardana, H.K.: Data-driven skin detection in cluttered search and rescue environments. IEEE Sens. J. 20, 3697–3708 (2020)

    Article  Google Scholar 

  5. Coombes, M., Eaton, W., Chen, W.-H.: Machine vision for UAS ground operations. J. Intell. Rob. Syst. 88(2–4), 527–546 (2017). https://doi.org/10.1007/s10846-017-0542-5

    Article  Google Scholar 

  6. Liu, P., Yu, H., Cang, S., Vladareanu, L.: Robot-assisted smart firefighting and interdisciplinary perspectives. In: 22nd International Conference on Automation and Computing, pp. 395–401 (2016)

    Google Scholar 

  7. Wilk-Jakubowski, G., Harabin, R., Ivanov, S.: Robotics in crisis management: a review. Technol. Soc. 68, 101935 (2022)

    Article  Google Scholar 

  8. Habib, M.K., Baudoin, Y.: Robot-assisted risky intervention, search, rescue and environmental surveillance. In: International Journal of Advanced Robotic Systems: SAGE Publications Sage UK: London, England, p. 10 (2010)

    Google Scholar 

  9. Castillo, C., Chang, C.: A method to detect victims in search and rescue operations using template matching. In: IEEE International Safety, Security and Rescue Robotics, Workshop, pp. 201–206 (2005)

    Google Scholar 

  10. De Cubber, G., Marton, G.: Human victim detection. In: 3rd International Workshop on Robotics for risky interventions and Environmental Surveillance-Maintenance (2009)

    Google Scholar 

  11. Jones, M.J., Viola, P.: Robust real-time object detection. In: Workshop on Statistical and Computational Theories of Vision, p. 56 (2001)

    Google Scholar 

  12. Viola, P., Jones, M.J.: Robust real-time face detection. Int. J. Comput. Vis. 57, 137–154 (2004)

    Article  Google Scholar 

  13. Soni, B., Sowmya, A.: Classifier ensemble with incremental learning for disaster victim detection. In: 2012 IEEE International Conference on Robotics and Biomimetics, pp. 446–451 (2012)

    Google Scholar 

  14. Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 886–893 (2005)

    Google Scholar 

  15. Oliveira, G.L., Valada, A., Bollen, C., Burgard, W., Brox, T.: Deep learning for human part discovery in images. In: 2016 IEEE International Conference on Robotics and Automation, pp. 1634–1641 (2016)

    Google Scholar 

  16. Ivašić-Kos, M., Krišto, M., Pobar, M.: Human detection in thermal imaging using YOLO. In: 5th International Conference on Computer and Technology Applications, pp. 20–24 (2019)

    Google Scholar 

  17. Jaradat, F.B., Valles, D.: A victims detection approach for burning building sites using convolutional neural networks. In: 10th Annual Computing and Communication Workshop and Conference, pp. 280–286 (2020)

    Google Scholar 

  18. Cruz Ulloa, C., Prieto Sánchez, G., Barrientos, A., Del Cerro, J.: Autonomous thermal vision robotic system for victims recognition in search and rescue missions. Sensors 21(21), 7346 (2021)

    Article  Google Scholar 

  19. Bañuls, A., Mandow, A., Vázquez-Martín, R., Morales, J., García-Cerezo, A.: Object detection from thermal infrared and visible light cameras in search and rescue scenes. In: 2020 IEEE International Symposium on Safety, Security and Rescue Robotics, pp. 380–386 (2020)

    Google Scholar 

  20. Hoshino, W., Seo, J., Yamazaki, Y.: A study for detecting disaster victims using multi-copter drone with a thermographic camera and image object recognition by SSD. In: 2021 IEEE/ASME International Conference on Advanced Intelligent Mechatronics, pp. 162–167 (2021)

    Google Scholar 

  21. Perdana, M.I., Risnumawan, A., Sulistijono, I.A.: Automatic aerial victim detection on low-cost thermal camera using convolutional neural network. In: 2020 International Symposium on Community-centric Systems, pp. 1–5 (2020)

    Google Scholar 

  22. Petřı́ček, T., Šalanský, V., Zimmermann, K., Svoboda, T.: Simultaneous exploration and segmentation for search and rescue. J. Field Robot. 36, 696–709 (2018)

    Google Scholar 

  23. Tsai, P.F., Liao, C.H., Yuan, S.M.: Using deep learning with thermal imaging for human detection in heavy smoke scenarios. Sensors 22(14), 5351 (2022)

    Article  Google Scholar 

  24. Liu, W., et al.: SSD: single shot multibox detector. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9905, pp. 21–37. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46448-0_2

    Chapter  Google Scholar 

  25. Sandler, M., Howard, A.G., Zhu, M., Zhmoginov, A., Chen, L.C.: Inverted residuals and linear bottlenecks: mobile networks for classification, detection and segmentation. CoRR (2018)

    Google Scholar 

  26. Ganesh, V., Kolluri, J., Maada, A.R., Ali, M.H., Thota, R., Nyalakonda, S.: Real-time video processing for ship detection using transfer learning. In: Chen, J.I.Z., Tavares, J.M.R.S., Shi, F. (eds.) Third International Conference on Image Processing and Capsule Networks. ICIPCN 2022. LNNS, vol. 514, pp. 685–703. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-12413-6_54

  27. Krzanowski, W.J., Hand, D.J.: ROC curves for continuous data (2009)

    Google Scholar 

  28. Refaeilzadeh, P., Tang, L., Liu, H.: Cross-Validation. In: Liu, L., Özsu, M.T. (eds.) Encyclopedia of Database Systems, pp. 532–538. Springer, Boston, MA (2009). https://doi.org/10.1007/978-0-387-39940-9_565

  29. Zhang, Z., Li, J.: Random climate networks and entropy, pp. 127–171 (2020)

    Google Scholar 

  30. Fritsche, P., Zeise, B., Hemme, P., Wagner, B.: Fusion of radar, LiDAR and thermal information for hazard detection in low visibility environments. In: 2017 IEEE International Symposium on Safety, Security and Rescue Robotics, pp. 96–101 (2017)

    Google Scholar 

  31. Snellen, H.: Probebuchstaben zur Bestimmung der Sehschärfe (1862)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Munster Technological University, Cork, Ireland

    Sebastian Gelfert

Authors
  1. Sebastian Gelfert
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sebastian Gelfert .

Editor information

Editors and Affiliations

  1. School of Information and Communication Technology, Griffith University, Southport, Australia

    Jun Jo

  2. Department of Aerospace Engineering, KAIST, Daejeon, Korea (Republic of)

    Han-Lim Choi

  3. School of Information and Communication Technology, Griffith University, Southport, Australia

    Marde Helbig

  4. Department of Mechanical Engineering, Ulsan National Institute of Science and Technology (UNIST), Ulsan, Korea (Republic of)

    Hyondong Oh

  5. Department of Mechanical Engineering, KAIST, Daejeon, Korea (Republic of)

    Jemin Hwangbo

  6. Department of Mechanical Engineering, KAIST, Daejeon, Korea (Republic of)

    Chang-Hun Lee

  7. School of Information and Communication Technology, Griffith University, Southport, Australia

    Bela Stantic

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Gelfert, S. (2023). Real Time Victim Detection in Smoky Environments with Mobile Robot and Multi-sensor Unit Using Deep Learning. In: Jo, J., et al. Robot Intelligence Technology and Applications 7. RiTA 2022. Lecture Notes in Networks and Systems, vol 642. Springer, Cham. https://doi.org/10.1007/978-3-031-26889-2_32

Download citation

Publish with us

Policies and ethics

Search

Navigation