Skip to main content
SpringerLink
Log in

3D video semantic segmentation for wildfire smoke

  • Original Paper
  • Published:
Machine Vision and Applications Aims and scope Submit manuscript

Abstract

Wildfires are a serious threat to ecosystems and human life. Usually, smoke is generated before the flame, and due to the diffusing nature of the smoke, we can detect smoke from a distance, so wildfire smoke detection is especially important for early warning systems. In this paper, we propose a 3D convolution-based encoder–decoder network architecture for video semantic segmentation in wildfire smoke scenes. In the encoder stage, we use 3D residual blocks to extract the spatiotemporal features of smoke. The downsampling feature from the encoder is upsampled by the decoder three times in succession. Then, three smoke map prediction modules are, respectively, passed, the output smoke prediction map is supervised by the binary image label, and finally, the final prediction is obtained by feature map fusion. Our model can achieve end-to-end training without pretraining from scratch. In addition, a dataset including 90 smoke videos is tested and trained in this paper. The experimental results of the smoke video show that our model quickly and accurately segmented the smoke area and produced few false positives.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Tőreyin, B.U., Çetin, A. E.: Wildfire detection using LMS based active learning. In: IEEE International Conference on Acoustics (2009)

  2. Tőreyin, B.U., Cinbiş, R.G., Dedeoğlu, Y., Çetin, A.E.: Fire detection in infrared video using wavelet analysis. Opt. Eng. 46(6), 7204 (2007)

    Article  Google Scholar 

  3. Tőreyin, B. U., Dedeoğlu, Y., Çetin, A. E.: Contour based smoke detection in video using wavelets. In: 2006 14th European Signal Processing Conference, pp. 1–5 (2006)

  4. Besbes, O., Benazza-Benyahia, A.: A novel video-based smoke detection method based on color invariants. In: 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), March, pp. 1911–1915 (2016)

  5. Guillemant, P., Vicente, J.: Real-time identification of smoke images by clustering motions on a fractal curve with a temporal embedding method. Opt. Eng. 40(4), 554563 (2001)

    Article  Google Scholar 

  6. Gomes, P., Santana, P., Barata, J.: A vision-based approach to fire detection. Int. J. Adv. Rob. Syst. 11(9), 149 (2014). https://doi.org/10.5772/58821. [Online]

    Article  Google Scholar 

  7. Yin, Z., Wan, B., Yuan, F., Xia, X., Shi, J.: A deep normalization and convolutional neural network for image smoke detection. IEEE Access 5(99), 18429–18438 (2017)

    Article  Google Scholar 

  8. Filonenko, A., Kurnianggoro, L., Jo, K. H.: Comparative study of modern convolutional neural networks for smoke detection on image data. In: 2017 10th International Conference on Human System Interactions (HSI), July (2017), pp. 64–68

  9. Frizzi, S., Kaabi, R., Bouchouicha, M., Ginoux, J. M., Moreau, E., Fnaiech, F.: Convolutional neural network for video fire and smoke detection. In: IECON 2016 - 42nd Annual Conference of the IEEE Industrial Electronics Society, pp. 877–882 (2016)

  10. Tao, C., Zhang, J., Wang, P.: Smoke detection based on deep convolutional neural networks. In: 2016 International Conference on Industrial Informatics—Computing Technology, Intelligent Technology, Industrial Information Integration (ICIICII), pp. 150–153 (2016)

  11. Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 39(4), 640–651 (2014)

    Google Scholar 

  12. Chen, L., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L.: Deeplab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Trans. Pattern Anal. Mach. Intell. 40(4), 834–848 (2018)

    Article  Google Scholar 

  13. Chen, L., Papandreou, G., Schroff, F., Adam, H.: Rethinking atrous convolution for semantic image segmentation. In: CVPR (2017)

  14. Ronneberger, O., Fischer, P., Brox, T.: U-net: convolutional networks for biomedical image segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 234–241 (2015)

  15. ÇiÇek, O., Abdulkadir, A., Lienkamp, S.S., Brox, T., Ronneberger, O.: 3D u-net: learning dense volumetric segmentation from sparse annotation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 424–432 (2016)

  16. Yuan, C., Liu, Z., Zhang, Y.: Learning-based smoke detection for unmanned aerial vehicles applied to forest fire surveillance. J Intell Robotic Syst 93, 337–349 (2019)

    Article  Google Scholar 

  17. Jia, Y., Lin, G., Wang, J., Fang, J., Zhang, Y.: Early video smoke segmentation algorithm based on saliency detection and gaussian mixture model. Comput Eng 42, 206 (2016)

    Google Scholar 

  18. Krstinic, D., Stipanicev, D., Jakovevi, T.: Histogram-based smoke segmentation in forest fire detection system. Inform Technol Control 38, 01 (2009)

    Google Scholar 

  19. Zhang, D., Hu, A., Rao, Y., Zhao, J., Zhao, J.: Forest fire and smoke detection based on video image segmentation. In: Proceedings of SPIE—The International Society for Optical Engineering, pp. 67 882H–67 882H–7 (2007)

  20. Wang, X., Jiang, A., Wang, Y.: A segmentation method of smoke in forest-fire image based on fbm and region growing. In: Proceedings—4th International Workshop on Chaos-Fractals Theories and Applications. IWCFTA vol. 2011 pp. 390–393 (2011)

  21. Zhao, Y.: Candidate smoke region segmentation of fire video based on rough set theory. J Electr Comput Eng 2015, 280 415.1–280 415.8 (2015)

    Google Scholar 

  22. Xing, D., Zhongming, Y., Lin, W., Jinlan, L.: Smoke image segmentation based on color model. RISUS J. Innov. Sustain. 6(2), 130–138 (2015)

    Article  Google Scholar 

  23. Zhou, Z., Shi, Y., Gao, Z., Sun, L.: Wildfire smoke detection based on local extremal region segmentation and surveillance. Fire Saf. J. 85, 50–58 (2016)

    Article  Google Scholar 

  24. Li, X., Chen, Z., Wu, Q.M.J., Liu, C.: 3D parallel fully convolutional networks for real-time video wildfire smoke detection. IEEE Trans. Circuits Syst. Video. Technol. 30(1), 01 (2020)

    Article  Google Scholar 

  25. Shuiwang, J., Ming, Y., Kai, Y.: 3D convolutional neural networks for human action recognition. IEEE Trans. Pattern Anal. Mach. Intell. 35(1), 221–231 (2013)

    Article  Google Scholar 

  26. Du, T., Bourdev, L., Fergus, R., Torresani, L., Paluri, M.: Learning spatiotemporal features with 3D convolutional networks. In: IEEE International Conference on Computer Vision (2015)

  27. Du, T., Wang, H., Torresani, L., Ray, J., Lecun, Y., Paluri, M.: A closer look at spatiotemporal convolutions for action recognition. In: 2018 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2018)

  28. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778 (2016)

  29. Ham, S., Ko, B.C., Nam, J. Y.: Vision based forest smoke detection using analyzing of temporal patterns of smoke and their probability models. In: Image Processing: Machine Vision Applications IV (2011)

  30. Ko, B., Kwak, J.Y., Nam, J.Y.: Wildfire smoke detection using temporospatial features and random forest classifiers. Opt. Eng. 51(1), 7208 (2012)

    Article  Google Scholar 

  31. Park, J.O., Ko, B.C., Nam, J.Y., Kwak, S.: Wildfire smoke detection using spatiotemporal bag-of-features of smoke. In: Applications of Computer Vision (2013)

Download references

Acknowledgements

This work was supported in part by the National Key R&D Program of China (2019YFB1311001), in part by the National Natural Science Foundation of China (61876099), in part by the Scientific and Technological Development Project of Shandong Province (2019GSF111002), in part by the Shenzhen Science and Technology Research and Development Funds (JCYJ20180305164401921), in part by the Foundation of Ministry of Education Key Laboratory of System Control and Information Processing (Scip201801), and in part by the Foundation of Key Laboratory of Intelligent Computing & Information Processing of Ministry of Education (2018ICIP03).

Author information

Authors and Affiliations

  1. School of Control Science and Engineering, Shandong University, Jinan, 250061, China

    Guodong Zhu, Zhenxue Chen, Chengyun Liu & Xuewen Rong

  2. Research Institute of Shandong University, Shandong University, Shenzhen, 518057, China

    Zhenxue Chen

  3. School of Aeronautics, Shandong Jiaotong University, Jinan, 250023, China

    Weikai He

Authors
  1. Guodong Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhenxue Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chengyun Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xuewen Rong
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Weikai He
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zhenxue Chen.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Guodong Zhu and Zhenxue Chen contributed equally to this work and should be considered as the co-first authors.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhu, G., Chen, Z., Liu, C. et al. 3D video semantic segmentation for wildfire smoke. Machine Vision and Applications 31, 50 (2020). https://doi.org/10.1007/s00138-020-01099-w

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s00138-020-01099-w

Keywords

Search

Navigation