Skip to main content
SpringerLink
Log in

Edge-reinforced attention network for smoke semantic segmentation

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

This paper proposes a smoke semantic segmentation framework EANet based on boundary enhancement and attention mechanism. It integrates semantic segmentation and semantic boundary detection tasks into a framework, and distinguishes the features on both sides of the boundary with the help of the supervision of the semantic boundary, so as to guide the semantic segmentation task to determine whether the features on both sides of the boundary belong to the same object. At the same time, three attention mechanisms are proposed, which are used to capture the long-range context-dependent information of the object, strengthen the boundary semantic information of the segmentation feature, and enhance the attention to the key features of the channel domain. Finally, an adaptive fusion layer is used to fuse the prediction results of the two sub-networks to further improve the details of the segmentation results and obtain sharper object boundaries. In addition, in order to solve the problem of sample imbalance in the semantic boundary detection task, we designed a boundary loss function EL. By improving the standard binary cross-entropy, the network can focus more on difficult-to-classify samples and improve the network’s ability to deal with sample imbalance problems. A large number of experimental results show that our method is better than the state-of-art algorithms, and the proposed loss function can also help the algorithm to obtain more accurate and clear object boundaries.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

Data availability

The data that support the findings of this study are available on request from the corresponding author LZ. The data are not publicly available because they contain information that could compromise research participant privacy/consent.

References

  1. Chen L, Zhang H, Xiao J, et al. (2017) SCA-CNN: spatial and channel-wise attention in convolutional networks for image captioning. Computer Vision and Pattern Recognition, 6298–6306

  2. Chen L, Papandreou G, Kokkinos I, et al. (2015) Semantic image segmentation with deep convolutional nets and fully connected CRFs. International Conference on Learning Representations

    Google Scholar 

  3. Chen B, Gong C, Yang J (2019) Importance-aware semantic segmentation for autonomous vehicles. IEEE Trans Intell Trans Syst 20(1):137–148

    Article  Google Scholar 

  4. Chen J, Liu Z, Jin D, Wang Y, Yang F, Bai X (2022) Light transport induced domain adaptation for semantic segmentation in thermal infrared urban scenes. IEEE Trans Intell Transp Syst 23:23194–23211. https://doi.org/10.1109/TITS.2022.3194931

    Article  Google Scholar 

  5. Dimitropoulos K, Barmpoutis P, Grammalidis N et al (2017) Higher order linear dynamical systems for smoke detection in video surveillance applications. IEEE Trans Circ Syst Video Technol 27(5):1143–1154

    Article  Google Scholar 

  6. Filonenko A, Hernández DC, Jo K (2018) Fast smoke detection for video surveillance using CUDA. IEEE Trans Industr Inform 14(2):725–733

    Article  Google Scholar 

  7. Fu J, Liu J, Tian H et al (2019) Dual attention network for scene segmentation. IEEE Conference on Computer Vision and. Pattern Recogn:3146–3154

  8. Gu J, Hu H, Wang L et al (2018) Learning region features for object detection. European Conference on Computer Vision:392–406

  9. Hu J, Shen L, Gan S (2018) Squeeze-and-excitation networks[C]. IEEE conference on computer vision and Pattern Recogn, 7132–7141

  10. Hu J, Shen L, Albanie S, et al. (2018) Gather-Excite: exploiting feature context in convolutional neural networks. neural information processing systems, 9401–9411

  11. Hu H, Gu J, Zhang Z et al (2018) Relation networks for object detection[C] IEEE conference on computer vision and. Pattern Recogn:3588–3597

  12. Hu X, Yang K, Fei L, et al. (2019) ACNet: attention based network to exploit complementary features for RGBD semantic segmentation, ICIP

  13. Huang G, Liu Z, Der Maaten LV et al (2017) Densely connected convolutional networks. IEEE Conference on Computer Vision and Pattern Recognition:2261–2269

  14. Huang Z, Wang X, Huang L et al (2018) CCNet: Criss-cross attention for semantic segmentation. International Conference on Computer Vision:603–612

  15. Islam MA, Naha S, Rochan M et al (2017) Label refinement network for coarse-to-fine semantic segmentation. arXiv: computer vision and. Pattern Recognition

    Google Scholar 

  16. Kaabi R, Sayadi M, Bouchouicha M, et al. (2018) Early smoke detection of forest wildfire video using deep belief network. International Conference on Advanced Technologies for Signal and Image Processing, 1–6

  17. Li X, Chen Z, Wu QM et al (2020) 3D parallel fully convolutional networks for real-time video wildfire smoke detection. IEEE Trans Circ Syst Video Technol 30(1):89–103

    Article  Google Scholar 

  18. Lin TY, Goyal P, Girshick R et al (2017) Focal loss for dense object detection[J]. IEEE Trans Pattern Anal Machine Intell 99:2999–3007

    Google Scholar 

  19. Lin G, Milan A, Shen C et al (2017) RefineNet: multi-path refinement networks for high-resolution semantic segmentation. IEEE Conference on Computer Vision and Pattern Recognition:5168–5177

  20. Lin Z, Liu HH, Wotton M et al (2019) Kalman filter-based large-scale wildfire monitoring with a system of UAVs. IEEE Trans Ind Electron 66(1):606–615

    Article  Google Scholar 

  21. Liu H., Zhang J., Yang K. et al. (2022) CMX: Cross-modal fusion for RGB-X semantic segmentation with transformers. arXiv:2203.04838

  22. Luo Y, Zhao L, Liu P, Huang D (2018) Fire smoke detection algorithm based on motion characteristic and convolutional neural networks. Multimed Tools Appl 77(12):15075–15092

    Article  Google Scholar 

  23. Mahmoud MAI, Ren H (2019) Forest fire detection and identification using image processing and SVM. J Inform Process Syst 15(1):159–168

    Google Scholar 

  24. Newell A, Yang K, Deng J et al (2016) Stacked hourglass networks for human pose estimation. European Conference on Computer Vision:483–499

  25. Peng C, Zhang X, Yu G et al (2017) Large kernel matters — improve semantic segmentation by global convolutional network. IEEE Conference on Computer Vision and Pattern Recognition:1743–1751

  26. Sagar AG, Nath S, Nagrath P (2018) Smoke detection in digital frames. Int Res J Eng Technol 5(4):3843–3846

    Google Scholar 

  27. Strudel R, Garcia R, Laptev I et al (2021) Segmenter: transformer for semantic segmentation. International Conference on Computer Vision:7262–7272

  28. Takikawa T, Acuna D, Jampani V, et al. (2019) Gated-SCNN: Gated Shape CNNs for Semantic Segmantation. International Conference on Computer Vision: 5229–5238

  29. Tian H, Li W, Ogunbona PO, Wang L (2018) Detection and separation of smoke from single image frames. IEEE Trans Image Process 27(3):1164–1177

    Article  MathSciNet  MATH  Google Scholar 

  30. Vaswani A, Shazeer N, Parmar N et al (2017) Attention is all you need. Neural Inform Process Syst:6000–6010

  31. Wang X, Girshick R, Gupta A, et al. (2018) Non-local neural networks. Computer Vision and Pattern Recognition, : 7794–7803

  32. Wang W, Shen J, Shao L et al (2018) Video salient object detection via fully convolutional networks[J]. IEEE Trans Image Process 27(1):38–49

    Article  MathSciNet  MATH  Google Scholar 

  33. Woo S, Park J, Lee J et al (2018) CBAM: convolutional block attention module[C]. European Conference on Computer Vision:3–19

  34. Xie S, Tu Z (2015) Holistically-nested edge detection. International Conference on Computer Vision:1395–1403

  35. Xie E, Wang W, Yu Z, et al. (2021) SegFormer: simple and efficient design for semantic segmentation with transformers. NeurIPS

  36. Yang J, Lin G, Wang J, Fang J, Zhang Y (2016) Early video smoke segmentation algorithm based on saliency detection and Gaussian mixture model. Comput Eng 42(2):206–209

    Google Scholar 

  37. Yang J, Price B, Cohen S, et al. (2016) Object contour detection with a fully convolutional encoder-decoder network. Computer Vision and Pattern Recognition, 193–202

  38. Yang M, Yu K, Zhang C et al (2018) DenseASPP for semantic segmentation in street scenes. IEEE Conference on Computer Vision and Pattern Recognition:3684–3692

  39. Ye L, Rochan M, Liu Z et al (2019) Cross-modal self-attention network for referring image segmentation. IEEE Conference on Computer Vision and Pattern Recognition:10502–10511

  40. Yu Z, Feng C, Liu M, et al. (2017) CASENet: deep category-aware semantic edge detection. Computer vision and pattern recognition: 1761-1770

  41. Yu C, Wang J, Peng C et al (2018) Learning a discriminative feature network for semantic segmentation[C]. IEEE Conference on Computer Vision and Pattern Recognition:1857–1866

  42. Yuan F, Zhang L, Xia X, Wan B, Huang Q, Li X (2019) Deep smoke segmentation. Neurocomputing 357:248–260

    Article  Google Scholar 

  43. Yuan C, Liu Z, Zhang Y et al (2019) Learning-based smoke detection for unmanned aerial vehicles applied to Forest fire surveillance. J Intell Robot Syst 93:337–349

    Article  Google Scholar 

  44. Yuan F, Zhang L, Xia X, Huang Q, Li X (2020) A wave-shaped deep neural network for smoke density estimation. IEEE Trans Image Process 29:2301–2313

    Article  MATH  Google Scholar 

  45. Yuan F, Zhang L, Xia X, Huang Q, Li X (2021) A gated recurrent network with dual classification assistance for smoke semantic segmentation. IEEE Trans Image Process 30:4409–4422

    Article  Google Scholar 

  46. Yuan Y, Huang L, Guo J et al (2021) OCNet: object context for semantic segmentation. Int J Comput Vis:1–24

  47. Zhang N, Wang H, Yan H (2015) Smoke image segmentation algorithm based on rough set and region growing. J Front Comput Sci Technol 11(8):1296–1299

    Google Scholar 

  48. Zhang Z, Zhang C, Shen W et al (2016) Multi-oriented text detection with fully convolutional networks. IEEE Conference on Computer Vision and Pattern Recognition:4159–4167

  49. Zhang H, Dana KJ, Shi J et al (2018) Context encoding for semantic segmentation. IEEE Conference on Computer Vision and Pattern Recognition:7151–7160

  50. Zhao Y (2015) Candidate smoke region segmentation of fire video based on rough set theory. J Electric Comput Eng 11

  51. Liu Y, Cheng M, Hu X et al (2017) Richer convolutional features for edge detection Computer vision and pattern recognition 41(8):1939–1946

    Google Scholar 

  52. Zhao H, Shi J, Qi X et al (2017) Pyramid scene parsing network. IEEE Conference on Computer Vision and Pattern Recognition:6230–6239

  53. Zhao H, Zhang Y, Liu S et al (2018) PSANet: point-wise spatial attention network for scene parsing. European Conference on Computer Vision:270–286

  54. Zhen M, Wang J, Zhou L, et al. Learning fully dense neural networks for image semantic segmentation[C]. AAAI2019.

  55. Zheng S, Lu J, Zhao H et al (2021) Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers. IEEE Conference on Computer Vision and Pattern Recognition:6881–6890

Download references

Funding

This work was supported by [The National Natural Science Foundation of China (No. 62262027 and No. 62162029), Ph.D. Research Startup Fund (No. 2020BSQD013), Jiangxi Provincial Natural Science Foundation (No. 20212BAB202012) and the Key Science Technology Application Projects of Jiangxi Province (GJJ2201311)].

Author information

Authors and Affiliations

  1. School of Mathematics and Computer Science, Jiangxi Science and Technology Normal University, Nanchang, 330045, Jiangxi, China

    Lin Zhang

  2. College of Information, Mechanical and Electrical Engineering, Shanghai Normal University, Shanghai, 201418, China

    Feiniu Yuan

  3. School of Information Technology, Jiangxi University of Finance and Economics, Nanchang, 330032, Jiangxi, China

    Xue Xia

Authors
  1. Lin Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Feiniu Yuan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xue Xia
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lin Zhang.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, L., Yuan, F. & Xia, X. Edge-reinforced attention network for smoke semantic segmentation. Multimed Tools Appl 82, 31259–31284 (2023). https://doi.org/10.1007/s11042-023-14879-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-14879-z

Keywords

Search

Navigation