Skip to main content
SpringerLink
Log in

Lightweight dual-branch network for vehicle exhausts segmentation

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Visual vehicle exhausts segmentation is a novel and highly challenging task. In this paper, we introduce a lightweight dual-branch vehicle exhausts segmentation network to quickly and accurately infer segmentation masks from multi-interference traffic scenes. Firstly, we propose an encoder-decoder architecture with lightweight residual modules, which is divided into a deep branch for global prediction and a shallow branch for spatial details. Secondly, pyramid attention structure and skip modules are used to expand the receiving range and integrate multi-scale features. Finally, we advance a fusion network to merge the results of two branches so that the entire model can be easily trained end-to-end. To replace the complicated manual annotation, we employ dynamic fluid simulation and computer graphics technology to generate synthetic vehicle exhausts datasets VED. Comprehensive experiments on our synthetic and real datasets demonstrate that the proposed network outperforms existing segmentation networks in terms of speed and accuracy trade-off. Vehicle exhausts segmentation results on real videos are also appealing.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  1. CBCL Street Scenes (2020) [Online] http://cbcl.mit.edu/software-datasets/streetscenes/

  2. Chen Y, Liu L, Tao J, Xia R, Chen X (2020) The improved image inpainting algorithm via encoder and similarity constraint. Visual Comput (3)

  3. Chen L, Papandreou G, Kokkinos I, Murphy K, Yuille AL (2018) DeepLab: Semantic image segmentation with deep convolutional Nets, atrous convolution, and fully connected CRFs. IEEE Trans Pattern Anal Machine Intell 40 (4):834–848

    Article  Google Scholar 

  4. Chen Y, Tao J, Liu L, Xiong J, Yang K (2020) Research of improving semantic image segmentation based on a feature fusion model. J Ambient Intell Human Comput (2)

  5. Chen Y, Wang J, Chen X, Zhu M, Yang K, Wang Z, Xia R (2019) Single-image super-resolution algorithm based on structural self-similarity and deformation block features. IEEE Access 58791–58801

  6. Chen Y, Wang J, Liu S, Chen X, Xiong J, Xie J, Yang K (2019) Multiscale fast correlation filtering tracking algorithm based on a feature fusion model. Concurr Comput Pract Exp

  7. Chen Y, Wang J, Xia R, Zhang Q, Cao Z, Yang K (2019) The visual object tracking algorithm research based on adaptive combination kernel. Ambient Intell

  8. Chen Y, Xu W, Zuo J (2019) The fire recognition algorithm using dynamic feature fusion and IV-SVM classifier. Cluster Comput 22(3):7665–7675

    Article  Google Scholar 

  9. Chen L, Zhu Y, Papandreou G, Schroff F, Adam H (2018) Encoder-decoder with atrous separable convolution for semantic image segmentation. In: ECCV

  10. Filonenko A, Hernandez DC, Jo KH (2017) Fast smoke detection for video surveillance using cuda. IEEE Trans Indust Inform 1–1

  11. Fu J, Liu J, Tian H, Li Y, Bao Y, Fang Z et al (2019) Dual attention network for scene segmentation. In: IEEE/CVF conference on computer vision and pattern recognition (CVPR). IEEE

  12. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: CVPR, pp 770–778

  13. Home of the Blender Project–Free and 3D Creation Software (2018) Accessed 8 Jul 2018. [Online]. https://www.blender.org/

  14. Howard A, Zhu M, Chen B, Kalenichenko D, Wang W, Weyand T, Adam H (2017) MobileNets: Efficient convolutional neural networks for mobile vision applications. arXiv: Comput Vision Pattern Recognit

  15. Huang H, Kalogerakis E, Chaudhuri S, Ceylan D, Kim VG, Yumer E (2017) Learning local shape descriptors from part correspondences with multiview convolutional networks. Int Conf Comput Graphics Interact Techniq

  16. Kajiya JT, Herzen BPV (1984) Ray tracing volume densities. Conference on computer graphics & interactive techniques. ACM

  17. Li X, Chen Z, Wu QM, Liu C (2020) 3D parallel fully convolutional networks for real-time video wildfire smoke detection. IEEE Trans Circ Syst Video Technol 30(1):89–103

    Article  Google Scholar 

  18. Li H, Xiong P, Fan H, Sun J (2019) DFANet: Deep feature aggregation for real-time semantic segmentation. In: CVPR. 2019, pp 9522–9531

  19. Na Z, Huiqin W, Yan HU (2017) Smoke image segmentation algorithm based on rough set and region growing. J Front Comput Sci Technol

  20. Pascal Visual Object Classes (VOC) (2012) [Online]. http://host.robots.ox.ac.uk/pascal/VOC/

  21. Paszke A, Chaurasia A, Kim S, Culurciello E (2017) ENet: A deep neural network architecture for real-time semantic segmentation. arXiv: Comput Vision Pattern Recognit

  22. Poudel RP, Bonde U, Liwicki S, Zach C (2018) ContextNet: Exploring context and detail for semantic segmentation in real-time. In: BMVC

  23. Poudel RP, Liwicki S, Cipolla R (2019) Fast-SCNN: Fast semantic segmentation network. In: BMVC

  24. Pyykonen P, Peussa P, Kutila M, Fong K (2016) Multi-camera-based smoke detection and traffic pollution analysis system. IEEE Int Conf Int Comput Commun Process

  25. Romera E, Alvarez JM, Bergasa LM, Arroyo R (2018) ERFNet: Efficient residual factorized ConvNet for real-time semantic segmentation. IEEE Trans Intell Transport Syst 19(1):263–272

    Article  Google Scholar 

  26. Ronneberger O, Fischer P, Brox T (2015) U-net: Convolutional networks for biomedical image segmentation. In: International conference on medical image computing and computer-assisted intervention, pp 234–241

  27. Sandler M, Howard A, Zhu M, Zhmoginov A, Chen L (2018) Mobilenetv2: Inverted residuals and linear bottlenecks. In: CVPR, pp 4510–4520

  28. Shelhamer E, Long J, Darrell T (2017) Fully convolutional networks for semantic segmentation. IEEE Trans Pattern Anal Machine Intell 39(4):640–651

    Article  Google Scholar 

  29. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. In: CVPR

  30. Sun L, Ma C, Chen Y, Zheng Y, Shim HJ, Wu Z, Jeon B (2019) Low rank component induced spatial-spectral kernel method for Hyperspectral image classification. IEEE Trans Circ Syst Video Technol 1–1

  31. Sun K, Xiao B, Liu D, Wang J (2019). In: IEEE/CVF conference on computer vision and pattern recognition (CVPR). IEEE. Deep high-resolution representation learning for human pose estimation

  32. Tao H (2019) Contour-based smoky vehicle detection from surveillance video for alarm systems. Signal Image Video Process 13(2):217–225

    Article  Google Scholar 

  33. Tao H, Lu X (2018) Smoky vehicle detection in surveillance video based on gray level co-occurrence matrix. Tenth Int Conf Digital Image Process

  34. Tao H, Lu X (2018) Smoky vehicle detection based on range filtering on three orthogonal planes and motion orientation histogram. IEEE Access 57180–57190

  35. Tao H, Lu X (2018) Smoky vehicle detection based on multi-scale block tamura features. Signal Image Video Process

  36. Tao H, Lu X (2018) Automatic smoky vehicle detection from traffic surveillance video based on vehicle rear detection and multi-feature fusion. IET Intell Transp Syst 13(2)

  37. Tao H, Lu X (2019) Smoke vehicle detection based on multi-feature fusion and hidden markov model. J Real-Time Image Process

  38. Tian H, Li W, Wang L, Ogunbona P (2014) Smoke detection in video: An image separation approach. Int J Comput Vis 106(2):192–209

    Article  Google Scholar 

  39. Tsafack N, Sankar S, Abd-El-Atty B, Kengne J, El-Latif AAA (2020) A new chaotic map with dynamic analysis and encryption application in internet of health things. IEEE Access PP(99):1–1

    Google Scholar 

  40. Udacity (2020) [Online]. https://www.udacity.com/self-driving-car/

  41. Wang H, Chen Y (2019) A smoke image segmentation algorithm based on rough set and region growing. J Forest Sci 65(8):321–329

    Article  Google Scholar 

  42. Wang Y, Zhou Q, Liu J, Xiong J, Gao G, Wu X, Latecki L J (2019) Lednet: A lightweight encoder-decoder network for real-time semantic segmentation. In: ICIP, pp 1860–1864

  43. Xu G, Zhang Y, Zhang Q, Lin G, Wang Z, Jia Y, Wang J (2019) Video smoke detection based on deep saliency network. Fire Safety J 277–285

  44. Yang J, Gaohua L, Jinjun W, Jun F, Yongming Z (2016) Early video smoke segmentation algorithm based on saliency detection and gaussian mixture model. Comput Eng 42(2):206–209

    Google Scholar 

  45. Yuan F, Zhang L, Xia X, Huang Q, Li X (2020) A wave-shaped deep neural network for smoke density estimation. IEEE Trans Image Process 2301–2313

  46. Yuan F, Zhang L, Xia X, Wan B, Huang Q, Li X (2019) Deep smoke segmentation. Neurocomputing 248–260

  47. Zhao Y (2015) Candidate smoke region segmentation of fire video based on rough set theory. J Electric Comput Eng

  48. Zhao H, Shi J, Qi X, Wang X, Jia J (2017) Pyramid scene parsing network. computer vision and pattern recognition. In: CVPR, pp 6230–6239

Download references

Acknowledgements

This work is supported by the Key Research and Development Program Project Foundation of Anhui Province, China, under Grant No. 1804a09020049.

Author information

Authors and Affiliations

  1. School of Information Science and Technology, University of Science and Technology of China, Hefei, Anhui, 230027, China

    Chiyun Sheng, Bin Hu, Fanjun Meng & Dong Yin

  2. Key Laboratory of Electromagnetic Space Information of CAS, Hefei, Anhui, 230027, China

    Chiyun Sheng, Bin Hu, Fanjun Meng & Dong Yin

Authors
  1. Chiyun Sheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bin Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Fanjun Meng
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Dong Yin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Dong Yin.

Ethics declarations

Conflict of Interests

The authors declare that they have no conflict of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sheng, C., Hu, B., Meng, F. et al. Lightweight dual-branch network for vehicle exhausts segmentation. Multimed Tools Appl 80, 17785–17806 (2021). https://doi.org/10.1007/s11042-021-10601-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-021-10601-z

Keywords

Search

Navigation