Skip to main content
SpringerLink
Log in

Optimized segmentation with image inpainting for semantic mapping in dynamic scenes

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Moving objects will obscure static objects in a dynamic scene. When the existing semantic segmentation methods deal with these static objects, there are often missing or errors in segmentation results. To solve this problem, we propose a framework that combines image inpainting and semantic segmentation, termed SIS. Our framework adds an image inpainting network and an identical semantic segmentation network in series following an original semantic segmentation network, which can make full use of the two semantic segmentation results to obtain the optimized semantic segmentation results in this scene. Moreover, we combined our framework with Simultaneous Localization and Mapping (SLAM), and conducted experiments on the TUM RGB-D dataset. Experimental results show, the combined SLAM system can construct a semantic octree map with more complete and stable semantic information in dynamic scenes.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21

Similar content being viewed by others

References

  1. Badrinarayanan V, Kendall A, Cipolla R (2017) Segnet: a deep convolutional encoder-decoder architecture for image segmentation. IEEE Trans Pattern Anal Machine Intell 39(12):2481–2495

    Article  Google Scholar 

  2. Becattini F, Berlincioni L, Galteri L, Seidenari L, Del Bimbo A (2018) Semantic road layout understanding by generative adversarial inpainting CoRR

  3. Bescos B, Cadena C, Neira J (2020) Empty cities: a dynamic-object-invariant space for visual slam. IEEE Trans Robot 37(2):433–451

    Article  Google Scholar 

  4. Chen LC, Papandreou G, Kokkinos I, Murphy K, Yuille AL (2014) Semantic image segmentation with deep convolutional nets and fully connected crfs. arXiv:1412.7062

  5. Chen LC, Papandreou G, Kokkinos I, Murphy K, Yuille AL (2017) Deeplab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Trans Pattern Anal Mach Intell 40(4):834–848

    Article  Google Scholar 

  6. Chen LC, Papandreou G, Schroff F, Adam H (2017) Rethinking atrous convolution for semantic image segmentation. arXiv:1706.05587

  7. Chen LC, Zhu Y, Papandreou G, Schroff F, Adam H (2018) Encoder-decoder with atrous separable convolution for semantic image segmentation. In: Proceedings of the European conference on computer vision (ECCV), pp 801–818

  8. Cordts M, Omran M, Ramos S, Rehfeld T, Enzweiler M, Benenson R, Franke U, Roth S, Schiele B (2016) The cityscapes dataset for semantic urban scene understanding. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3213–3223

  9. Everingham M, Eslami SA, Van Gool L, Williams CK, Winn J, Zisserman A (2015) The pascal visual object classes challenge: a retrospective. Int J Comput Vis 111(1):98–136

    Article  Google Scholar 

  10. He K, Sun J (2014) Image completion approaches using the statistics of similar patches. IEEE Trans Pattern Anal Mach Intell 36(12):2423–2435

    Article  Google Scholar 

  11. Huang Z, Wang X, Huang L, Huang C, Wei Y, Liu W (2019) Ccnet: criss-cross attention for semantic segmentation. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 603–612

  12. Iizuka S, Simo-Serra E, Ishikawa H (2017) Globally and locally consistent image completion. ACM Trans Graph (ToG) 36(4):1–14

    Article  Google Scholar 

  13. Kipf TN, Welling M (2016) Semi-supervised classification with graph convolutional networks. arXiv:1609.02907

  14. Li X, Yang Y, Zhao Q, Shen T, Lin Z, Liu H (2020) Spatial pyramid based graph reasoning for semantic segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 8950–8959

  15. Liu G, Reda FA, Shih KJ, Wang TC, Tao A, Catanzaro B (2018) Image inpainting for irregular holes using partial convolutions. In: Proceedings of the European conference on computer vision (ECCV), pp 85–100

  16. Long J, Shelhamer E, Darrell T (2015) Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3431–3440

  17. Lu C, Dubbelman G (2020) Semantic foreground inpainting from weak supervision. IEEE Robot Autom Lett 5(2):1334–1341

    Article  Google Scholar 

  18. Ma L, Stückler J, Kerl C, Cremers D (2017) Multi-view deep learning for consistent semantic mapping with rgb-d cameras. In: 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp 598–605. IEEE

  19. McCormac J, Handa A, Davison A, Leutenegger S (2017) Semanticfusion: dense 3d semantic mapping with convolutional neural networks. In: 2017 IEEE International conference on robotics and automation (ICRA), pp 4628–4635. IEEE

  20. Newcombe RA, Izadi S, Hilliges O, Molyneaux D, Kim D, Davison AJ, Kohi P, Shotton J, Hodges S, Fitzgibbon A (2011) Kinectfusion: real-time dense surface mapping and tracking. In: 2011 10th IEEE international symposium on mixed and augmented reality, pp 127–136. IEEE

  21. Ngo L, Cha J, Han JH (2019) Deep neural network regression for automated retinal layer segmentation in optical coherence tomography images. IEEE Trans Image Process 29:303–312

    Article  MathSciNet  MATH  Google Scholar 

  22. Song Y, Yang C, Shen Y, Wang P, Huang Q, Kuo CCJ (2018) Spg-net: segmentation prediction and guidance network for image inpainting. arXiv:1805.03356

  23. Sturm J, Engelhard N, Endres F, Burgard W, Cremers D (2012) A benchmark for the evaluation of rgb-d slam systems. In: 2012 IEEE/RSJ international conference on intelligent robots and systems, pp 573–580. IEEE

  24. Wu P, Li H, Zeng N, Li F (2022) Fmd-yolo: an efficient face mask detection method for covid-19 prevention and control in public. Image Vis Comput 117:104341

    Article  Google Scholar 

  25. Xiang Y, Fox D (2017) Da-rnn: Semantic mapping with data associated recurrent neural networks. arXiv:1703.03098

  26. Yu C, Liu Z, Liu X, Xie F, Yang Y, Wei Q, Fei Q (2018) Ds-slam: a semantic visual slam towards dynamic environments. In: 2018 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp 1168–1174. IEEE

  27. Yu J, Lin Z, Yang J, Shen X, Lu X, Huang TS (2018) Generative image inpainting with contextual attention. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 5505–5514

  28. Yu J, Lin Z, Yang J, Shen X, Lu X, Huang TS (2019) Free-form image inpainting with gated convolution. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 4471–4480

  29. Yu X, Lu Y, Gao Q (2021) Pipeline image diagnosis algorithm based on neural immune ensemble learning. Int J Press Vessel Pip 189:104249

    Article  Google Scholar 

  30. Yu X, Zhou Z, Gao Q, Li D, Ríha K (2018) Infrared image segmentation using growing immune field and clone threshold. Infrared Phys Technol 88:184–193

    Article  Google Scholar 

  31. Zeng N, Li H, Peng Y (2021) A new deep belief network-based multi-task learning for diagnosis of alzheimer’s disease. Neural Comput and Applic, 1–12

  32. Zeng N, Wang Z, Zhang H, Kim KE, Li Y, Liu X (2019) An improved particle filter with a novel hybrid proposal distribution for quantitative analysis of gold immunochromatographic strips. IEEE Trans Nanotechnol 18:819–829

    Article  Google Scholar 

Download references

Acknowledgements

We acknowledge the support of the National Key Research and De-velopment Program of China under Grant (2018YFB1305001), Wuhan Science and Technology Planning Application Foundation Frontier Project (No.2019010701011413) and Open Fund of Hubei Luojia Laboratory.

Author information

Authors and Affiliations

  1. School of Computer Science, Wuhan University, Wuhan, China

    Jianfeng Zhang

  2. Global Navigation Satellite System Research Center, Wuhan University, Wuhan, China

    Yang Liu, Chi Guo & Jiao Zhan

  3. Intelligence Institute of Wuhan University, Wuhan University, Wuhan, China

    Chi Guo

  4. Hubei Luojia Laboratory, Wuhan, China

    Chi Guo

Authors
  1. Jianfeng Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yang Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chi Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jiao Zhan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Chi Guo.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, J., Liu, Y., Guo, C. et al. Optimized segmentation with image inpainting for semantic mapping in dynamic scenes. Appl Intell 53, 2173–2188 (2023). https://doi.org/10.1007/s10489-022-03487-3

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-022-03487-3

Keywords

Search

Navigation