Skip to main content
SpringerLink
Log in

Multi-representation decoupled joint network for semantic segmentation of remote sensing images

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

In recent years, semantic segmentation has become an important means of processing remote sensing images, and it is widely used in various fields such as natural disaster detection, environmental protection, and land resource management. In response to this, the mainstream method of the deep convolutional network is constantly innovating and iterating. However, previous methods usually do not fully exploit the information associations between different representations, and the information of low-level representations is usually not well applied. In response to this, we propose a multi-representation decoupled joint network (MDJN) based on a three-branch architecture to improve the performance of semantic segmentation on remote sensing images, which utilizes multi-representation decoupling (MRD) to decouple the original single-branch network into the main branch, body branch and edge branch to enhance information fusion for different representations. Specifically, based on representation learning, we first propose a cross-representation graph convolution (CGC) module to fully mine and learn the context information between different representations with the help of graph convolutional networks (GCN). Secondly, we propose a new three-branch information interaction (TII) module to perform three-way interaction for the information of the three branches, so that the intra-class consistency and inter-class expressivity between different representations can fully play a role. The mean intersection over union (mIoU) of MDJN reaches 78.19% and 81.26% respectively on on both International Society for Photogrammetry and Remote Sensing (ISPRS) Vaihingen and Potsdam datasets.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Data Availability

All data analysed during this study are included in this published article [32].

References

  1. Badrinarayanan V, Kendall A, Cipolla R (2017) Segnet: A deep convolutional encoder-decoder architecture for image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence 39(12):2481–2495

    Article  Google Scholar 

  2. Bhatti UA, Huang M, Wu D, Zhang Y, Mehmood A, Han H (2019) Recommendation system using feature extraction and pattern recognition in clinical care systems. Enterp Inf Syst 13(3):329–351

    Article  Google Scholar 

  3. Bhatti UA, Yu Z, Yuan L, Zeeshan Z, Nawaz SA, Bhatti M, Mehmood A, Ain QU, Wen L (2020) Geometric algebra applications in geospatial artificial intelligence and remote sensing image processing. IEEE Access 8:155783–155796

    Article  Google Scholar 

  4. Bhatti UA, Yu Z, Chanussot J, Zeeshan Z, Yuan L, Luo W, Nawaz SA, Bhatti MA, Ain QU, Mehmood A (2021) Local similarity-based spatial-spectral fusion hyperspectral image classification with deep cnn and gabor filtering. IEEE Transactions on Geoscience and Remote Sensing 60:1–15

    Article  Google Scholar 

  5. Bhatti UA, Ming-Quan Z, Qing-Song H, Ali S, Hussain A, Yuhuan Y, Yu Z, Yuan L, Nawaz SA (2021) Advanced color edge detection using clifford algebra in satellite images. IEEE Photon J 13(2):1–20

    Article  Google Scholar 

  6. Bhatti UA, Zeeshan Z, Nizamani MM, Bazai S, Yu Z, Yuan L (2022) Assessing the change of ambient air quality patterns in jiangsu province of china pre-to post-covid-19. Chemosphere 288:132569

    Article  Google Scholar 

  7. Bruna J, Zaremba W, Szlam A, LeCun Y (2013) Spectral networks and locally connected networks on graphs. arXiv preprint arXiv:1312.6203

  8. Chaurasia A, Culurciello E (2017) Linknet: Exploiting encoder representations for efficient semantic segmentation. In: 2017 IEEE Visual Communications and Image Processing (VCIP). IEEE, pp 1–4

  9. Chen L-C, Papandreou G, Schroff F, Adam H (2017) Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587

  10. Chen L-C, Zhu Y, Papandreou G, Schroff F, Adam H (2018) Encoder-decoder with atrous separable convolution for semantic image segmentation. In: Proceedings of the European Conference on Computer Vision (ECCV). pp 801–818

  11. Chen L-C, Papandreou G, Kokkinos I, Murphy K, Yuille AL (2017) Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Transactions on Pattern Analysis and Machine Intelligence 40(4):834–848

    Article  Google Scholar 

  12. Chen Y, Rohrbach M, Yan Z, Shuicheng Y, Feng J, Kalantidis Y (2019) Graph-based global reasoning networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp 433–442

  13. Chen A, Zhou Y (2020) An attention enhanced graph convolutional network for semantic segmentation. In: Chinese Conference on Pattern Recognition and Computer Vision (PRCV). Springer, pp 734–745

  14. Chollet F (2017) Xception: Deep learning with depthwise separable convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp 1251–1258

  15. Chong Y, Chen X, Pan S (2020) Context union edge network for semantic segmentation of small-scale objects in very high resolution remote sensing images. IEEE Geoscience and Remote Sensing Letters

  16. Defferrard M, Bresson X, Vandergheynst P (2016) Convolutional neural networks on graphs with fast localized spectral filtering. Advances in neural information processing systems, 29

  17. Fu J, Liu J, Tian H, Li Y, Bao Y, Fang Z, Lu H (2019) Dual attention network for scene segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp 3146–3154

  18. Gilmer J, Schoenholz SS, Riley PF, Vinyals O, Dahl GE (2017) Neural message passing for quantum chemistry. In: International Conference on Machine Learning. PMLR, pp 1263–1272

  19. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp 770–778

  20. Huang Z, Wang X, Huang L, Huang C, Wei Y, Liu W (2019) Ccnet: Criss-cross attention for semantic segmentation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp 603–612

  21. Hu X, Yang K, Fei L, Wang K (2019) Acnet: Attention based network to exploit complementary features for rgbd semantic segmentation. In: 2019 IEEE International Conference on Image Processing (ICIP). IEEE, pp 1440–1444

  22. Kipf TN, Welling M (2016) Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907

  23. Li R, Zheng S, Zhang C, Duan C, Wang L, Atkinson PM (2021) Abcnet: Attentive bilateral contextual network for efficient semantic segmentation of fine-resolution remotely sensed imagery. ISPRS J Photogramm Remote Sens 181:84–98

    Article  Google Scholar 

  24. Liang X, Hu Z, Zhang H, Lin L, Xing EP (2018) Symbolic graph reasoning meets convolutions. Advances in Neural Information Processing Systems, 31

  25. Li X, Li X, Zhang L, Cheng G, Shi J, Lin Z, Tan S, Tong Y (2020) Improving semantic segmentation via decoupled body and edge supervision. In: European Conference on Computer Vision. Springer, pp 435–452

  26. Lin D, Zhang R, Ji Y, Li P, Huang H (2018) Scn: Switchable context network for semantic segmentation of rgb-d images. IEEE Trans Cybern 50(3):1120–1131

    Article  Google Scholar 

  27. Lin G, Milan A, Shen C, Reid I (2017) Refinenet: Multi-path refinement networks for high-resolution semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp 1925–1934

  28. Long J, Shelhamer E, Darrell T (2015) Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp 3431–3440

  29. Niu R, Sun X, Tian Y, Diao W, Chen K, Fu K (2021) Hybrid multiple attention network for semantic segmentation in aerial images. IEEE Transactions on Geoscience and Remote Sensing 60:1–18

    Google Scholar 

  30. Nogueira K, Dalla Mura M, Chanussot J, Schwartz WR, Dos Santos JA (2019) Dynamic multicontext segmentation of remote sensing images based on convolutional networks. IEEE Trans Geosci Remote Sens 57(10):7503–7520

    Article  Google Scholar 

  31. Ronneberger O, Fischer P, Brox T (2015) U-net: Convolutional networks for biomedical image segmentation. In: International Conference on Medical Image Computing and Computer-assisted Intervention. Springer, pp 234–241

  32. Rottensteiner F, Sohn G, Gerke M, Wegner JD (2014) Isprs semantic labeling contest. ISPRS: Leopoldshöhe, Germany 1:4

  33. Sulla-Menashe D, Gray JM, Abercrombie SP, Friedl MA (2019) Hierarchical mapping of annual global land cover 2001 to present: The modis collection 6 land cover product. Remote Sens Environ 222:183–194

    Article  Google Scholar 

  34. Takikawa T, Acuna D, Jampani V, Fidler S (2019) Gated-scnn: Gated shape cnns for semantic segmentation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp 5229–5238

  35. Wang X, Girshick R, Gupta A, He K (2018) Non-local neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp 7794–7803

  36. Xie E, Wang W, Yu Z, Anandkumar A, Alvarez JM, Luo P (2021) Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in Neural Information Processing Systems 34:12077–12090

    Google Scholar 

  37. Yin M, Yao Z, Cao Y, Li X, Zhang Z, Lin S, Hu H (2020) Disentangled non-local neural networks. In: European Conference on Computer Vision. pp 191–207

  38. Yuan Y, Huang L, Guo J, Zhang C, Chen X, Wang J (2021) Ocnet: Object context for semantic segmentation. Int J Comput Vision 129(8):2375–2398

    Article  Google Scholar 

  39. Zeiler MD, Fergus R (2014) Visualizing and understanding convolutional networks. In: European Conference on Computer Vision. Springer, pp 818–833

  40. Zhang J, Feng L, Yao F (2014) Improved maize cultivated area estimation over a large scale combining modis-evi time series data and crop phenological information. ISPRS J Photogramm Remote Sens 94:102–113

  41. Zhang C, Sargent I, Pan X, Li H, Gardiner A, Hare J, Atkinson PM (2019) Joint deep learning for land cover and land use classification. Remote Sensing of Environment 221:173–187

    Article  Google Scholar 

  42. Zhang X, Wei Y, Yang Y, Huang TS (2020) Sg-one: Similarity guidance network for one-shot semantic segmentation. IEEE Trans Cybern 50(9):3855–3865

    Article  Google Scholar 

  43. Zhang C, Harrison PA, Pan X, Li H, Sargent I, Atkinson PM (2020) Scale sequence joint deep learning (ss-jdl) for land use and land cover classification. Remote Sens Environ 237:111593

    Article  Google Scholar 

  44. Zhang F, Chen Y, Li Z, Hong Z, Liu J, Ma F, Han J, Ding E (2019) Acfnet: Attentional class feature network for semantic segmentation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp 6798–6807

  45. Zhang J, Yang K, Constantinescu A, Peng K, Müller K, Stiefelhagen R (2021) Trans4trans: Efficient transformer for transparent object segmentation to help visually impaired people navigate in the real world. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp 1760–1770

  46. Zhao H, Qi X, Shen X, Shi J, Jia J (2018) Icnet for real-time semantic segmentation on high-resolution images. In: Proceedings of the European Conference on Computer Vision (ECCV). pp 405–420

  47. Zhao H, Shi J, Qi X, Qi X, Wang X, Jia J (2017) Pyramid scene parsing network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp 2881–2890

  48. Zheng S, Lu J, Zhao H, Zhu X, Luo Z, Wang Y, Fu Y, Feng J, Xiang T, Torr PH et al (2021) Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp 6881–6890

  49. Zhen M, Wang J, Zhou L, Li S, Shen T, Shang J, Fang T, Quan L (2020) Joint semantic segmentation and boundary detection using iterative pyramid contexts. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp 13666–13675

  50. Zhong Z, Lin ZQ, Bidart R, Hu X, Daya IB, Li Z, Zheng W-S, Li J, Wong A (2020) Squeeze-and-attention networks for semantic segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp 13065–13074

  51. Zhu Z, Xu M, Bai S, Huang T, Bai X (2019) Asymmetric non-local neural networks for semantic segmentation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp 593–602

Download references

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China(62072418, 62172376); the Major Scientific and Technological Innovation Project of Shandong (2019JZZY020705); the Fundamental Research Funds for the Central Universities (202042008).

Author information

Authors and Affiliations

  1. Faculty of Information Science and Engineering, Ocean University of China, 238 Songling Road, 266100, Qingdao, China

    Xiaowei Lv, Rui Wang, Chengyu Zheng, Qicheng Yang, Zhaoxin Wang & Jie Nie

Authors
  1. Xiaowei Lv
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rui Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chengyu Zheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Qicheng Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Zhaoxin Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Jie Nie
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jie Nie.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lv, X., Wang, R., Zheng, C. et al. Multi-representation decoupled joint network for semantic segmentation of remote sensing images. Multimed Tools Appl 83, 13291–13313 (2024). https://doi.org/10.1007/s11042-023-15660-y

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-15660-y

Keywords

Search

Navigation