Skip to main content
Springer Nature Link
Log in

EBANet: Efficient Boundary-Aware Network for RGB-D Semantic Segmentation

  • Conference paper
  • First Online:
Cognitive Systems and Information Processing (ICCSIP 2021)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1515))

Included in the following conference series:

  • 1465 Accesses

Abstract

Semantic segmentation is widely used in robot perception and can be used for various subsequent tasks. Depth information has been proven to be a useful clue in the semantic segmentation of RGB-D images for providing a geometric counterpart to the RGB representation. At the same time, considering the importance of object boundaries in the robot’s perception process, it is very necessary to add attention to the boundaries of the objects in the semantic segmentation model.

In this paper, we propose Efficient Boundary-Aware Network (EBANet) which relies on both RGB and depth images as input. We design a boundary attention branch to extract more boundary features of objects in the scene and generate boundary labels for supervision by a Canny edge detector. We also adopt a hybrid loss function fusing Cross-Entropy (CE) and structural similarity (SSIM) loss to guide the network to learn the transformation between the input image and the ground truth at the pixel and patch level. We evaluate our proposed EBANet on the common RGB-D dataset NYUv2 and show that we reach the state-of-the-art performance.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, United States, pp. 431–440. IEEE Computer Society (2015)

    Google Scholar 

  2. Fu, J., et al.: Dual attention network for scene segmentation. In: 32nd IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, United States, pp. 3141–3149. IEEE Computer Society (2019)

    Google Scholar 

  3. He, J., Deng, Z., Qiao, Y.: Dynamic multi-scale filters for semantic segmentation. In: 17th IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Republic of Korea, pp. 3561–3571. Institute of Electrical and Electronics Engineers Inc., United States (2019)

    Google Scholar 

  4. Fu, J., et al.: Adaptive context network for scene parsing. In: 17th IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Republic of Korea, pp. 6747–6756. Institute of Electrical and Electronics Engineers Inc., United States (2019)

    Google Scholar 

  5. Cheng, B., et al.: SPGNet: semantic prediction guidance for scene parsing. In: 17th IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Republic of Korea, pp. 5217–5227. Institute of Electrical and Electronics Engineers Inc., United States (2019)

    Google Scholar 

  6. Zhang, F., et al.: ACFNet: attentional class feature network for semantic segmentation. In: 17th IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Republic of Korea, pp. 6797–6806. Institute of Electrical and Electronics Engineers Inc., United States (2019)

    Google Scholar 

  7. Lee, S., Park, S.J., Hong, K.S.: RDFNet: RGB-D multi-level residual feature fusion for indoor semantic segmentation. In: 16th IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, pp. 4990–4999. Institute of Electrical and Electronics Engineers Inc., United States (2017)

    Google Scholar 

  8. Wang, W., Neumann, U.: Depth-aware CNN for RGB-D segmentation. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11215, pp. 144–161. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01252-6_9

    Chapter  Google Scholar 

  9. Zhang, Z., Cui, Z., Xu, C., Yan, Y., Sebe, N., Yang, J.: Pattern-affinitive propagation across depth, surface normal and semantic segmentation. In: 32nd IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, United States, pp. 4106–4115. IEEE Computer Society (2019)

    Google Scholar 

  10. Chen, Y., Mensink, T., Gavves, E.: 3D neighborhood convolution: learning depthaware features for RGB-D and RGB semantic segmentation. In: 7th International Conference on 3D Vision, 3DV 2019, Quebec, QC, Canada, pp. 173–182. Institute of Electrical and Electronics Engineers Inc., United States (2019)

    Google Scholar 

  11. He, Y., Chiu, W.C., Keuper, M., Fritz, M.: STD2P: RGBD semantic segmentation using spatio-temporal data-driven pooling. In: 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, United States, pp. 7158–7167. Institute of Electrical and Electronics Engineers Inc. (2017)

    Google Scholar 

  12. Li, Z., Gan, Y., Liang, X., Yu, Y., Cheng, H., Lin, L.: LSTM-CF: unifying context modeling and fusion with LSTMs for RGB-D scene labeling. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 541–557. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46475-6_34

    Chapter  Google Scholar 

  13. Cheng, Y., Cai, R., Li, Z., Zhao, X., Huang, K.: Locality-sensitive deconvolution networks with gated fusion for RGB-D indoor semantic segmentation. In: 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, United States, pp. 1475–1483. Institute of Electrical and Electronics Engineers Inc. (2017)

    Google Scholar 

  14. Hung, S.W., Lo, S.Y., Hang, H.M.: Incorporating luminance, depth and color information by a fusion-based network for semantic segmentation. In: 26th IEEE International Conference on Image Processing, ICIP 2019, Taipei, Taiwan, pp. 2374–2378. IEEE Computer Society (2019)

    Google Scholar 

  15. Hazirbas, C., Ma, L., Domokos, C., Cremers, D.: FuseNet: incorporating depth into semantic segmentation via fusion-based CNN architecture. In: Lai, S.-H., Lepetit, V., Nishino, K., Sato, Y. (eds.) ACCV 2016. LNCS, vol. 10111, pp. 213–228. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-54181-5_14

    Chapter  Google Scholar 

  16. Jiang, J., Zheng, L., Luo, F., Zhang, Z.: RedNet: residual encoder-decoder network for indoor RGB-D semantic segmentation. arXiv preprint arXiv:1806.01054 (2018)

  17. Hu, X., Yang, K., Fei, L., Wang, K.: ACNet: attention based network to exploit complementary features for RGBD semantic segmentation. In: 26th IEEE International Conference on Image Processing, ICIP 2019, Taipei, Taiwan, pp. 1440–1444. IEEE Computer Society (2019)

    Google Scholar 

  18. Zhu, X., Hu, H., Lin, S., Dai, J.: Deformable ConvNets v2: more deformable, better results. In: 32nd IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, United States, pp. 9300–9308. IEEE Computer Society (2019)

    Google Scholar 

  19. Cai, Z., Vasconcelos, N.: Cascade R-CNN: delving into high quality object detection. In: 31st Meeting of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, United States, pp. 6154–6162. IEEE Computer Society (2018)

    Google Scholar 

  20. Bodla, N., Singh, B., Chellappa, R., Davis, L.S.: Soft-NMS - improving object detection with one line of code. In: 16th IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, pp. 5562–5570. Institute of Electrical and Electronics Engineers Inc., United States (2017)

    Google Scholar 

  21. Huang, Z., Huang, L., Gong, Y., Huang, C., Wang, X.: Mask scoring R-CNN. In: 32nd IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, United States, pp. 6402–6411. IEEE Computer Society (2019)

    Google Scholar 

  22. Li, Z., Zhuang, Y., Zhang, X., Yu, G., Sun, J.: COCO instance segmentation challenges 2018: winner (2018). http://presentations.cocodataset.org/ECCV18/COCO18-Detect-Megvii.pdf

  23. Qin, X., Zhang, Z., Huang, C., Gao, C., Dehghan, M., Jagersand, M.: BASNet: boundary-aware salient object detection. In: 32nd IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, United States, pp. 7471–7481. IEEE Computer Society (2019)

    Google Scholar 

  24. Chen, X., et al.: Bi-directional cross-modality feature propagation with separation-and-aggregation gate for RGB-D semantic segmentation. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12356, pp. 561–577. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58621-8_33

    Chapter  Google Scholar 

  25. Seichter, D., Köhler, M., Lewandowski, B., Wengefeld, T., Gross, H.: M Efficient RGB-D semantic segmentation for indoor scene analysis. arXiv preprint arXiv:2011.06961

  26. Kong, S., Fowlkes, C.: Recurrent scene parsing with perspective understanding in the loop. In: 31st Meeting of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, United States, pp. 956–965. IEEE Computer Society (2018)

    Google Scholar 

  27. Lin, D., Chen, G., Cohen-Or, D., Heng, P.A., Huang, H.: Cascaded feature network for semantic segmentation of RGB-D images. In: 16th IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, pp. 1320–1328. Institute of Electrical and Electronics Engineers Inc., United States (2017)

    Google Scholar 

  28. Chen, X., Lin, K., Qian, C., Zeng, G., Li, H.: 3D sketch-aware semantic scene completion via semi-supervised structure prior. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Virtual, Online, United States, pp. 4192–4201. IEEE Computer Society (2020)

    Google Scholar 

  29. Valada, A., Mohan, R., Burgard, W.: Self-supervised model adaptation for multimodal semantic segmentation. Int. J. Comput. Vision 128(5), 1239–1285 (2020)

    Article  Google Scholar 

  30. Fooladgar, F., Kasaei, S.: Multi-modal attention-based fusion model for semantic segmentation of RGB-depth images. arXiv preprint arXiv:1912.11691 (2019)

  31. Zhong, Y., Dai, Y., Li, H.: 3D geometry-aware semantic labeling of outdoor street scenes. In: 24th International Conference on Pattern Recognition, ICPR 2018, Beijing, China, pp. 2343–2349. Institute of Electrical and Electronics Engineers Inc., United States (2018)

    Google Scholar 

  32. Xing, Y., Wang, J., Chen, X., Zeng, G.: 2.5D convolution for RGB-D semantic segmentation. In: 26th IEEE International Conference on Image Processing, ICIP 2019, Taipei, Taiwan, pp. 1410–1414. IEEE Computer Society (2019)

    Google Scholar 

  33. Xing, Y., Wang, J., Zeng, G.: Malleable 2.5D convolution: learning receptive fields along the depth-axis for RGB-D scene parsing. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12364, pp. 555–571. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58529-7_33

    Chapter  Google Scholar 

  34. Chen, L., Lin, Z., Wang, Z., Yang, Y.L., Cheng, M.M.: Spatial information guided convolution for real-time RGBD semantic segmentation. IEEE Trans. Image Process. 30(2021), 2313–2324 (2021)

    Article  Google Scholar 

  35. Chen, Y., Mensink, T., Gavves, E.: 3D neighborhood convolution: learning depth-aware features for RGB-D and RGB semantic segmentation. In: 7th International Conference on 3D Vision, 3DV 2019, Quebec, QC, Canada, pp. 173–182. Institute of Electrical and Electronics Engineers Inc., United States (2019)

    Google Scholar 

  36. Hu, J., Shen, L., Albanie, S., Sun, G., Wu, E.: Squeeze-and-excitation networks. IEEE Trans. Pattern Anal. Mach. Intell. 42(8), 2011–2023 (2020)

    Article  Google Scholar 

  37. Li, X., Wang, W., Hu, X., Yang, J.: Selective kernel networks. In: 32nd IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, United States, pp. 510–519. IEEE Computer Society (2019)

    Google Scholar 

  38. Wang, X., Girshick, R., Gupta, A., He, K.: Non-local neural networks. In: 31st Meeting of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, United States, pp. 7794–7803. IEEE Computer Society (2018)

    Google Scholar 

  39. Yu, C., Wang, J., Peng, C., Gao, C., Yu, G., Sang, N.: Learning a discriminative feature network for semantic segmentation. In: 31st Meeting of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, United States, pp. 1857–1866. IEEE Computer Society (2018)

    Google Scholar 

  40. Chen, X., Qi, D., Shen, J.: Boundary-aware network for fast and high-accuracy portrait segmentation. arXiv preprint arXiv:1901.03814

  41. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 29th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, United states, pp. 770–778. IEEE Computer Society (2016)

    Google Scholar 

  42. Zhao, H., Shi, J., Qi, X., Wang, X., Jia, J.: Pyramid scene parsing network. In: 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, United states, pp. 6230–6239. Institute of Electrical and Electronics Engineers Inc., United States (2017)

    Google Scholar 

  43. Orsic, M., Kreso, I., Bevandic, P., Segvic, S.: In defense of pre-trained imagenet architectures for real-time semantic segmentation of road-driving images. In: 32nd IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, United states, pp. 12599–12608. IEEE Computer Society (2019)

    Google Scholar 

  44. Wang, Z., Simoncelli, E.P., Bovik, A.C.: Multiscale structural similarity for image quality assessment. In: The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, vol. 2, no. 2003, pp. 1398–1402 (2003)

    Google Scholar 

  45. Murphy, K.P.: Machine Learning: A Probabilistic Perspective. MIT Press, Cambridge (2012)

    MATH  Google Scholar 

  46. Silberman, N., Hoiem, D., Kohli, P., Fergus, R.: Indoor segmentation and support inference from RGBD images. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7576, pp. 746–760. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33715-4_54

    Chapter  Google Scholar 

  47. Eigen, D., Fergus, R.: Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture. In: 15th IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, pp. 2650–2658. Institute of Electrical and Electronics Engineers Inc., United States (2015)

    Google Scholar 

  48. Paszke, A., et al.: PyTorch: an imperative style, high-performance deep learning library. In: Advances in Neural Information Processing Systems, vol. 32, no. 2019, pp. 8024–8035 (2019)

    Google Scholar 

  49. Kingma, D.P., Ba, J.L.: Adam: a method for stochastic optimization. In: 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, United states (2015)

    Google Scholar 

Download references

Acknowledgment

This work was supported by Major Project of the New Generation of Artificial Intelligence (No. 2018AAA0102900).

Author information

Authors and Affiliations

  1. Beijing University of Posts and Telecommunications, Beijing, China

    Ruiquan Wang, Qingxuan Jia, Yue Shen, Zeyuan Huang, Gang Chen & Junting Fei

Authors
  1. Ruiquan Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Qingxuan Jia
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yue Shen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zeyuan Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Gang Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Junting Fei
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ruiquan Wang .

Editor information

Editors and Affiliations

  1. Tsinghua University, Beijing, China

    Fuchun Sun

  2. National University of Defense Technology, Changsha, China

    Dewen Hu

  3. Universität Hamburg, Hamburg, Germany

    Stefan Wermter

  4. Tsingzhan Artificial Intelligence Research Institute, Nanjing, China

    Lei Yang

  5. Tsinghua University, Beijing, China

    Huaping Liu

  6. Tsinghua University, Beijing, China

    Bin Fang

Rights and permissions

Reprints and permissions

Copyright information

© 2022 Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Wang, R., Jia, Q., Shen, Y., Huang, Z., Chen, G., Fei, J. (2022). EBANet: Efficient Boundary-Aware Network for RGB-D Semantic Segmentation. In: Sun, F., Hu, D., Wermter, S., Yang, L., Liu, H., Fang, B. (eds) Cognitive Systems and Information Processing. ICCSIP 2021. Communications in Computer and Information Science, vol 1515. Springer, Singapore. https://doi.org/10.1007/978-981-16-9247-5_16

Download citation

  • DOI: https://doi.org/10.1007/978-981-16-9247-5_16

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-16-9246-8

  • Online ISBN: 978-981-16-9247-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics