Skip to main content

Advertisement

Springer Nature Link
Log in

A cascaded refined rgb-d salient object detection network based on the attention mechanism

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

The RGB-D salient object detection algorithm simulates human attention behavior and attempts to locate the most visually prominent object(s) from a set of RGB and depth images. Existing works often follow a deterministic decoding network, with few methods explicitly considering how to establish connections between features at various levels. To this end, we first propose a cascaded refined RGB-D salient object detection network based on the attention mechanism (CRNet), whose primary contribution is a cascaded refined upsampling network layout. Specifically, we have developed an adaptive channel transformation ratio α in the micro modification module of convolutional block attention (MM), adaptively adjusting the feature channel conversion ratio according to the original input depth feature level to maximize the integration of contextual information during the feature extraction phase. For the multi-modal feature interaction section, we propose a contextual feature aggregation module (ACF) consisting of separable convolution, dilated convolution, and adaptive averaging pooling. Extend multi-modal fused features’ receptive fields, reduce redundant information, and decrease background noise interference. Furthermore, we first propose a cascaded refined upsampling network, a precise refining process that includes personal refinement, team expansion, and sequential execution operations. Among them, most of the actions were performed in a new sequential refinement module based on attention mechanism (SRM-Wm). We put the training of CRNet under the supervision of a new hybrid loss function. The experiment results show that the structure of our model is simple but very effective and outperforms the 19 SOTAs on six public datasets using four metrics (\(\sim \)1.6% improvement in F-measure vs. the top-ranked model: BBSNet-TIP2021). You can find the code and results of our method at https://github.com/guanyuzong/CR-Net.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  1. Minaee S, Boykov YY, Porikli F, Plaza AJ, Kehtarnavaz N, Terzopoulos D (2021) Image segmentation using deep learning: a survey. IEEE Trans Pattern Anal Mach Intell, pp 3523–3542

  2. Qi S, Ning X, Yang G, Zhang L, Long P, Cai W, Li W (2021) Review of multi-view 3D object recognition methods based on deep learning. Displays

  3. Zhang D, Han J, Cheng G, Yang M-H (2021) Weakly supervised object localization and detection: a survey. IEEE Trans Pattern Anal Mach Intell, pp 5866–5885

  4. Pal SK, Pramanik A, Maiti J, Mitra P (2021) Deep learning in multi-object detection and tracking: state of the art. Appl Intell 51(9):6400–6429

    Article  Google Scholar 

  5. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556

  6. Long J, Shelhamer E, Darrell T (2015) Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3431–3440

  7. Feng D, Barnes N, You S, McCarthy C (2016) Local background enclosure for RGB-D salient object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2343–2350

  8. Guo J, Ren T, Bei J (2016) Salient object detection for RGB-D image via saliency evolution. In: 2016 IEEE international conference on multimedia and expo (ICME), pp 1–6

  9. Song H, Liu Z, Du H, Sun G, Le Meur O, Ren T (2017) Depth-aware salient object detection and segmentation via multiscale discriminative saliency fusion and bootstrap learning. IEEE Trans Image Process 26(9):4204–4216

    Article  MathSciNet  MATH  Google Scholar 

  10. Liu Z, Shi S, Duan Q, Zhang W, Zhao P (2019) Salient object detection for RGB-D image by single stream recurrent convolution neural network. Neurocomputing 363:46–57

    Article  Google Scholar 

  11. Chen H, Li Y (2019) Three-stream attention-aware network for RGB-D salient object detection. IEEE Trans Image Process 28(6):2825–2835

    Article  MathSciNet  MATH  Google Scholar 

  12. Fu K, Fan D-P, Ji G-P, Zhao Q (2020) JL-DCF: Joint learning and densely-cooperative fusion framework for RGB-D salient object detection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 3052–3062

  13. Li C, Cong R, Kwong S, Hou J, Fu H, Zhu G, Zhang D, Huang Q (2020) ASIF-Net: Attention steered interweave fusion network for RGB-D salient object detection. IEEE transactions on cybernetics 51(1):88–100

    Article  Google Scholar 

  14. Fan D-P, Zhai Y, Borji A, Yang J, Shao L (2020) BBS-Net: RGB-D salient object detection with a bifurcated backbone strategy network. In: European conference on computer vision, pp 275–292

  15. Wu J, Zhou W, Luo T, Yu L, Lei J (2021) Multiscale multilevel context and multimodal fusion for RGB-D salient object detection. Signal Process 178:107766

    Article  Google Scholar 

  16. Zhu X, Li Y, Fu H, Fan X, Shi Y, Lei J (2021) RGB-D salient object detection via cross-modal joint feature extraction and low-bound fusion loss. Neurocomputing 453:623–635

    Article  Google Scholar 

  17. Peng H, Li B, Xiong W, Hu W, Ji R (2014) RGBD salient object detection: A benchmark and algorithms. In: European conference on computer vision, pp 92–109

  18. Hou Q, Cheng M-M, Hu X, Borji A, Tu Z, Torr PH (2017) Deeply supervised salient object detection with short connections. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3203–3212

  19. Han J, Chen H, Liu N, Yan C, Li X (2017) CNNS-based RGB-D saliency detection via cross-view transfer and multiview fusion. IEEE transactions on cybernetics 48(11):3171–3183

    Article  Google Scholar 

  20. Zhu C, Cai X, Huang K, Li TH, Li G (2019) PDNet: Prior-model guided depth-enhanced network for salient object detection. In: 2019 IEEE International conference on multimedia and expo (ICME), pp 199–204

  21. Piao Y, Ji W, Li J, Zhang M, Lu H (2019) Depth-induced multi-scale recurrent attention network for saliency detection. In: Proceedings of the IEEE/CVF international conference on Computer Vision, pp 7254–7263

  22. Zhao J-X, Cao Y, Fan D-P, Cheng M-M, Li X-Y, Zhang L (2019) Contrast prior and fluid pyramid integration for RGBD salient object detection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 3927–3936

  23. Cong R, Lei J, Fu H, Huang Q, Cao X, Ling N (2018) HSCS: hierarchical sparsity based co-saliency detection for RGBD images. IEEE Trans Multimedia 21(7):1660–1671

    Article  Google Scholar 

  24. Cheng M-M, Mitra NJ, Huang X, Torr PH, Hu S-M (2014) Global contrast based salient region detection. IEEE Trans Pattern Anal Mach Intell 37(3):569–582

    Article  Google Scholar 

  25. Wei Y, Wen F, Zhu W, Sun J (2012) Geodesic saliency using background priors. In: European conference on computer vision, pp 29–42

  26. Judd T, Ehinger K, Durand F, Torralba A (2009) Learning to predict where humans look. In: 2009 IEEE 12th International conference on computer vision, pp 2106–2113

  27. Hu X, Zhu L, Qin J, Fu C-W, Heng P-A (2018) Recurrently aggregating deep features for salient object detection. In: Proceedings of the AAAI conference on artificial intelligence

  28. Deng Z, Hu X, Zhu L, Xu X, Qin J, Han G, Heng P-A (2018) R3net: Recurrent residual refinement network for saliency detection. In: Proceedings of the 27th international joint conference on artificial intelligence, pp 684–690

  29. Zhao J-X, Liu J-J, Fan D-P, Cao Y, Yang J, Cheng M-M (2019) EGNet: Edge guidance network for salient object detection. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 8779–8788

  30. Qin X, Zhang Z, Huang C, Gao C, Dehghan M, Jagersand M (2019) Basnet: Boundary-aware salient object detection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 7479–7489

  31. Feng M, Lu H, Ding E (2019) Attentive feedback network for boundary-aware salient object detection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 1623–1632

  32. Liu N, Han J, Yang M-H (2018) Picanet: Learning pixel-wise contextual attention for saliency detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3089–3098

  33. Zhao T, Wu X (2019) Pyramid feature attention network for saliency detection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 3085–3094

  34. Chen Z, Xu Q, Cong R, Huang Q (2020) Global context-aware progressive aggregation network for salient object detection. In: Proceedings of the AAAI conference on artificial intelligence, pp 10599–10606

  35. Liu J.-J, Hou Q, Cheng M-M, Feng J, Jiang J (2019) A simple pooling-based design for real-time salient object detection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 3917–3926

  36. Zhang J, Zhang T, Dai Y, Harandi M, Hartley R (2018) Deep unsupervised saliency detection: A multiple noisy labeling perspective. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 9029–9038

  37. Guo J, Ren T, Bei J, Zhu Y (2015) Salient object detection in RGB-D image based on saliency fusion and propagation. In: Proceedings of the 7th international conference on internet multimedia computing and service, pp 1–5

  38. Wu P, Duan L, Kong L (2015) RGB-D salient object detection via feature fusion and multi-scale enhancement. In: CCF chinese conference on computer vision, pp 359–368

  39. Chen H, Li Y, Su D (2019) Multi-modal fusion network with multi-scale multi-path and cross-modal interactions for RGB-D salient object detection. Pattern Recogn 86:376–385

    Article  Google Scholar 

  40. Chen Q, Liu Z, Zhang Y, Fu K, Zhao Q, Du H (2021) RGB-D salient object detection via 3d convolutional neural networks. In: Proceedings of the AAAI conference on artificial intelligence, pp 1063–1071

  41. Luo A, Li, X, Yang F, Jiao Z, Cheng H, Lyu S (2020) Cascade graph neural networks for rgb-d salient object detection. In: European conference on computer vision, pp 346–364

  42. Cong R, Lei J, Fu H, Hou J, Huang Q, Kwong S (2019) Going from RGB to RGBD saliency: a depth-guided transformation model. IEEE Trans Cybern 50(8):3627–3639

    Article  Google Scholar 

  43. Chen C, Wei J, Peng C, Qin H (2021) Depth-quality-aware salient object detection. IEEE Trans Image Process 30:2350–2363

    Article  Google Scholar 

  44. Wang X, Li S, Chen C, Hao A, Qin H (2021) Depth quality-aware selective saliency fusion for RGB-D image salient object detection. Neurocomputing 432:44–56

    Article  Google Scholar 

  45. Bai Z, Liu Z, Li G, Ye L, Wang Y (2021) Circular complement network for RGB-D salient object detection. Neurocomputing 451:95–106

    Article  Google Scholar 

  46. Chen Z, Cong R, Xu Q, Huang Q (2020) DPANet: Depth potentiality-aware gated attention network for RGB-D salient object detection. IEEE Trans Image Process 30:7012–7024

    Article  Google Scholar 

  47. Li C, Cong R, Piao Y, Xu Q, Loy CC (2020) RGB-D salient object detection with cross-modality modulation and selection. In: European conference on computer vision, pp 225–241

  48. Liu Z, Zhang W, Zhao P (2020) A cross-modal adaptive gated fusion generative adversarial network for RGB-D salient object detection. Neurocomputing 387:210–220

    Article  Google Scholar 

  49. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  50. Woo S, Park J, Lee J.-Y, Kweon I.S (2018) Cbam: Convolutional block attention module. In: Proceedings of the european conference on computer vision (ECCV), pp 3–19

  51. De Boer P-T, Kroese DP, Mannor S, Rubinstein RY (2005) A tutorial on the cross-entropy method. Ann Oper Res 134(1):19–67

    Article  MathSciNet  MATH  Google Scholar 

  52. Máttyus G, Luo W, Urtasun R (2017) Deeproadmapper: Extracting road topology from aerial images. In: Proceedings of the IEEE international conference on computer vision, pp 3438–3446

  53. Liu N, Zhang N, Shao l, Han J (2021) Learning selective mutual attention and contrast for rgb-d saliency detection. IEEE Trans Pattern Anal Mach Intell

  54. Cheng Y, Fu H, Wei X, Xiao J, Cao X (2014) Depth enhanced saliency detection method. In: Proceedings of international conference on internet multimedia computing and service, pp 23–27

  55. Ju R, Ge L, Geng W, Ren T, Wu G (2014) Depth saliency based on anisotropic center-surround difference. In: 2014 IEEE International conference on image processing (ICIP), pp 1115–1119

  56. Fan D-P, Lin Z, Zhang Z, Zhu M, Cheng M-M (2020) Rethinking RGB-d salient object detection: models, data sets, and large-scale benchmarks. IEEE Trans Neural Netw Learn Syst 32(5):2075–2089

    Article  Google Scholar 

  57. Niu Y, Geng Y, Li X, Liu F (2012) Leveraging stereopsis for saliency analysis. In: 2012 IEEE Conference on computer vision and pattern recognition, pp 454–461

  58. Zhu C, Li G (2017) A three-pathway psychobiological framework of salient object detection using stereoscopic technology. In: Proceedings of the IEEE international conference on computer vision workshops, pp 3008–3014

  59. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. Adv Neural Inf Process Syst, vol 25

  60. Chen H, Li Y (2018) Progressively complementarity-aware fusion network for RGB-D salient object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3051–3060

  61. KingaD A (2015) Adam : A method for stochastic optimization. In: Anon. international conferenceon learning representations. SanDego: ICLR

  62. Ketkar N, Moolayil J (2021) Introduction to pytorch. In: Deep learning with python, pp 27–91

  63. Borji A, Cheng M-M, Jiang H, Li J (2015) Salient object detection: a benchmark. IEEE Trans Image Process 24(12):5706–5722

    Article  MathSciNet  MATH  Google Scholar 

  64. Achanta R, Hemami S, Estrada F, Susstrunk S (2009) Frequency-tuned salient region detection. In: 2009 IEEE Conference on computer vision and pattern recognition, pp 1597–1604

  65. Margolin R, Zelnik-Manor L, Tal A (2014) How to evaluate foreground maps?. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 248–255

  66. Fan D-P, Gong C, Cao Y, Ren B, Cheng M-M, Borji A (2018) Enhanced-alignment measure for binary foreground map evaluation. arXiv:1805.10421

  67. Perazzi F, Krähenbühl P, Pritch Y, Hornung A (2012) Saliency filters: Contrast based filtering for salient region detection. In: 2012 IEEE Conference on computer vision and pattern recognition, pp 733–740

  68. Wang N, Gong X (2019) Adaptive fusion for RGB-d salient object detection. IEEE Access 7:55277–55284

    Article  Google Scholar 

  69. Piao Y, Rong Z, Zhang M, Ren W, Lu H (2020) A2dele: Adaptive and attentive depth distiller for efficient rgb-d salient object detection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 9060–9069

  70. Liu Z, Tang J, Xiang Q, Zhao P (2020) Salient object detection for RGB-d images by generative adversarial network. Multimed Tools Appl 79(35):25403–25425

    Article  Google Scholar 

  71. Zhao X, Zhang L, Pang Y, Lu H, Zhang L (2020) A single stream network for robust and real-time RGB-D salient object detection. In: European conference on computer vision, pp 646–662

  72. Chen H, Deng Y, Li Y, Hung T-Y, Lin G (2020) RGBD salient object detection via disentangled cross-modal fusion. IEEE Trans Image Process 29:8407–8416

    Article  MATH  Google Scholar 

  73. Zhai Y, Fan D-P, Yang J, Borji A, Shao L, Han J, Wang L (2021) Bifurcated backbone strategy for rgb-d salient object detection. IEEE Trans Image Process 30:8727–8742

    Article  Google Scholar 

  74. Li J, Ji W, Bi Q, Yan C, Zhang M, Piao Y, Lu H et al (2021) Joint semantic mining for weakly supervised RGB-d salient object detection. Adv Neural Inf Process Syst 34:11945–11959

    Google Scholar 

  75. Jiang B, Zhou Z, Wang X, Tang J, Luo B (2020) CmsalGAN: RGB-d salient object detection with cross-view generative adversarial networks. IEEE Trans Multimedia 23:1343–1353

    Article  Google Scholar 

  76. Wu Y-H, Liu Y, Xu J, Bian J-W, Gu Y-C, Cheng M-M (2021) Mobilesal extremely efficient RGB-D salient object detection. IEEE Trans Pattern Anal Mach Intell, pp 1–1

  77. Chen T, Hu X, Xiao J, Zhang G, Wang S (2022) CFIDNEt: cascaded feature interaction decoder for RGB-d salient object detection. Neural Comput & Applic, pp 1–17

Download references

Acknowledgements

This work was supported by the Opening Fund of Key Laboratory of Geological Survey and Evaluation of Ministry of Education (Grant No. GLAB2020 ZR06) and the Fundamental Research Funds for the Central Universities, and the National Natural Science Foundation of China under contract on 61603357.

Author information

Authors and Affiliations

  1. School of Automation, China University of Geosciences, Wuhan, 430074, China

    Guanyu Zong, Longsheng Wei, Siyuan Guo & Yongtao Wang

  2. Hubei key Laboratory of Advanced Control and Intelligent Automation for Complex Systems, Wuhan, 430074, China

    Guanyu Zong, Longsheng Wei, Siyuan Guo & Yongtao Wang

  3. Key Laboratory of Geological Survey and Evaluation of Ministry of Education, Wuhan, 430074, China

    Guanyu Zong, Longsheng Wei & Siyuan Guo

Authors
  1. Guanyu Zong
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Longsheng Wei
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Siyuan Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yongtao Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Longsheng Wei.

Ethics declarations

Conflict of Interests

The authors declare that they have no conflict of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zong, G., Wei, L., Guo, S. et al. A cascaded refined rgb-d salient object detection network based on the attention mechanism. Appl Intell 53, 13527–13548 (2023). https://doi.org/10.1007/s10489-022-04186-9

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-022-04186-9

Keywords

Search

Navigation