Abstract
The RGB-D salient object detection algorithm simulates human attention behavior and attempts to locate the most visually prominent object(s) from a set of RGB and depth images. Existing works often follow a deterministic decoding network, with few methods explicitly considering how to establish connections between features at various levels. To this end, we first propose a cascaded refined RGB-D salient object detection network based on the attention mechanism (CRNet), whose primary contribution is a cascaded refined upsampling network layout. Specifically, we have developed an adaptive channel transformation ratio α in the micro modification module of convolutional block attention (MM), adaptively adjusting the feature channel conversion ratio according to the original input depth feature level to maximize the integration of contextual information during the feature extraction phase. For the multi-modal feature interaction section, we propose a contextual feature aggregation module (ACF) consisting of separable convolution, dilated convolution, and adaptive averaging pooling. Extend multi-modal fused features’ receptive fields, reduce redundant information, and decrease background noise interference. Furthermore, we first propose a cascaded refined upsampling network, a precise refining process that includes personal refinement, team expansion, and sequential execution operations. Among them, most of the actions were performed in a new sequential refinement module based on attention mechanism (SRM-Wm). We put the training of CRNet under the supervision of a new hybrid loss function. The experiment results show that the structure of our model is simple but very effective and outperforms the 19 SOTAs on six public datasets using four metrics (\(\sim \)1.6% improvement in F-measure vs. the top-ranked model: BBSNet-TIP2021). You can find the code and results of our method at https://github.com/guanyuzong/CR-Net.









Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Minaee S, Boykov YY, Porikli F, Plaza AJ, Kehtarnavaz N, Terzopoulos D (2021) Image segmentation using deep learning: a survey. IEEE Trans Pattern Anal Mach Intell, pp 3523–3542
Qi S, Ning X, Yang G, Zhang L, Long P, Cai W, Li W (2021) Review of multi-view 3D object recognition methods based on deep learning. Displays
Zhang D, Han J, Cheng G, Yang M-H (2021) Weakly supervised object localization and detection: a survey. IEEE Trans Pattern Anal Mach Intell, pp 5866–5885
Pal SK, Pramanik A, Maiti J, Mitra P (2021) Deep learning in multi-object detection and tracking: state of the art. Appl Intell 51(9):6400–6429
Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556
Long J, Shelhamer E, Darrell T (2015) Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3431–3440
Feng D, Barnes N, You S, McCarthy C (2016) Local background enclosure for RGB-D salient object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2343–2350
Guo J, Ren T, Bei J (2016) Salient object detection for RGB-D image via saliency evolution. In: 2016 IEEE international conference on multimedia and expo (ICME), pp 1–6
Song H, Liu Z, Du H, Sun G, Le Meur O, Ren T (2017) Depth-aware salient object detection and segmentation via multiscale discriminative saliency fusion and bootstrap learning. IEEE Trans Image Process 26(9):4204–4216
Liu Z, Shi S, Duan Q, Zhang W, Zhao P (2019) Salient object detection for RGB-D image by single stream recurrent convolution neural network. Neurocomputing 363:46–57
Chen H, Li Y (2019) Three-stream attention-aware network for RGB-D salient object detection. IEEE Trans Image Process 28(6):2825–2835
Fu K, Fan D-P, Ji G-P, Zhao Q (2020) JL-DCF: Joint learning and densely-cooperative fusion framework for RGB-D salient object detection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 3052–3062
Li C, Cong R, Kwong S, Hou J, Fu H, Zhu G, Zhang D, Huang Q (2020) ASIF-Net: Attention steered interweave fusion network for RGB-D salient object detection. IEEE transactions on cybernetics 51(1):88–100
Fan D-P, Zhai Y, Borji A, Yang J, Shao L (2020) BBS-Net: RGB-D salient object detection with a bifurcated backbone strategy network. In: European conference on computer vision, pp 275–292
Wu J, Zhou W, Luo T, Yu L, Lei J (2021) Multiscale multilevel context and multimodal fusion for RGB-D salient object detection. Signal Process 178:107766
Zhu X, Li Y, Fu H, Fan X, Shi Y, Lei J (2021) RGB-D salient object detection via cross-modal joint feature extraction and low-bound fusion loss. Neurocomputing 453:623–635
Peng H, Li B, Xiong W, Hu W, Ji R (2014) RGBD salient object detection: A benchmark and algorithms. In: European conference on computer vision, pp 92–109
Hou Q, Cheng M-M, Hu X, Borji A, Tu Z, Torr PH (2017) Deeply supervised salient object detection with short connections. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3203–3212
Han J, Chen H, Liu N, Yan C, Li X (2017) CNNS-based RGB-D saliency detection via cross-view transfer and multiview fusion. IEEE transactions on cybernetics 48(11):3171–3183
Zhu C, Cai X, Huang K, Li TH, Li G (2019) PDNet: Prior-model guided depth-enhanced network for salient object detection. In: 2019 IEEE International conference on multimedia and expo (ICME), pp 199–204
Piao Y, Ji W, Li J, Zhang M, Lu H (2019) Depth-induced multi-scale recurrent attention network for saliency detection. In: Proceedings of the IEEE/CVF international conference on Computer Vision, pp 7254–7263
Zhao J-X, Cao Y, Fan D-P, Cheng M-M, Li X-Y, Zhang L (2019) Contrast prior and fluid pyramid integration for RGBD salient object detection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 3927–3936
Cong R, Lei J, Fu H, Huang Q, Cao X, Ling N (2018) HSCS: hierarchical sparsity based co-saliency detection for RGBD images. IEEE Trans Multimedia 21(7):1660–1671
Cheng M-M, Mitra NJ, Huang X, Torr PH, Hu S-M (2014) Global contrast based salient region detection. IEEE Trans Pattern Anal Mach Intell 37(3):569–582
Wei Y, Wen F, Zhu W, Sun J (2012) Geodesic saliency using background priors. In: European conference on computer vision, pp 29–42
Judd T, Ehinger K, Durand F, Torralba A (2009) Learning to predict where humans look. In: 2009 IEEE 12th International conference on computer vision, pp 2106–2113
Hu X, Zhu L, Qin J, Fu C-W, Heng P-A (2018) Recurrently aggregating deep features for salient object detection. In: Proceedings of the AAAI conference on artificial intelligence
Deng Z, Hu X, Zhu L, Xu X, Qin J, Han G, Heng P-A (2018) R3net: Recurrent residual refinement network for saliency detection. In: Proceedings of the 27th international joint conference on artificial intelligence, pp 684–690
Zhao J-X, Liu J-J, Fan D-P, Cao Y, Yang J, Cheng M-M (2019) EGNet: Edge guidance network for salient object detection. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 8779–8788
Qin X, Zhang Z, Huang C, Gao C, Dehghan M, Jagersand M (2019) Basnet: Boundary-aware salient object detection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 7479–7489
Feng M, Lu H, Ding E (2019) Attentive feedback network for boundary-aware salient object detection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 1623–1632
Liu N, Han J, Yang M-H (2018) Picanet: Learning pixel-wise contextual attention for saliency detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3089–3098
Zhao T, Wu X (2019) Pyramid feature attention network for saliency detection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 3085–3094
Chen Z, Xu Q, Cong R, Huang Q (2020) Global context-aware progressive aggregation network for salient object detection. In: Proceedings of the AAAI conference on artificial intelligence, pp 10599–10606
Liu J.-J, Hou Q, Cheng M-M, Feng J, Jiang J (2019) A simple pooling-based design for real-time salient object detection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 3917–3926
Zhang J, Zhang T, Dai Y, Harandi M, Hartley R (2018) Deep unsupervised saliency detection: A multiple noisy labeling perspective. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 9029–9038
Guo J, Ren T, Bei J, Zhu Y (2015) Salient object detection in RGB-D image based on saliency fusion and propagation. In: Proceedings of the 7th international conference on internet multimedia computing and service, pp 1–5
Wu P, Duan L, Kong L (2015) RGB-D salient object detection via feature fusion and multi-scale enhancement. In: CCF chinese conference on computer vision, pp 359–368
Chen H, Li Y, Su D (2019) Multi-modal fusion network with multi-scale multi-path and cross-modal interactions for RGB-D salient object detection. Pattern Recogn 86:376–385
Chen Q, Liu Z, Zhang Y, Fu K, Zhao Q, Du H (2021) RGB-D salient object detection via 3d convolutional neural networks. In: Proceedings of the AAAI conference on artificial intelligence, pp 1063–1071
Luo A, Li, X, Yang F, Jiao Z, Cheng H, Lyu S (2020) Cascade graph neural networks for rgb-d salient object detection. In: European conference on computer vision, pp 346–364
Cong R, Lei J, Fu H, Hou J, Huang Q, Kwong S (2019) Going from RGB to RGBD saliency: a depth-guided transformation model. IEEE Trans Cybern 50(8):3627–3639
Chen C, Wei J, Peng C, Qin H (2021) Depth-quality-aware salient object detection. IEEE Trans Image Process 30:2350–2363
Wang X, Li S, Chen C, Hao A, Qin H (2021) Depth quality-aware selective saliency fusion for RGB-D image salient object detection. Neurocomputing 432:44–56
Bai Z, Liu Z, Li G, Ye L, Wang Y (2021) Circular complement network for RGB-D salient object detection. Neurocomputing 451:95–106
Chen Z, Cong R, Xu Q, Huang Q (2020) DPANet: Depth potentiality-aware gated attention network for RGB-D salient object detection. IEEE Trans Image Process 30:7012–7024
Li C, Cong R, Piao Y, Xu Q, Loy CC (2020) RGB-D salient object detection with cross-modality modulation and selection. In: European conference on computer vision, pp 225–241
Liu Z, Zhang W, Zhao P (2020) A cross-modal adaptive gated fusion generative adversarial network for RGB-D salient object detection. Neurocomputing 387:210–220
He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778
Woo S, Park J, Lee J.-Y, Kweon I.S (2018) Cbam: Convolutional block attention module. In: Proceedings of the european conference on computer vision (ECCV), pp 3–19
De Boer P-T, Kroese DP, Mannor S, Rubinstein RY (2005) A tutorial on the cross-entropy method. Ann Oper Res 134(1):19–67
Máttyus G, Luo W, Urtasun R (2017) Deeproadmapper: Extracting road topology from aerial images. In: Proceedings of the IEEE international conference on computer vision, pp 3438–3446
Liu N, Zhang N, Shao l, Han J (2021) Learning selective mutual attention and contrast for rgb-d saliency detection. IEEE Trans Pattern Anal Mach Intell
Cheng Y, Fu H, Wei X, Xiao J, Cao X (2014) Depth enhanced saliency detection method. In: Proceedings of international conference on internet multimedia computing and service, pp 23–27
Ju R, Ge L, Geng W, Ren T, Wu G (2014) Depth saliency based on anisotropic center-surround difference. In: 2014 IEEE International conference on image processing (ICIP), pp 1115–1119
Fan D-P, Lin Z, Zhang Z, Zhu M, Cheng M-M (2020) Rethinking RGB-d salient object detection: models, data sets, and large-scale benchmarks. IEEE Trans Neural Netw Learn Syst 32(5):2075–2089
Niu Y, Geng Y, Li X, Liu F (2012) Leveraging stereopsis for saliency analysis. In: 2012 IEEE Conference on computer vision and pattern recognition, pp 454–461
Zhu C, Li G (2017) A three-pathway psychobiological framework of salient object detection using stereoscopic technology. In: Proceedings of the IEEE international conference on computer vision workshops, pp 3008–3014
Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. Adv Neural Inf Process Syst, vol 25
Chen H, Li Y (2018) Progressively complementarity-aware fusion network for RGB-D salient object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3051–3060
KingaD A (2015) Adam : A method for stochastic optimization. In: Anon. international conferenceon learning representations. SanDego: ICLR
Ketkar N, Moolayil J (2021) Introduction to pytorch. In: Deep learning with python, pp 27–91
Borji A, Cheng M-M, Jiang H, Li J (2015) Salient object detection: a benchmark. IEEE Trans Image Process 24(12):5706–5722
Achanta R, Hemami S, Estrada F, Susstrunk S (2009) Frequency-tuned salient region detection. In: 2009 IEEE Conference on computer vision and pattern recognition, pp 1597–1604
Margolin R, Zelnik-Manor L, Tal A (2014) How to evaluate foreground maps?. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 248–255
Fan D-P, Gong C, Cao Y, Ren B, Cheng M-M, Borji A (2018) Enhanced-alignment measure for binary foreground map evaluation. arXiv:1805.10421
Perazzi F, Krähenbühl P, Pritch Y, Hornung A (2012) Saliency filters: Contrast based filtering for salient region detection. In: 2012 IEEE Conference on computer vision and pattern recognition, pp 733–740
Wang N, Gong X (2019) Adaptive fusion for RGB-d salient object detection. IEEE Access 7:55277–55284
Piao Y, Rong Z, Zhang M, Ren W, Lu H (2020) A2dele: Adaptive and attentive depth distiller for efficient rgb-d salient object detection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 9060–9069
Liu Z, Tang J, Xiang Q, Zhao P (2020) Salient object detection for RGB-d images by generative adversarial network. Multimed Tools Appl 79(35):25403–25425
Zhao X, Zhang L, Pang Y, Lu H, Zhang L (2020) A single stream network for robust and real-time RGB-D salient object detection. In: European conference on computer vision, pp 646–662
Chen H, Deng Y, Li Y, Hung T-Y, Lin G (2020) RGBD salient object detection via disentangled cross-modal fusion. IEEE Trans Image Process 29:8407–8416
Zhai Y, Fan D-P, Yang J, Borji A, Shao L, Han J, Wang L (2021) Bifurcated backbone strategy for rgb-d salient object detection. IEEE Trans Image Process 30:8727–8742
Li J, Ji W, Bi Q, Yan C, Zhang M, Piao Y, Lu H et al (2021) Joint semantic mining for weakly supervised RGB-d salient object detection. Adv Neural Inf Process Syst 34:11945–11959
Jiang B, Zhou Z, Wang X, Tang J, Luo B (2020) CmsalGAN: RGB-d salient object detection with cross-view generative adversarial networks. IEEE Trans Multimedia 23:1343–1353
Wu Y-H, Liu Y, Xu J, Bian J-W, Gu Y-C, Cheng M-M (2021) Mobilesal extremely efficient RGB-D salient object detection. IEEE Trans Pattern Anal Mach Intell, pp 1–1
Chen T, Hu X, Xiao J, Zhang G, Wang S (2022) CFIDNEt: cascaded feature interaction decoder for RGB-d salient object detection. Neural Comput & Applic, pp 1–17
Acknowledgements
This work was supported by the Opening Fund of Key Laboratory of Geological Survey and Evaluation of Ministry of Education (Grant No. GLAB2020 ZR06) and the Fundamental Research Funds for the Central Universities, and the National Natural Science Foundation of China under contract on 61603357.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of Interests
The authors declare that they have no conflict of interest.
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Zong, G., Wei, L., Guo, S. et al. A cascaded refined rgb-d salient object detection network based on the attention mechanism. Appl Intell 53, 13527–13548 (2023). https://doi.org/10.1007/s10489-022-04186-9
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10489-022-04186-9