Skip to main content
SpringerLink
Log in

SRFFNet: Self-refine, Fusion and Feedback for Salient Object Detection

  • Published:
Cognitive Computation Aims and scope Submit manuscript

Abstract

Many existing salient object detection models have achieved excellent results by fusing the progressive multi-layer features extracted by the backbone network. However, although the convolutional neural network can extract features from different levels, the computer obviously cannot distinguish the helpful information in the feature map. Feature fusion is usually achieved by adding or concatenating different levels of feature maps at the pixel level. However, these operations ignore the connection between multi-level features and global features. At the same time, the traditional U-shaped network structure can easily cause the salient map boundary to be blurred. In this paper, we proposed a Self-refine Fusion Feedback Network (SRFFNet) to solve these problems, which mainly consists of the self-refine module (SRM), feature fusion module (FFM), global optimization module (GOM) and feedback module (FM). Inspired by the cognitive process of biology, we designed a self-refine module based on human self-regulation ability. Similarly, we designed a feature fusion module based on biological diversity to extract diversity feature information. Furthermore, we also designed a feedback module based on biofeedback nerves. Particularly, SRM is used to realize the integration and optimization of the feature information. GOM is to obtain the feature map of global information, which is obtained after processing by the SRM module. FFM adaptively selects the feature information of two adjacent layers and global information for progressive fusion. FM can pass the feature map of the prediction result into the feature layer for the second stage to optimize the final salient map. In addition, we also proposed a weighted loss function to optimize the training loss to achieve better performance. Experimental results demonstrate that the proposed method achieves advanced performance with state-of-the-art methods. The SRFFNet can accurately segment the salient object area and provide clear boundaries and more detailed information.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Data Availability

The dataset ECSSD [3] that supports the findings of this study is available in http://www.cse.cuhk.edu.hk/leojia/projects/hsaliency/dataset.html for free. The dataset DUTS [34] that supports the findings of this study is available in http://saliencydetection.net/duts/ for free. The dataset DUT-OMRON [10] that supports the findings of this study is available in http://saliencydetection.net/dut-omron/ for free. The dataset PASCAL-S [33] that supports the findings of this study is available in http://cbi.gatech.edu/salobj/ for free. The dataset HKU-IS [11] that supports the findings of this study is available in https://i.cs.hku.hk/~gbli/deep_saliency for free.

References

  1. Wang W, Lai Q, Fu H, Shen J, Ling H, Yang R. Salient object detection in the deep learning era: An in-depth survey. IEEE Trans Patt Anal Machine Intell 2021.

  2. Cheng MM, Mitra NJ, Huang X, Torr PH, Hu SM. Global contrast based salient region detection. IEEE Trans Patt Anal Mach Intell. 2014;37(3):569–82.

    Article  Google Scholar 

  3. Yan Q, Xu L, Shi J, Jia J. Hierarchical saliency detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition; 2013;1155–62.

  4. Jiang Z, Davis LS. Submodular salient region detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; 2013;2043:5.

  5. Hou Q, Cheng MM, Hu X, Borji A, Tu Z, Torr PH. Deeply supervised salient object detection with short connections. In: Proceedings of the IEEE conference on computer vision and pattern recognition; 2017;3203–12.

  6. He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; 2016;770–8.

  7. Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. arXiv preprint http://arxiv.org/abs/1409.1556. 2014.

  8. Liu S, Huang D, et al. Receptive field block net for accurate and fast object detection. In: Proceedings of the European Conference on Computer Vision (ECCV); 2018;385–400.

  9. Itti L, Koch C, Niebur E. A model of saliency-based visual attention for rapid scene analysis. IEEE Trans Patt Anal Mach Intell. 1998;20(11):1254–9.

    Article  Google Scholar 

  10. Wang L, Lu H, Ruan X, Yang MH. Deep networks for saliency detection via local estimation and global search. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; 2015;3183–92.

  11. Li G, Yu Y. Visual saliency based on multiscale deep features. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2015;5455–63.

  12. Lee G, Tai YW, Kim J. Deep saliency with encoded low level distance map and high level features. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; 2016;660–8.

  13. Chen T, Lin L, Liu L, Luo X, Li X. Disc: Deep image saliency computing via progressive representation learning. IEEE Trans Neur Netw Learn Syst. 2016;27(6):1135–49.

    Article  MathSciNet  Google Scholar 

  14. Liu N, Han J. Dhsnet: Deep hierarchical saliency network for salient object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016;678–86.

  15. Deng Z, Hu X, Zhu L, Xu X, Qin J, Han G, et al. R3net: Recurrent residual refinement network for saliency detection. In: Proceedings of the 27th International Joint Conference on Artificial Intelligence. AAAI Press. 2018;684–90.

  16. Liu JJ, Hou Q, Cheng MM, Feng J, Jiang J. A simple pooling-based design for real-time salient object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019;3917–26.

  17. Chen T, Hu X, Xiao J, Zhang G, Wang S. BINet: Bidirectional interactive network for salient object detection. Neurocomputing. 2021;465:490–502.

    Article  Google Scholar 

  18. Qin X, Zhang Z, Huang C, Gao C, Dehghan M, Jagersand M. Basnet: Boundary-aware salient object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019;7479–89.

  19. Liu N, Han J, Yang MH. Picanet: Learning pixel-wise contextual attention for saliency detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018;3089–98.

  20. Chen Z, Xu Q, Cong R, Huang Q. Global context-aware progressive aggregation network for salient object detection. In: Proceedings of the AAAI Conference on Artificial Intelligence. 2020;(34);10599–606.

  21. Zhao JX, Liu JJ, Fan DP, Cao Y, Yang J, Cheng MM. EGNet: Edge guidance network for salient object detection. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019;8779–88.

  22. Zhao X, Pang Y, Zhang L, Lu H, Zhang L. Suppress and balance: A simple gated network for salient object detection. In: European Conference on Computer Vision. Springer. 2020;35–51.

  23. Feng M, Lu H, Ding E. Attentive feedback network for boundary-aware salient object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2019;1623-32.

  24. Chen T, Hu X, Xiao J, Zhang G. BPFINet: Boundary-aware progressive feature integration network for salient object detection. Neurocomputing. 2021;451:152–66.

    Article  Google Scholar 

  25. Pang Y, Zhao X, Zhang L, Lu H. Multi-scale interactive network for salient object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2020;9413–22.

  26. Zhang H, Lan X, Wan L, Yang C, Zhou X, Zheng N. RPRG: Toward real-time robotic perception reasoning and grasping with one multi-task convolutional neural network. 2018;1–7 arXiv preprint https://doi.org/10.48550/arXiv.1809.07081.

  27. Szegedy C, Ioffe S, Vanhoucke V, Alemi AA. Inception-v4, inception-resnet and the impact of residual connections on learning. In: Thirty-first AAAI Conference on Artificial Intelligence. 2017.

  28. Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z. Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016;2818–26.

  29. Yu F, Koltun V. Multi-scale context aggregation by dilated convolutions. arXiv preprint http://arxiv.org/abs/1511.07122arXiv:1511.07122. 2015.

  30. Krizhevsky A, Sutskever I, Hinton GE. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems. 2012;25:1097–105.

    Google Scholar 

  31. Wei J, Wang S, Huang Q. F\(^3\)Net: Fusion, Feedback and Focus for Salient Object Detection. In: Proceedings of the AAAI Conference on Artificial Intelligence. 2020(24);12321–8.

  32. Yang C, Zhang L, Lu H, Ruan X, Yang MH. Saliency detection via graph-based manifold ranking. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; 2013;3166–73.

  33. Li Y, Hou X, Koch C, Rehg JM, Yuille AL. The secrets of salient object segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; 2014;280–7.

  34. Wang L, Lu H, Wang Y, Feng M, Wang D, Yin B, et al. Learning to detect salient objects with image-level supervision. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017;136–45.

  35. Wu R, Feng M, Guan W, Wang D, Lu H, Ding E. A mutual learning method for salient object detection with intertwined multi-supervision. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019;8150–9.

  36. Perazzi F, Krähenbühl P, Pritch Y, Hornung A, Saliency filters: Contrast based filtering for salient region detection. In,. IEEE Conference on Computer Vision and Pattern Recognition. IEEE. 2012;733–40.

  37. Margolin R, Zelnik-Manor L, Tal A. How to evaluate foreground maps? In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; 2014;248–55.

  38. Fan DP, Gong C, Cao Y, Ren B, Cheng MM, Borji A. Enhanced-alignment measure for binary foreground map evaluation. arXiv preprint http://arxiv.org/abs/1805.10421arXiv:1805.10421. 2018.

  39. Achanta R, Hemami S, Estrada F, Susstrunk S, Frequency-tuned salient region detection. In,. IEEE conference on computer vision and pattern recognition. IEEE. 2009;1597–604.

  40. Bottou L. Large-scale machine learning with stochastic gradient descent. In: Proceedings of COMPSTAT’2010. Springer; 2010;177–86.

  41. Mohammadi S, Noori M, Bahri A, Majelan SG, Havaei M. CAGNet: Content-aware guidance for salient object detection. Pattern Recognition. 2020;103:107303.

    Article  Google Scholar 

  42. Liu JJ, Hou Q, Cheng MM. Dynamic feature integration for simultaneous detection of salient object, edge, and skeleton. IEEE Transactions on Image Processing. 2020;29:8652–67.

    Article  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Artificial Intelligence, Chongqing University of Technology, Chongqing, China

    Shuang Wu & Guangjian Zhang

Authors
  1. Shuang Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Guangjian Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Guangjian Zhang.

Ethics declarations

Ethical Approval

This article does not contain any studies that used human participants or animals.

Conflict of Interest

The authors declare no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wu, S., Zhang, G. SRFFNet: Self-refine, Fusion and Feedback for Salient Object Detection. Cogn Comput 15, 943–955 (2023). https://doi.org/10.1007/s12559-023-10130-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12559-023-10130-x

Keywords

Search

Navigation