Skip to main content
SpringerLink
Log in

WUSL–SOD: Joint weakly supervised, unsupervised and supervised learning for salient object detection

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Deep learning methods for salient object detection (SOD) have been studied actively and promisingly. However, it is still challenging for the studies with two aspects. The first one is a single type of label from the network to convey limit information, which leads to the poor generalization ability of the network. The second one is the difficulty to improve the accuracy and detect details of target. To address these challenges, we develop a novel approach via joint weakly supervised, unsupervised and supervised learning for SOD (WUSL–SOD), which differs from existing methods just based on ground-truth or other sparse labels. Specifically, to optimize the objective of the image, the unsupervised learning module (ULM) is designed to generate coarse saliency feature and suppress background noises via attention guiding mechanism. Then, we propose the weakly supervised learning module (WLM) based on scribbles for producing relatively accurate saliency feature. Note that this structure is used to enhance the details and remedy the deficiency of scribbles in WLM. For further refining information from the ULM and WLM, we propose a supervised learning module (SLM), which is not only applied to process and refine information from the ULM and WLM, but also enhance the image details and capture the entire target area. Furthermore, we also exchange information between the SLM and the WLM to obtain more accurate saliency maps. Extensive experiments on five datasets demonstrate that the proposed approach can effectively outperform the state-of-the-art approaches and achieve real-time.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

Data availability

The datasets generated during and/or analyzed during the current study are available from the corresponding author on reasonable request

References

  1. Borji A, Cheng MM, Jiang H, Li J (2015) Salient object detection: a benchmark. IEEE Transact Image process 24(12):5706–5722

    Article  MathSciNet  MATH  Google Scholar 

  2. Borji A, Itti L (2012) Exploiting local and global patch rarities for saliency detection. In: 2012 IEEE conference on computer vision and pattern recognition, pp. 478–485. IEEE

  3. Chen S, Tan X, Wang B, Hu X (2018) Reverse attention for salient object detection. In: proceedings of the european conference on computer vision (ECCV), pp. 234–250

  4. Chen Z, Xu Q, Cong R, Huang Q (2020) Global context-aware progressive aggregation network for salient object detection. In: proceedings of the AAAI conference on artificial intelligence, 34, pp. 10599–10606

  5. Cheng MM, Mitra NJ, Huang X, Torr PH, Hu SM (2014) Global contrast based salient region detection. IEEE Transact Pattern Anal Mach Intell 37(3):569–582

    Article  Google Scholar 

  6. Cheng MM, Zhang FL, Mitra NJ, Huang X, Hu SM (2010) Repfinder: finding approximately repeated scene elements for image editing. ACM Transact Graphics (TOG) 29(4):1–8

    Article  Google Scholar 

  7. Everingham M, Van Gool L, Williams CK, Winn J, Zisserman A (2010) The pascal visual object classes (voc) challenge. Int J Comput Vision 88(2):303–338

    Article  Google Scholar 

  8. Fan DP, Cheng MM, Liu Y, Li T, Borji A (2017) Structure-measure: A new way to evaluate foreground maps. In: Proceedings of the IEEE international conference on computer vision, pp. 4548–4557

  9. Fan DP, Gong C, Cao Y, Ren B, Cheng MM, Borji A (2018) Enhanced-alignment measure for binary foreground map evaluation. arXiv preprint arXiv:1805.10421

  10. Fang H, Zhang D, Zhang Y, Chen M, Li J, Hu Y, Cai D, He X (2021) Salient object ranking with position-preserved attention. In: Proceedings of the IEEE/CVF international conference on computer vision (ICCV), pp. 16331–16341

  11. Feng M, Lu H, Ding E (2019) Attentive feedback network for boundary-aware salient object detection. In: Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pp. 1623–1632

  12. Fu J, Liu J, Tian H, Li Y, Bao Y, Fang Z, Lu H (2019) Dual attention network for scene segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3146–3154

  13. Gao S, Zhang W, Wang Y, Guo Q, Zhang C, He Y, Zhang W (2022) Weakly-supervised salient object detection using point supervision. In: AAAI

  14. Gao Y, Wang M, Tao D, Ji R, Dai Q (2012) 3-d object retrieval and recognition with hypergraph analysis. IEEE Transact Image Process 21(9):4290–4303

    Article  MathSciNet  MATH  Google Scholar 

  15. Hou Q, Cheng MM, Hu X, Borji A, Tu Z, Torr PH (2017) Deeply supervised salient object detection with short connections. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3203–3212

  16. Huang Z, Wang X, Huang L, Huang C, Wei Y, Liu W (2019) Ccnet: Criss-cross attention for semantic segmentation. In: Proceedings of the IEEE/CVF international conference on computer vision, pp. 603–612

  17. Jiang P, Ling H, Yu J, Peng J (2013) Salient region detection by ufo: Uniqueness, focusness and objectness. In: Proceedings of the IEEE international conference on computer vision, pp. 1976–1983

  18. Lee S, Lee M, Lee J, Shim H (2021) Railroad is not a train: Saliency as pseudo-pixel supervision for weakly supervised semantic segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5495–5505

  19. Li G, Xie Y, Lin L (2018) Weakly supervised salient object detection using image labels. In: proceedings of the AAAI conference on artificial intelligence, 32

  20. Li G, Yu Y (2015)Visual saliency based on multiscale deep features. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5455–5463

  21. Li G, Yu Y (2015) Visual saliency based on multiscale deep features. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5455–5463

  22. Li G, Yu Y (2016) Visual saliency detection based on multiscale deep cnn features. IEEE Tansact Iage Pocess 25(11):5012–5024

    Article  MathSciNet  MATH  Google Scholar 

  23. Lin X, Wu Z, Chen G, Li G, Yu Y(2022) A causal debiasing framework for unsupervised salient object detection

  24. Liu JJ, Hou Q, Cheng MM (2020) Dynamic feature integration for simultaneous detection of salient object, edge, and skeleton. IEEE Transact Image Process 29:8652–8667

    Article  MATH  Google Scholar 

  25. Liu JJ, Hou Q, Cheng MM, Feng J, Jiang J (2019) A simple pooling-based design for real-time salient object detection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3917–3926

  26. Liu JJ, Liu ZA, Peng P, Cheng MM (2021) Rethinking the u-shape structure for salient object detection. IEEE Transact Image Process 30:9030–9042

    Article  Google Scholar 

  27. Liu N, Han J, Yang MH (2018) Picanet: Learning pixel-wise contextual attention for saliency detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3089–3098

  28. Liu N, Zhang N, Wan K, Shao L, Han J (2021) Visual saliency transformer. In: Proceedings of the IEEE/CVF international conference on computer vision, pp. 4722–4732

  29. Liu Y, Cheng MM, Zhang XY, Nie GY, Wang M (2022) DNA: Deeply supervised nonlinear aggregation for salient object detection. IEEE Transact Cybernet 52(7):6131–6142

    Article  Google Scholar 

  30. Liu Y, Wang P, Cao Y, Liang Z, Lau RW (2021) Weakly-supervised salient object detection with saliency bounding boxes. IEEE Transact Image Process 30:4423–4435

    Article  Google Scholar 

  31. Liu Y, Zhang Y, Liu S, Coleman S, Wang Z, Qiu F (2021) Salient object detection by aggregating contextual information. Pattern Recognit Lett

  32. Boosting broader receptive fields for salient object detection (2023) Ma, M., Xia, C., Xie, C., Chen, X., Li. J. IEEE Transact Image Process 32:1026–1038

    Google Scholar 

  33. Mohammadi S, Noori M, Bahri A, Majelan SG, Havaei M (2020) Cagnet: Content-aware guidance for salient object detection. Pattern Recognit 103:107303

    Article  Google Scholar 

  34. Mukherjee S, Valenzise G, Cheng I (2019) Potential of deep features for opinion-unaware, distortion-unaware, no-reference image quality assessment. In: International conference on smart multimedia, pp. 87–95. Springer

  35. Nguyen DT, Dax M, Mummadi CK, Ngo TPN, Nguyen THP, Lou Z, Brox T (2019) Deepusps: Deep robust unsupervised saliency prediction with self-supervision. arXiv preprint arXiv:1909.13055

  36. Pang Y, Zhao X, Zhang L, Lu H (2020) Multi-scale interactive network for salient object detection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9413–9422

  37. Piao Y, Ji W, Li J, Zhang M, Lu H (019) Depth-induced multi-scale recurrent attention network for saliency detection. In: Proceedings of the IEEE/CVF international conference on computer vision, pp. 7254–7263

  38. Qin X, Zhang Z, Huang C, Dehghan M, Zaiane OR, Jagersand M (2020) U2-net: Going deeper with nested u-structure for salient object detection. Pattern Recognit 106:107404

    Article  Google Scholar 

  39. Qin X, Zhang Z, Huang C, Gao C, Dehghan M, Jagersand M (2019) Basnet: Boundary-aware salient object detection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 7479–7489

  40. Qin Y, Lu H, Xu Y, Wang H (2015) Saliency detection via cellular automata. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 110–119

  41. Sun F, Li W (2019) Saliency guided deep network for weakly-supervised image segmentation. Pattern Recognit Lett

  42. Tang M, Djelouah A, Perazzi F, Boykov Y, Schroers C (2018) Normalized cut loss for weakly-supervised cnn segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1818–1827

  43. Tong N, Lu H, Ruan X, Yang MH (2015) Salient object detection via bootstrap learning. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1884–1892

  44. Wang L, Lu H, Ruan X, Yang MH (2015) Deep networks for saliency detection via local estimation and global search. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3183–3192

  45. Wang L, Lu H, Wang Y, Feng M, Wang D, Yin B, Ruan X (2017) Learning to detect salient objects with image-level supervision. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 136–145

  46. Wang T, Borji A, Zhang L, Zhang P, Lu H (2017) A stagewise refinement model for detecting salient objects in images. In: Proceedings of the IEEE international conference on computer vision, pp. 4019–4028

  47. Wang T, Zhang L, Wang S, Lu H, Yang G, Ruan X, Borji A (2018) Detect globally, refine locally: A novel approach to saliency detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3127–3135

  48. Wang W, Shen J, Dong X, Borji A (2018) Salient object detection driven by fixation prediction. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1711–1720

  49. Wang W, Zhao S, Shen J, Hoi SC, Borji A (2019) Salient object detection with pyramid attention and salient edges. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1448–1457

  50. Wang Z, Zhang Y, Liu Y, Liu S, Coleman S, Kerr D (2021) Mfc-net: Multi-feature fusion cross neural network for salient object detection. Image Vision Comput. 104243

  51. Wang Z, Zhang Y, Liu Y, Wang Z, Coleman S, Kerr D (2022) Tf-sod: a novel transformer framework for salient object detection. Neural Comput Appl 34(14):11789–11806

    Article  Google Scholar 

  52. Wei J, Wang S, Huang Q (2020) F\(^3\)net: Fusion, feedback and focus for salient object detection. In: Proceedings of the AAAI conference on artificial intelligence, 34, pp. 12321–12328

  53. Wei Y, Liang X, Chen Y, Shen X, Cheng MM, Feng J, Zhao Y, Yan S (2016) Stc: A simple to complex framework for weakly-supervised semantic segmentation. IEEE transactions on pattern analysis and machine intelligence 39(11):2314–2320

    Article  Google Scholar 

  54. Wei Y, Wen F, Zhu W, Sun J (2012) Geodesic saliency using background priors. In: European conference on computer vision, pp. 29–42. Springer

  55. Wu R, Feng M, Guan W, Wang D, Lu H, Ding E (2019) A mutual learning method for salient object detection with intertwined multi-supervision. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8150–8159

  56. Wu Z, Su L, Huang Q (2019) Cascaded partial decoder for fast and accurate salient object detection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3907–3916

  57. Wu Z, Su L, Huang Q (2019) Stacked cross refinement network for edge-aware salient object detection. In: Proceedings of the IEEE/CVF international conference on computer vision, pp. 7264–7273

  58. Yan Q, Xu L, Shi J, Jia J (2013) Hierarchical saliency detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1155–1162

  59. Yang C, Zhang L, Lu H, Ruan X, Yang MH (2013) Saliency detection via graph-based manifold ranking. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3166–3173

  60. Yu S, Zhang B, Xiao J, Lim EG (2020) Structure-consistent weakly supervised salient object detection with local saliency coherence. arXiv preprint arXiv:2012.04404

  61. Zeng Y, Feng M, Lu H, Yang G, Borji A (2018) An unsupervised game-theoretic approach to saliency detection. IEEE Transact Image Process 27(9):4545–4554

    Article  MathSciNet  MATH  Google Scholar 

  62. Zeng Y, Zhang P, Zhang J, Lin Z, Lu H (2019) Towards high-resolution salient object detection. In: Proceedings of the IEEE/CVF international conference on computer vision, pp. 7234–7243

  63. Zeng Y, Zhuge Y, Lu H, Zhang L, Qian M, Yu Y (2019) Multi-source weak supervision for saliency detection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6074–6083

  64. Zhai Y, Shah M (2006) Visual attention detection in video sequences using spatiotemporal cues. In: Proceedings of the 14th ACM international conference on Multimedia, pp. 815–824

  65. Zhang D, Han J, Zhang Y (2017) Supervision by fusion: Towards unsupervised learning of deep salient object detector. In: Proceedings of the IEEE international conference on computer vision, pp. 4048–4056

  66. Zhang D, Tian H, Han J (2021) Few-cost salient object detection with adversarial-paced learning. arXiv preprint arXiv:2104.01928

  67. Zhang J, Yu X, Li A, Song P, Liu B, Dai Y (2020) Weakly-supervised salient object detection via scribble annotations. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12546–12555

  68. Zhang J, Zhang T, Dai Y, Harandi M, Hartley R (2018) Deep unsupervised saliency detection: A multiple noisy labeling perspective. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 9029–9038

  69. Zhang L, Ai J, Jiang B, Lu H, Li X (2017) Saliency detection via absorbing markov chain with learnt transition probability. IEEE Transact Image Process 27(2):987–998

    Article  MathSciNet  MATH  Google Scholar 

  70. Zhang N, Deng S, Ye H, Zhang W, Chen H (2022) Robust triple extraction with cascade bidirectional capsule network. Expert Syst Appl 187:115806

    Article  Google Scholar 

  71. Zhang Q, Zhang L, Wang D, Shi Y, Lin J (2021) Global and local information aggregation network for edge-aware salient object detection. J Visual Commun Image Represent 81:103350

    Article  Google Scholar 

  72. Zhang X, Wang T, Qi J, Lu H, Wang G (2018) Progressive attention guided recurrent network for salient object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 714–722

  73. Zhao JX, Liu JJ, Fan DP, Cao Y, Yang J, Cheng MM (2019) Egnet: Edge guidance network for salient object detection. In: Proceedings of the IEEE/CVF international conference on computer vision, pp. 8779–8788

  74. Zhao R, Oyang W, Wang X (2016) Person re-identification by saliency learning. IEEE Transact Pattern Anal Mach Intell 39(2):356–370

    Article  Google Scholar 

  75. Zhao T, Wu X (2019) Pyramid feature attention network for saliency detection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3085–3094

  76. Zhao X, Pang Y, Zhang L, Lu H, Zhang L (2020) Suppress and balance: A simple gated network for salient object detection. In: European conference on computer vision, pp. 35–51. Springer

  77. Zhou H, Xie X, Lai JH, Chen Z, Yang L (2020) Interactive two-stream decoder for accurate and fast saliency detection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9141–9150

  78. Zhou T, Brown M, Snavely N, Lowe DG (2017) Unsupervised learning of depth and ego-motion from video. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1851–1858

  79. Zhou X, Fang H, Liu Z, Zheng B, Sun Y, Zhang J, Yan C (2021) Dense attention-guided cascaded network for salient object detection of strip steel surface defects. IEEE Transactions on Instrumentation and Measurement

  80. Zhu D, Dai L, Luo Y, Zhang G, Shao X, Itti L, Lu J (2018) Multi-scale adversarial feature learning for saliency detection. Symmetry 10(10):457

    Article  Google Scholar 

  81. Zhu Y, Zhao C, Guo H, Wang J, Zhao X, Lu H (2018) Attention couplenet: Fully convolutional attention coupling network for object detection. IEEE Transact Image Process 28(1):113–126

    Article  MathSciNet  Google Scholar 

  82. Zhuge M, Fan DP, Liu N, Zhang D, Xu D, Shao L (2022) Salient object detection via integrity learning. IEEE Transact Pattern Anal Mach Intell

Download references

Acknowledgements

This work was supported by National Natural Science Foundation of China (No. 61973066, 61471110), Foundation of Key Laboratory of Aerospace System Simulation(6142002200301), Foundation of Key Laboratory of Equipment Reliability(61420030302), Major Science and technology innovation engineering projects of Shandong Province(2019JZZY010128) and Distinguished Creative Talent Program of Liaoning Colleges and Universities (LR2019027).

Author information

Authors and Affiliations

  1. Faculty of Robot Science and Engineering, Northeastern University, Shenyang, China

    Yan Liu & Zhenyu Wang

  2. College of Information Science and Engineering, Northeastern University, Shenyang, China

    Yunzhou Zhang & Feng Qiu

  3. Beijing Simulation Center, Beijing, China

    Rong Ma

  4. School of Computing, Engineering and Intelligent Systems, Ulster University, Coleraine, Northern Ireland, UK

    Sonya Coleman & Dermot Kerr

Authors
  1. Yan Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yunzhou Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zhenyu Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Rong Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Feng Qiu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Sonya Coleman
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Dermot Kerr
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yunzhou Zhang.

Ethics declarations

Conflict of interest

No potential conflict of interest was reported by the authors.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, Y., Zhang, Y., Wang, Z. et al. WUSL–SOD: Joint weakly supervised, unsupervised and supervised learning for salient object detection. Neural Comput & Applic 35, 15837–15856 (2023). https://doi.org/10.1007/s00521-023-08545-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-023-08545-6

Keywords

Search

Navigation