Skip to main content
SpringerLink
Log in

A novel dynamic graph evolution network for salient object detection

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

The advanced deep convolution neural networks (CNNs) based salient object detection (SOD) models still suffer from the coarse object edge. The traditional graph-based SOD models can preserve the object edge well benefitting from the superpixels, but they perform weaker in highlighting the whole object compared to recent deep learning models. To tackle this problem, we attempt to find a new way to address this issue under the framework of graph convolution networks (GCNs). Specifically, we first model the image as a set of superpixels and construct the graph structure by connecting the k nearest neighbors for each node. For the connected nodes, rather than only leveraging on the predefined edges, we propose a multi-relations edge convolution operation, expecting to learn multiple implicit relations in the pair-wise nodes and aggregate the information from their neighbors relying on the learned edges. Then, a channel-wise attention operation is also proposed to boost the intra-node massage propagation across channels within the same node. In order to optimize the graph structure from layer to layer, we learn a new metric to re-measure the distance between any pair of nodes, and the graph structure evolves dynamically by recomputing the k nearest neighbors in the saliency feature space at different layers. Finally, a residual structure is applied to enable our graph network to go as deep as CNNs models. The graph nodes (superpixels) inherently belonging to the same class will be ideally clustered together in the learned embedding space. Experiments show that this work is a good practice for designing GCNs for image SOD and achieves the comparable performance with the recent state-of-the-art deep CNNs-based models.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Borji A, Cheng MM, Jiang H, Li J (2015) Salient object detection: A benchmark. IEEE Trans Image Process 24(12):5706–5722

    Article  MathSciNet  Google Scholar 

  2. Cong R, Lei J, Fu H, Cheng MM, Lin W, Huang Q (2018) Review of visual saliency detection with comprehensive information. IEEE Transactions on Circuits and Systems for Video Technology

  3. Wang W, Lai Q, Fu H, Shen J, Ling H (2019) Salient object detection in the deep learning era An in-depth survey. arXiv:1904.09146

  4. Ren Z, Gao S, Chia LT, Tsang WH (2014) Region-based saliency detection and its application in object recognition. IEEE Trans Circ Syst Video Technol 24(5):769–779

    Article  Google Scholar 

  5. Feng W, Han R, Guo Q, Zhu J, Wang S (2019) Dynamic saliency-aware regularization for correlation filter-based object tracking. IEEE Trans Image Process 28(7):3232–3245

    Article  MathSciNet  Google Scholar 

  6. Fang Y, Chen Z, Lin W, Lin CW (2012) Saliency detection in the compressed domain for adaptive image retargeting. IEEE Trans Image Process Public IEEE Signal Process Soc 21(9):3888–3901

    Article  MathSciNet  Google Scholar 

  7. Chen Z, Lin J, Liao N, Chen CW (2017) Full reference quality assessment for image retargeting based on natural scene statistics modeling and bi-directional saliency similarity. IEEE Trans Image Process 26(11):5138–5148

    Article  MathSciNet  Google Scholar 

  8. Ramanishka V, Das A, Zhang J, Saenko K (2017) Top-down visual saliency guided by captions. In: 2017 IEEE conference on computer vision and pattern recognition (CVPR), pp 3135–3144

  9. Zhou L, Zhang Y, Jiang Y, Zhang T, Fan W (2020) Re-caption: Saliency-enhanced image captioning through two-phase learning. IEEE Trans Image Process 29:694–709

    Article  MathSciNet  Google Scholar 

  10. Liu N, Han J (2016) Dhsnet: Deep hierarchical saliency network for salient object detection. In: Computer vision and pattern recognition, pp 678–686

  11. Wang S, Wang M, Yang S, Zhang K (2018) Salient region detection via discriminative dictionary learning and joint bayesian inference. IEEE Trans Circ Syst Video Technol 28(5):1116– 1129

    Article  Google Scholar 

  12. Li G, Yu Y (2018) Contrast-oriented deep neural networks for salient object detection. IEEE Transactions on Neural Networks and Learning Systems PP(99)

  13. Zhang P, Liu W, Lu H, Shen C (2019) Salient object detection with lossless feature reflection and weighted structural loss. IEEE Trans Image Process 28(6):3048–3060

    Article  MathSciNet  Google Scholar 

  14. Feng M, Lu H, Ding E (2019) Attentive feedback network for boundary-aware salient object detection. In: CVPR

  15. Li X, Yang F, Cheng H, Liu W, Shen D (2018) Contour knowledge transfer for salient object detection

  16. Wang W, Zhao S, Shen J, Hoi SCH, Borji A (2019) Salient object detection with pyramid attention and salient edges. In: 2019 IEEE/CVF conference on computer vision and pattern recognition (CVPR), pp 1448–1457

  17. Zhao J-X, Liu J-J, Fan D-P, Cao Y, Yang J, Cheng M-M (2019) Egnet:edge guidance network for salient object detection. In: IEEE international conference on computer vision

  18. Tu Z, Ma Y, Li C, Li C, Tang J, Luo B (2020) Edge-guided non-local fully convolutional network for salient object detection. In: IEEE transactions on circuits and systems for video technology, pp 1–1

  19. Mohammadi S, Noori M, Bahri A, Majelan SG, Cagnet MH (2020) Content-aware guidance for salient object detection. Pattern Recogn 103:107303

    Article  Google Scholar 

  20. Yi L, Han J, Zhang Q, Wang L (2018) Salient object detection via two-stage graphs. IEEE Trans Circ Syst Video Technol PP(99):1–1

    Google Scholar 

  21. Wang W, Shen J, Yang R, Porikli F (2018) Saliency-aware video object segmentation. IEEE Trans Pattern Anal Machine Intell 40(1):20–33

    Article  Google Scholar 

  22. Deng C, Yang X, Nie F, Tao D (2019) Saliency detection via a multiple self-weighted graph-based manifold ranking. IEEE Transactions on Multimedia 1–1

  23. Xu M, Liu B, Fu P, Li J, Hu YH (2019) Video saliency detection via graph clustering with motion energy and spatiotemporal objectness. IEEE Trans Multimed 21(11):2790–2805

    Article  Google Scholar 

  24. Xu M, Liu B, Fu P, Li J, Hu YH, Feng S (2019) Video salient object detection via robust seeds extraction and multi-graphs manifold propagation. IEEE Trans Circ Syst Video Technol 1–1

  25. Shuman DI, Narang SK, Frossard P, Ortega A, Vandergheynst P (2013) The emerging field of signal processing on graphs: Extending high-dimensional data analysis to networks and other irregular domains. IEEE Signal Proc Mag 30(3):83–98

    Article  Google Scholar 

  26. Bruna J, Zaremba W, Szlam A, LeCun Y (2013) Spectral networks and locally connected networks on graphs. arXiv:1312.6203

  27. Defferrard M, Bresson X, Vandergheynst P (2016) Convolutional neural networks on graphs with fast localized spectral filtering. In: Advances in neural information processing systems, pp 3844–3852

  28. Kipf TN, Welling M (2017) Semi-supervised classification with graph convolutional networks. In: International conference on learning representations (ICLR)

  29. Li G, Yu YZ (2015) Visual saliency based on multiscale deep features. In: Computer vision and pattern recognition, pp 5455–5463

  30. Zhao R, Ouyang W, Li H, Wang X (2015) Saliency detection by multi-context deep learning. In: Computer vision and pattern recognition, pp 1265–1274

  31. Wang W, Shen J, Cheng M, Shao L (2019) An iterative and cooperative top-down and bottom-up inference network for salient object detection. In: In 2019 IEEE/CVF conference on computer vision and pattern recognition (CVPR), pp 5961–5970

  32. Huang M, Liu Z, Ye L, Zhou X, Wang Y (2019) Saliency detection via multi-level integration and multi-scale fusion neural networks. Neurocomputing 364:310–321

    Article  Google Scholar 

  33. Qi J, Lu H, Wang G, Zhang X, Wang T (2018) Progressive attention guided recurrent network for salient object detection. In: CVPR

  34. Zhao T, Wu X (2019) Pyramid feature attention network for saliency detection. In: IEEE conference on computer vision and pattern recognition (CVPR)

  35. Chen S, Wang B, Tan X, Hu X (2018) Embedding attention and residual network for accurate salient object detection. IEEE Transactions on Cybernetics 1–13

  36. Wang W, Shen J, Dong X, Borji A, Yang R (2019) Inferring salient objects from human fixations. IEEE Transactions on Pattern Analysis and Machine Intelligence 1–1

  37. Yang C, Zhang L, Lu H, Ruan X, Yang M-H (2013) Saliency detection via graph-based manifold ranking. In: The IEEE conference on computer vision and pattern recognition (CVPR)

  38. Wang Q, Zheng W, Piramuthu R (2016) Grab: Visual saliency via novel graph model and background priors. In: The IEEE conference on computer vision and pattern recognition (CVPR)

  39. Wang Z, Xiang D, Hou S, Wu F (2017) Background-driven salient object detection. IEEE Trans Multimed 19(4):750–762

    Article  Google Scholar 

  40. Zhang L, Ai J, Jiang B, Lu H, Li X (2018) Saliency detection via absorbing markov chain with learnt transition probability. IEEE Trans Image Process 27(2):987–998

    Article  MathSciNet  Google Scholar 

  41. Ye L, Liu Z, Li L, Shen L, Bai C, Wang Y (2017) Salient object segmentation via effective integration of saliency and objectness. IEEE Trans multimed PP(99):1–1

    Google Scholar 

  42. Hamilton W, Ying Z, Leskovec J (2017) Inductive representation learning on large graphs. In: Advances in neural information processing systems, pp 1024–1034

  43. Veličković P, Cucurull G, Casanova A, Romero A, lio P, Bengio Y (2017) Graph attention networks. arXiv:1710.10903

  44. Xu K, Hu W, Leskovec J, Jegelka S (2018) How powerful are graph neural networks? arXiv:1810.00826

  45. Wang Y, Sun Y, Liu Z, Sarma SE, Bronstein MM, Solomon JM (2019) Dynamic graph cnn for learning on point clouds. ACM Trans Graph (TOG) 38(5):1–12

    Article  Google Scholar 

  46. Knyazev B, Lin X, Amer MR, Taylor GW (2019) Image classification with hierarchical multigraph networks. arXiv:1907.09000

  47. Wang W, Lu X, Shen J, Crandall DJ, Shao L (2019) Zero-shot video object segmentation via attentive graph neural networks. In: The IEEE international conference on computer vision (ICCV)

  48. Bo J, Jiang X, Tang J, Luo B, Shilei H (2019) Multiple graph convolutional networks for co-saliency detection. In: IEEE international conference on multimedia and expo (ICME). IEEE, p 2019

  49. Wang B, Liu W, Han G, He S (2020) Learning long-term structural dependencies for video salient object detection. IEEE Trans Image Process 29:9017–9031

    Article  Google Scholar 

  50. Achanta R, Süsstrunk S (2017) Superpixels and polygons using simple non-iterative clustering. In: 2017 IEEE conference on computer vision and pattern recognition (CVPR), pp 4895–4904

  51. Zhao H, Shi J, Qi X, Wang X, Jia J (2017) Pyramid scene parsing network. In: CVPR

  52. Zhao H, Zhang Y, Liu S, Shi J, Loy CC, Lin D, Jia J (2018) PSANet: Point-wise spatial attention network for scene parsing. In: ECCV

  53. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: 2016 IEEE conference on computer vision and pattern recognition (CVPR), pp 770–778

  54. Huang G, Liu Z, Maaten LVD, Weinberger KQ (2017) Densely connected convolutional networks. In: 2017 IEEE conference on computer vision and pattern recognition (CVPR), pp 2261–2269

  55. Wang L, Lu H, Wang Y, Feng M, Wang D, Yin B, Ruan X (2017) Learning to detect salient objects with image-level supervision. In: CVPR

  56. Yan Q, Xu L, Shi J, Jia J (2013) Hierarchical saliency detection. In: 2013 IEEE conference on computer vision and pattern recognition, pp 1155–1162

  57. Cheng M, Mitra NJ, Huang X, Torr PHS, Hu S (2015) Global contrast based salient region detection. IEEE Trans Pattern Anal Machine Intell 37(3):569–582

    Article  Google Scholar 

  58. Li Y, Hou X, Koch C, Rehg JM, Yuille AL (2014) The secrets of salient object segmentation. In: 2014 IEEE conference on computer vision and pattern recognition, pp 280–287

  59. Li G, Yu Y (2016) Visual saliency detection based on multiscale deep cnn features. IEEE Trans Image Process 25(11):5012– 5024

    Article  MathSciNet  Google Scholar 

  60. Wang T, Zhang L, Wang S, Lu H, Yang G, Ruan X, Borji A (2018) Detect globally, refine locally: A novel approach to saliency detection. In: 2018 IEEE/CVF conference on computer vision and pattern recognition, pp 3127–3135

  61. Achanta R, Hemami S, Estrada F, Susstrunk S (2009) Frequency-tuned salient region detection. In: Computer vision and pattern recognition, 2009. CVPR 2009. IEEE conference on, pp 1597–1604

  62. Fan D-P, Cheng M-M, Liu Y, Li T, Borji A (2017) Structure-measure: A new way to evaluate foreground maps. In: ICCV

  63. Fey M, Lenssen JE (2019) Fast graph representation learning with PyTorch Geometric. In: ICLR workshop on representation learning on graphs and manifolds

  64. Wu Y, Liu Z, Zhou X (2020) Saliency detection using adversarial learning networks. J Vis Commun Image Represent 67:102761

    Article  Google Scholar 

  65. Zhang P, Wang D, Lu H, Wang H, Ruan X (2017) Amulet: Aggregating multi-level convolutional features for salient object detection. In: 2017 IEEE international conference on computer vision (ICCV), pp 202–211

  66. Zhang P, Wang D, Lu H, Wang H, Yin B (2017) Learning uncertain convolutional features for accurate saliency detection. In: 2017 IEEE international conference on computer vision (ICCV), pp 212–221

  67. Li G, Yu Y (2018) Contrast-oriented deep neural networks for salient object detection. IEEE Trans Neural Netw LearnSyst 29(12):6038–6051

    Article  Google Scholar 

  68. Wang L, Wang L, Lu H, Zhang P, Ruan X (2019) Salient object detection with recurrent fully convolutional networks. IEEE Trans Pattern Anal Machine Intell 41(7):1734–1746

    Article  Google Scholar 

  69. Li X, Zhao L, Wei L, Yang M, Wu F, Zhuang Y, Ling H, Wang J (2016) Deepsaliency: Multi-task deep neural network model for salient object detection. IEEE Trans Image Process 25(8):3919–3930

    Article  MathSciNet  Google Scholar 

  70. Aytekin Ç, Ozan EC, Kiranyaz S, Gabbouj M (2015) Visual saliency by extended quantum cuts. In: 2015 IEEE international conference on image processing (ICIP), pp 1692–1696

  71. Jiang H, Wang J, Yuan Z, Wu Y, Zheng N, Li S (2013) Salient object detection: A discriminative regional feature integration approach. In: 2013 IEEE conference on computer vision and pattern recognition, pp 2083–2090

  72. Li X, Lu H, Zhang L, Ruan X, Yang M (2013) Saliency detection via dense and sparse reconstruction. In: 2013 IEEE international conference on computer vision, pp 2976–2983

  73. Jiang B, Zhang L, Lu H, Yang C, Yang M (2013) Saliency detection via absorbing markov chain. In: 2013 IEEE international conference on computer vision, pp 1665–1672

  74. Zhu W, Liang S, Wei Y, Sun J (2014) Saliency optimization from robust background detection. In: 2014 IEEE conference on computer vision and pattern recognition, pp 2814–2821

  75. Liu Z, Zou W, Le Meur O (2014) Saliency tree A novel saliency detection framework. IEEE Trans Image Process 23(5):1937–1952

    Article  MathSciNet  Google Scholar 

Download references

Acknowledgements

This work was supported by the National Natural Science Foundation of China (NSFC) under Grants 61671170, and also supported by the Equipment Pre-research Field Fund (JZX7Y20190243001201). Finally, the authors would like to appreciate the editors and reviewers for their valuable comments and constructive suggestions.

Author information

Authors and Affiliations

  1. Department of Measurement and Control Engineering, School of Electronics and Information Engineering, Harbin Institute of Technology, Harbin, 150001, China

    Mingzhu Xu, Ping Fu, Bing Liu, Hongtao Yin & Junbao Li

Authors
  1. Mingzhu Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ping Fu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Bing Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hongtao Yin
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Junbao Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Bing Liu.

Ethics declarations

Conflict of Interests

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Xu, M., Fu, P., Liu, B. et al. A novel dynamic graph evolution network for salient object detection. Appl Intell 52, 2854–2871 (2022). https://doi.org/10.1007/s10489-021-02479-z

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-021-02479-z

Keywords

Search

Navigation