Skip to main content
SpringerLink
Log in

A novel deep network and aggregation model for saliency detection

  • Original Article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Recent deep learning-based methods for saliency detection have proved the effectiveness of integrating features with different scales. They usually design various complex architectures of network, e.g., multiple networks, to explore the multi-scale information of images, which is expensive in computation and memory. Feature maps produced with different subsampling convolutional layers have different spatial resolutions; therefore, they can be used as the multi-scale features to reduce the costs. In this paper, by exploiting the in-network feature hierarchy of convolutional networks, we propose a novel multi-scale network for saliency detection (MSNSD) consisting of three modules, i.e., bottom-up feature extraction, top-down feature connection and multi-scale saliency prediction. Moreover, to further boost the performance of MSNSD, an input image-aware saliency aggregation method is proposed based on the ridge regression, which combines MSNSD with some well-performed handcrafted shallow models. Extensive experiments on several benchmarks show that the proposed MSNSD outperforms the state-of-the-art saliency methods with less computational and memory complexity. Meanwhile, our aggregation method for saliency detection is effective and efficient to combine deep and shallow models and make them complementary to each other.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  1. Yang, C., Pu, J., Dong, Y., Xie, G.-S., Si, Y., Liu, Z.: Scene classification-oriented saliency detection via the modularized prescription. Vis. Comput. 35, 1–16 (2018)

    Google Scholar 

  2. Zhou, X., Wang, Y., Zhu, Q., Xiao, C., Lu, X.: Ssg: superpixel segmentation and grabcut-based salient object segmentation. Vis. Comput. 35, 1–14 (2018)

    Google Scholar 

  3. Wei, Y., Liang, X., Chen, Y., Shen, X., Cheng, M.-M., Feng, J., Zhao, Y., Yan, S.: Stc: a simple to complex framework for weakly-supervised semantic segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 39(11), 2314–2320 (2016)

    Google Scholar 

  4. Guo, J., Ren, T., Huang, L., Liu, X., Cheng, M.-M., Wu, G.: Video salient object detection via cross-frame cellular automata. In: 2017 IEEE International Conference on Multimedia and Expo (ICME), pp. 325–330. IEEE (2017)

  5. Cheng, M.-M., Zhang, F.-L., Mitra, N.J., Huang, X., Hu, S.-M.: Repfinder: finding approximately repeated scene elements for image editing. In: ACM Transactions on Graphics (TOG), vol. 29, no. 4, p. 83. ACM (2010)

  6. Cheng, M.-M., Hou, Q.-B., Zhang, S.-H., Rosin, P.L.: Intelligent visual media processing: when graphics meets vision. J. Comput. Sci. Technol. 32(1), 110–121 (2017)

    Google Scholar 

  7. Goferman, S., Zelnik-Manor, L., Tal, A.: Context-aware saliency detection. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 2376–2383 (2010)

  8. Liu, T., Yuan, Z., Sun, J., Wang, J., Zheng, N., Tang, X., Shum, H.: Learning to detect a salient object. IEEE Trans. Pattern Anal. Mach. Intell. 33(2), 353–367 (2011)

    Google Scholar 

  9. Cheng, M., Zhang, G., Mitra, N.J., Huang, X., Hu, S.: Global contrast based salient region detection. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 409–416 (2011)

  10. Tong, N., Lu, H., Zhang, L., Ruan, X.: Saliency detection with multi-scale superpixels. IEEE Signal Process. Lett. 21(9), 1035–1039 (2014)

    Google Scholar 

  11. Lu, Y., Zhou, K., Wu, X., Gong, P.: A novel multi-graph framework for salient object detection. Vis. Comput. 35, 1–17 (2019)

    Google Scholar 

  12. Wang, B., Zhang, T., Wang, X.: Salient object detection based on Laplacian similarity metrics. Vis. Comput. 34(5), 645–658 (2018)

    Google Scholar 

  13. Kapoor, A., Biswas, K., Hanmandlu, M.: An evolutionary learning based fuzzy theoretic approach for salient object detection. Vis. Comput. 33(5), 665–685 (2017)

    Google Scholar 

  14. Yang, Z., Xiong, H.: Computing object-based saliency via locality-constrained linear coding and conditional random fields. Vis. Comput. 33(11), 1403–1413 (2017)

    Google Scholar 

  15. Yang, C., Pu, J., Dong, Y., Liu, Z., Liang, L., Wang, X.: Salient object detection in complex scenes via ds evidence theory based region classification. Vis. Comput. 33(11), 1415–1428 (2017)

    Google Scholar 

  16. Li, R., Cai, J., Zhang, H., Wang, T.: Aggregating complementary boundary contrast with smoothing for salient region detection. Vis. Comput. 33(9), 1155–1167 (2017)

    Google Scholar 

  17. Zhang, Q., Lin, J., Li, W., Shi, Y., Cao, G.: Salient object detection via compactness and objectness cues. Vis. Comput. 34(4), 473–489 (2018)

    Google Scholar 

  18. Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 3431–3440 (2015)

  19. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Advances in Neural Processing Systems, pp. 1106–1114 (2012)

  20. Wang, L., Ouyang, W., Wang, X., Lu, H.: Visual tracking with fully convolutional networks. In: Proceedings of IEEE International Conference on Computer Vision, pp. 3119–3127 (2015)

  21. Wang, L., Ouyang, W., Wang, X., Lu, H.: STCT: sequentially training convolutional networks for visual tracking. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 1373–1381 (2016)

  22. Wang, L., Lu, H. Ruan, X., Yang, M.: Deep networks for saliency detection via local estimation and global search. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 3183–3192 (2015)

  23. Li, G., Yu, Y.: Visual saliency based on multiscale deep features. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 5455–5463 (2015)

  24. Zhao, R., Ouyang, W., Li, H., Wang, X.: Saliency detection by multi-context deep learning. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 1265–1274 (2015)

  25. Wang, L., Wang, L., Lu, H., Zhang, P., Ruan, X.: Saliency detection with recurrent fully convolutional networks. In: Proceedings of European Conference on Computer Vision, pp. 825–841 (2016)

  26. Zhu, L., Hu, X., Fu, C.-W., Qin, J., Heng, P.-A.: Saliency-aware texture smoothing. In: IEEE Transactions on Visualization and Computer Graphics (2018)

  27. Li, X., Yang, F., Cheng, H., Liu, W., Shen, D.: Contour knowledge transfer for salient object detection. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 355–370 (2018)

  28. Chen, S., Tan, X., Wang, B., Hu, X.: Reverse attention for salient object detection. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 234–250 (2018)

  29. Wang, W., Shen, J., Shao, L., et al.: Correspondence driven saliency transfer. IEEE Trans. Image Process. 25(11), 5025–5034 (2016)

    MathSciNet  MATH  Google Scholar 

  30. Yan, Q., Xu, L., Shi, J., Jia, J.: Hierarchical saliency detection. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 1155–1162 (2013)

  31. Wang, L., Lu, H., Ruan, X., Yang, M.-H.: Deep networks for saliency detection via local estimation and global search. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3183–3192 (2015)

  32. Li, G., Xie, Y., Lin, L., Yu, Y.: Instance-level salient object segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2386–2395 (2017)

  33. Chen, X., Zheng, A., Li, J., Lu, F.: Look, perceive and segment: finding the salient objects in images via two-stream fixation-semantic cnns. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1050–1058 (2017)

  34. Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S.E., Fu, C., Berg, A.C.: SSD: single shot multibox detector. In: Proceedings of European Conference on Computer Vision, pp. 21–37 (2016)

  35. Lin, T., Dollár, P., Girshick, R.B., He, K., Hariharan, B., Belongie, S.J.: Feature pyramid networks for object detection. CoRR, vol. arXiv:1612.03144 (2016)

  36. Zhang, L., Dai, J., Lu, H., He, Y., Wang, G.: A bi-directional message passing model for salient object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1741–1750 (2018)

  37. Zhang, X., Wang, T., Qi, J., Lu, H., Wang, G.: Progressive attention guided recurrent network for salient object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 714–722 (2018)

  38. Jiang, H., Wang, J., Yuan, Z., Wu, Y., Zheng, N., Li, S.: Salient object detection: a discriminative regional feature integration approach. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 2083–2090 (2013)

  39. W, W., J, S.: Deep visual attention prediction. IEEE Trans. Image Process. 27(5), 2368–2378 (2017)

    MathSciNet  Google Scholar 

  40. W, W., J, S., Y, Y., et al.: Stereoscopic thumbnail creation via efficient stereo saliency detection. IEEE Trans. Vis. Comput. Graph. 23(8), 2014–2027 (2016)

    Google Scholar 

  41. W, W., J, S., L, S.: Video salient object detection via fully convolutional networks. IEEE Trans. Image Process. 27(1), 38–49 (2017)

    MathSciNet  Google Scholar 

  42. Wang, W., Shen, J., Guo, F., et al.: Revisiting video saliency: a large-scale benchmark and a new model. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4894–4903 (2018)

  43. W, W., J, S., R, Y., et al.: Saliency-aware video object segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 40(1), 20–33 (2017)

    Google Scholar 

  44. Borji, A., Cheng, M.-M., Jiang, H., Li, J.: Salient object detection: a benchmark. IEEE Trans. Image Process. 24(12), 5706–5722 (2015)

    MathSciNet  MATH  Google Scholar 

  45. Wand, W., Shen, J., Dong, X., et al.: Salient object detection driven by fixation prediction. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1711–1720 (2018)

  46. Itti, L., Koch, C., Niebur, E.: A model of saliency-based visual attention for rapid scene analysis. IEEE Trans. Pattern Anal. Mach. Intell. 20(11), 1254–1259 (1998)

    Google Scholar 

  47. Yang, C., Zhang, L., Lu, H., Ruan, X., Yang, M.: Saliency detection via graph-based manifold ranking. In: 2013 IEEE Conference on Computer Vision and Pattern Recognition, Portland, OR, USA, June 23–28, 2013, pp. 3166–3173 (2013)

  48. Li, X., Lu, H., Zhang, L., Ruan, X., Yang, M.: Saliency detection via dense and sparse reconstruction. In: Proceedings of IEEE International Conference on Computer Vision, pp. 2976–2983 (2013)

  49. Li, X., Zhao, L., Wei, L., Yang, M., Wu, F., Zhuang, Y., Ling, H., Wang, J.: Deepsaliency: multi-task deep neural network model for salient object detection. IEEE Trans. Image Process. 25(8), 3919–3930 (2016)

    MathSciNet  MATH  Google Scholar 

  50. Li, G., Yu, Y.: Deep contrast learning for salient object detection. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 478–487 (2016)

  51. Deng, Z., Hu, X., Zhu, L., Xu, X., Qin, J., Han, G., Heng, P.-A.: R3net: recurrent residual refinement network for saliency detection. In: Proceedings of the 27th International Joint Conference on Artificial Intelligence, pp. 684–690. AAAI Press (2018)

  52. Liu, N., Han, J.: Dhsnet: deep hierarchical saliency network for salient object detection. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 678–686 (2016)

  53. Hou, Q., Cheng, M.-M., Hu, X., Borji, A., Tu, Z., Torr, P.H.: Deeply supervised salient object detection with short connections. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3203–3212 (2017)

  54. Zhang, P., Wang, D., Lu, H., Wang, H., Ruan, X.: Amulet: aggregating multi-level convolutional features for salient object detection. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 202–211 (2017)

  55. Hu, X., Zhu, L., Qin, J., Fu, C.-W., Heng, P.-A.: Recurrently aggregating deep features for salient object detection. In: Thirty-Second AAAI Conference on Artificial Intelligence (2018)

  56. Liu, J.J., Hou, Q., Cheng, M.M., et al.: A simple pooling-based design for real-time salient object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

  57. Qin, X., Zhang, Z., Huang, C., et al.: Basnet: boundary-aware salient object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

  58. Hu, X., Fu, C.W., Zhu, L., et al.: Sac-net: spatial attenuation context for salient object detection. arXiv preprint arXiv:1903.10152 (2019)

  59. Wu, Z., Su, L., Huang, Q.: Cascaded partial decoder for fast and accurate salient object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

  60. Borji, A., Sihite, D.N., Itti, L.: Salient object detection: a benchmark. In: Computer Vision—ECCV 2012—12th European Conference on Computer Vision, Florence, Italy, October 7–13, 2012, Proceedings, Part II, pp. 414–429 (2012)

  61. Mai, L., Niu, Y., Liu, F.: Saliency aggregation: a data-driven approach. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 1131–1138 (2013)

  62. Tikhonov, A.N., Arsenin, V.Y.: Solution of ill-posed problems. Math. Comput. 32(144), 491–491 (1977)

    Google Scholar 

  63. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. CoRR, vol. arXiv:1409.1556 (2014)

  64. Goodfellow, I.J., Warde-Farley, D., Mirza, M., Courville, A.C., Bengio, Y.: Maxout networks. In: Proceedings of the 30th International Conference on Machine Learning, ICML 2013, Atlanta, GA, USA, 16–21 June 2013, pp. 1319–1327 (2013)

  65. González, R.C., Woods, R.E., Eddins, S.L.: Digital Image Processing Using MATLAB. Pearson-Prentice-Hall, Upper Saddle River (2004)

    Google Scholar 

  66. Li, Y., Hou, X., Koch, C., Rehg, J.M., Yuille, A.L.: The secrets of salient object segmentation. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 280–287 (2014)

  67. Achanta, R., Hemami, S.S., Estrada, F.J., Süsstrunk, S.: Frequency-tuned salient region detection. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 1597–1604 (2009)

  68. Cheng, M., Mitra, N.J., Huang, X., Torr, P.H.S., Hu, S.: Global contrast based salient region detection. IEEE Trans. Pattern Anal. Mach. Intell. 37(3), 569–582 (2015)

    Google Scholar 

  69. Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R.B., Guadarrama, S., Darrell, T.: Caffe: convolutional architecture for fast feature embedding. CoRR, vol. arXiv:1408.5093 (2014)

  70. Peng, H., Li, B., Ling, H., Hu, W., Xiong, W., Maybank, S.J.: Salient object detection via structured matrix decomposition. IEEE Trans. Pattern Anal. Mach. Intell. 39(4), 818–832 (2017)

    Google Scholar 

  71. Tong, N., Lu, H., Ruan, X., Yang, M.: Salient object detection via bootstrap learning. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 1884–1892 (2015)

  72. Jiang, B., Zhang, L., Lu, H., Yang, C., Yang, M.: Saliency detection via absorbing Markov chain. In: Proceedings of IEEE International Conference on Computer Vision, pp. 1665–1672 (2013)

  73. Zhu, W., Liang, S., Wei, Y., Sun, J.: Saliency optimization from robust background detection. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 2814–2821 (2014)

  74. Zhang, J., Sclaroff, S., Lin, Z.L., Shen, X., Price, B.L., Mech, R.: Minimum barrier salient object detection at 80 FPS. In: Proceedings of IEEE International Conference on Computer Vision, pp. 1404–1412 (2015)

  75. Qin, Y., Lu, H., Xu, Y., Wang, H.: Saliency detection via cellular automata. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 110–119 (2015)

  76. Lee, G., Tai, Y., Kim, J.: Deep saliency with encoded low level distance map and high level features. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 660–668 (2016)

Download references

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China (61871038, 61871039) and Beijing Natural Science Foundation (4182022).

Author information

Authors and Affiliations

  1. Beijing Key Laboratory of Information Service Engineering, Beijing Union University, Beijing, China

    Ye Liang & Hongzhe Liu

  2. College of Robotics, Beijing Union University, Beijing, China

    Nan Ma

Authors
  1. Ye Liang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hongzhe Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Nan Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ye Liang.

Ethics declarations

Conflict of interest

All authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liang, Y., Liu, H. & Ma, N. A novel deep network and aggregation model for saliency detection. Vis Comput 36, 1883–1895 (2020). https://doi.org/10.1007/s00371-019-01781-9

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-019-01781-9

Keywords

Search

Navigation