Skip to main content
SpringerLink
Log in

\(\hbox {C}^{2}\)Net: a complementary co-saliency detection network

  • Original Article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

The main purpose of image Co-saliency detection is to detect objects with similar properties or prospects in a group of images. In this paper, we propose a novel Converged Complementary Co-saliency detection framework in an end-to-end manner for Co-saliency detection. Specifically, considering the consistency and the difference between the features of the intra-image and the inter-image, our network contains two parts [forward information transfer module and complementary information enhancement module (CIEM)]. Each part is a double-branch network with short connections to avoid incomplete information transmission, including single-saliency detection branch (SSDB) and Co-saliency detection branch (CSDB). In FITM, the SSDB and CSDB extract the corresponding features, respectively. In order to produce more accurate results, we feed the SSDB and CSDB into the holistic attention and perform further feature extraction. In CIEM, we extract the useful information of the non-salient area. At the end of the network, we fuse the FITM and CIEM to generate the final Co-saliency map. Furthermore, we rearranged a new data set based on the existing public data set. Compared with other Co-saliency methods, the experimental results show that our proposed model achieves state-of-the-art performance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Achanta, R., Hemami, S., Estrada, F., Süsstrunk, S.: Frequency-tuned salient region detection. In: IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1597–1604 (2009)

  2. Arbelaez, P., Maire, M., Fowlkes, C., Malik, J.: Contour detection and hierarchical image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI) 33(5), 898–916 (2010)

    Article  Google Scholar 

  3. Batra, D., Kowdle, A., Parikh, D., Luo, J., Chen, T.: icoseg: Interactive co-segmentation with intelligent scribble guidance. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3169–3176. IEEE (2010)

  4. Cao, X., Tao, Z., Zhang, B., Fu, H., Feng, W.: Self-adaptively weighted co-saliency detection via rank constraint. IEEE Trans. Image Process. 23(9), 4175–4186 (2014)

    MathSciNet  MATH  Google Scholar 

  5. Chang, K.Y., Liu, T.L., Lai, S.H.: From co-saliency to co-segmentation: An efficient and fully unsupervised energy minimization model. In: The 24th IEEE Conference on Computer Vision and Pattern Recognition, (CVPR), pp. 22–25 (2011)

  6. Chen, H.T.: Preattentive co-saliency detection. In: 2010 IEEE International Conference on Image Processing (ICIP), pp. 1117–1120. IEEE (2010)

  7. Chen, S., Tan, X., Wang, B., Hu, X.: Reverse attention for salient object detection. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 234–250 (2018)

  8. Cheng, M.M., Mitra, N.J., Huang, X., Torr, P.H., Hu, S.M.: Global contrast based salient region detection. IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI) 37(3), 569–582 (2014)

    Article  Google Scholar 

  9. Cong, R., Lei, J., Fu, H., Huang, Q., Cao, X., Ling, N.: Hscs: Hierarchical sparsity based co-saliency detection for rgbd images. IEEE Trans. Multimed. 21(7), 1660–1667 (2018)

    Article  Google Scholar 

  10. Cornia, M., Baraldi, L., Serra, G., Cucchiara, R.: A deep multi-level network for saliency prediction. In: 2016 23rd International Conference on Pattern Recognition (ICPR), pp. 3488–3493. IEEE (2016)

  11. Dong, X., Shen, J., Wu, D., Guo, K., Jin, X., Porikli, F.: Quadruplet network with one-shot learning for fast visual object tracking. IEEE Trans. Image Process. (TIP) 28(7), 3516–3527 (2019)

    Article  MathSciNet  Google Scholar 

  12. Du, S., Chen, S.: Detecting co-salient objects in large image sets. IEEE Signal Process. Lett. (SPL) 22(2), 145–148 (2014)

    Article  Google Scholar 

  13. Fan, D.P., Cheng, M.M., Liu, J.J., Gao, S.H., Hou, Q., Borji, A.: Salient objects in clutter: Bringing salient object detection to the foreground. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 186–202 (2018)

  14. Fan, D.P., Cheng, M.M., Liu, Y., Li, T., Borji, A.: Structure-measure: a new way to evaluate foreground maps. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 4548–4557 (2017)

  15. Fan, D.P., Gong, C., Cao, Y., Ren, B., Cheng, M.M., Borji, A.: Enhanced-alignment measure for binary foreground map evaluation. In: International Joint Conference on Artificial Intelligence (IJCAI), pp. 698–704 (2018)

  16. Fan, D.P., Lin, Z., Zhao, J.X., Liu, Y., Zhang, Z., Hou, Q., Zhu, M., Cheng, M.M.: Rethinking rgb-d salient object detection: Models, datasets, and large-scale benchmarks. arXiv preprint arXiv:1907.06781 (2019)

  17. Fan, D.P., Wang, W., Cheng, M.M., Shen, J.: Shifting more attention to video salient object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8554–8564 (2019)

  18. Fu, H., Cao, X., Tu, Z.: Cluster-based co-saliency detection. IEEE Trans. Image Process. (TIP) 22(10), 3766–3778 (2013)

    Article  MathSciNet  Google Scholar 

  19. Fu, H., Dong, X., Bao, Z., Lin, S.: Object-based multiple foreground video co-segmentation. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2014)

  20. Fu, H., Dong, X., Lin, S., Jiang, L.: Object-based RGBD image co-segmentation with mutex constraint. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015)

  21. Han, J., Cheng, G., Li, Z., Zhang, D.: A unified metric learning-based framework for co-saliency detection. IEEE Trans. Circuits Syst. Video Technol. (TCSVT) 28(10), 2473–2483 (2017)

    Article  Google Scholar 

  22. Itti, L., Koch, C., Niebur, E.: A model of saliency-based visual attention for rapid scene analysis. IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI) 11, 1254–1259 (1998)

    Article  Google Scholar 

  23. Jacobs, D.E., Goldman, D.B., Shechtman, E.: Cosaliency: Where people look when comparing images. In: Proceedings of the 23nd annual ACM symposium on User interface software and technology, pp. 219–228. ACM (2010)

  24. Lee, G., Tai, Y.W., Kim, J.: Deep saliency with encoded low level distance map and high level features. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 660–668 (2016)

  25. Li, G., Yu, Y.: Visual saliency based on multiscale deep features. In: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (CVPR), pp. 5455–5463 (2015)

  26. Li, H., Ngan, K.N.: A co-saliency model of image pairs. IEEE Trans. Image Process. (TIP) 20(12), 3365–3375 (2011)

    Article  MathSciNet  Google Scholar 

  27. Li, M., Dong, S., Zhang, K., Gao, Z., Wu, X., Zhang, H., Yang, G., Li, S.: Deep learning intra-image and inter-images features for co-saliency detection. In: BMVC, p. 291 (2018)

  28. Li, W., Zhu, X., Gong, S.: Harmonious attention network for person re-identification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2285–2294 (2018)

  29. Li, Y., Fu, K., Liu, Z., Yang, J.: Efficient saliency-model-guided visual co-saliency detection. IEEE Signal Process. Lett. (SPL) 22(5), 588–592 (2014)

    Article  Google Scholar 

  30. Liu, N., Han, J.: Dhsnet: Deep hierarchical saliency network for salient object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 678–686 (2016)

  31. Liu, N., Han, J., Yang, M.H.: Picanet: Learning pixel-wise contextual attention for saliency detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3089–3098 (2018)

  32. Liu, Z., Zou, W., Li, L., Shen, L., Le Meur, O.: Co-saliency detection based on hierarchical segmentation. IEEE Signal Process. Lett. (SPL) 21(1), 88–92 (2013)

    Article  Google Scholar 

  33. Lowe, D.G., et al.: Object recognition from local scale-invariant features. Proc. IEEE Int. Conf. Comput. Vis. (ICCV) 99, 1150–1157 (1999)

    Google Scholar 

  34. Meng, F., Li, H., Liu, G.: A new co-saliency model via pairwise constraint graph matching. In: 2012 International Symposium on Intelligent Signal Processing and Communications Systems, pp. 781–786. IEEE (2012)

  35. Papushoy, A., Bors, A.G.: Image retrieval based on query by saliency content. Digital Signal Process. 36, 156–173 (2015)

    Article  MathSciNet  Google Scholar 

  36. Prest, A., Leistner, C., Civera, J., Schmid, C., Ferrari, V.: Learning object class detectors from weakly annotated video. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3282–3289. IEEE (2012)

  37. Quan, H., Feng, S., Chen, B.: Two birds with one stone: a unified approach to saliency and co-saliency detection via multi-instance learning. IEEE Access 5, 23519–23531 (2017)

    Article  Google Scholar 

  38. Ronneberger, O., Fischer, P., Brox, T.n.: Convolutional networks for biomedical image segmentation. In: Paper Presented at: International Conference on Medical Image Computing and Computer-Assisted Intervention2015 (2015)

  39. Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al.: Imagenet large scale visual recognition challenge. Int. J. Comput. Vis. (IJCV) 115(3), 211–252 (2015)

    Article  MathSciNet  Google Scholar 

  40. Wang, L., Lu, H., Ruan, X., Yang, M.H.: Deep networks for saliency detection via local estimation and global search. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3183–3192 (2015)

  41. Wang, W., Shen, J.: Deep visual attention prediction. IEEE Trans. Image Process. (TIP) 27(5), 2368–2378 (2017)

    Article  MathSciNet  Google Scholar 

  42. Wang, W., Shen, J., Dong, X., Borji, A., Yang, R.: Inferring salient objects from human fixations. In: IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) (2019)

  43. Wang, W., Shen, J., Ling, H.: A deep network solution for attention and aesthetics aware photo cropping. IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI) 41(7), 1531–1544 (2018)

    Article  Google Scholar 

  44. Wang, W., Shen, J., Shao, L., Porikli, F.: Correspondence driven saliency transfer. IEEE Trans. Image Process. (TIP) 25(11), 5025–5034 (2016)

    Article  MathSciNet  Google Scholar 

  45. Wang, W., Shen, J., Sun, H., Shao, L.: Video co-saliency guided co-segmentation. IEEE Trans. Circuits Syst. Video Technol. (TCSVT) 28(8), 1727–1736 (2017)

    Article  Google Scholar 

  46. Wang, W., Shen, J., Xie, J., Cheng, M.M., Ling, H., Borji, A.: Revisiting video saliency prediction in the deep learning era. IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI) (2019)

  47. Wang, W., Shen, J., Yang, R., Porikli, F.: Saliency-aware video object segmentation. IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI) 40(1), 20–33 (2017)

    Article  Google Scholar 

  48. Wei, L., Zhao, S., Bourahla, O.E.F., Li, X., Wu, F.: Group-wise deep co-saliency detection. arXiv preprint arXiv:1707.07381 (2017)

  49. Winn, J.M., Criminisi, A., Minka, T.P.: Object categorization by learned universal visual dictionary. In: 10th IEEE International Conference on Computer Vision (ICCV 2005), Beijing, China (2005)

  50. Wu, Z., Su, L., Huang, Q.: Cascaded partial decoder for fast and accurate salient object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3907–3916 (2019)

  51. Yang, L., Geng, B., Cai, Y., Hanjalic, A., Hua, X.S.: Object retrieval using visual query context. IEEE Trans. Multimed. 13(6), 1295–1307 (2011)

    Article  Google Scholar 

  52. Ye, L., Liu, Z., Li, J., Zhao, W.L., Shen, L.: Co-saliency detection via co-salient object discovery and recovery. IEEE Signal Process. Lett. (SPL) 22(11), 2073–2077 (2015)

    Article  Google Scholar 

  53. Zhang, D., Han, J., Han, J., Shao, L.: Cosaliency detection based on intrasaliency prior transfer and deep intersaliency mining. IEEE Trans. Neural Netw. Learn. Syst. (TNNLS) 27(6), 1163–1176 (2015)

    Article  MathSciNet  Google Scholar 

  54. Zhang, D., Han, J., Li, C., Wang, J., Li, X.: Detection of co-salient objects by looking deep and wide. Int. J. Comput. Vis. (IJCV) 120(2), 215–232 (2016)

    Article  MathSciNet  Google Scholar 

  55. Zhang, D., Meng, D., Han, J.: Co-saliency detection via a self-paced multiple-instance learning framework. IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI) 39(5), 865–878 (2016)

    Article  Google Scholar 

  56. Zhang, K., Li, T., Liu, B., Liu, Q.: Co-saliency detection via mask-guided fully convolutional networks with multi-scale label smoothing. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3095–3104 (2019)

  57. Zhang, L., Dai, J., Lu, H., He, Y., Wang, G.: A bi-directional message passing model for salient object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1741–1750 (2018)

  58. Zhang, P., Wang, D., Lu, H., Wang, H., Ruan, X.: Amulet: Aggregating multi-level convolutional features for salient object detection. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 202–211 (2017)

  59. Zhao, J.X., Cao, Y., Fan, D.P., Cheng, M.M., Li, X.Y., Zhang, L.: Contrast prior and fluid pyramid integration for rgbd salient object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3927–3936 (2019)

  60. Zhao, J.X., Liu, J.J., Fan, D.P., Cao, Y., Yang, J., Cheng, M.M.: Egnet: Edge guidance network for salient object detection. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 8779–8788 (2019)

  61. Zhao, R., Ouyang, W., Li, H., Wang, X.: Saliency detection by multi-context deep learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1265–1274 (2015)

  62. Zheng, X., Zha, Z.J., Zhuang, L.: A feature-adaptive semi-supervised framework for co-saliency detection. In: 2018 ACM Multimedia Conference on Multimedia Conference, pp. 959–966. ACM (2018)

Download references

Author information

Authors and Affiliations

  1. The Northeast Petroleum University, Daqing, China

    Hongbo Bi, Kang Wang, Di Lu, Chenlei Wu, Wei Wang & Lina Yang

Authors
  1. Hongbo Bi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kang Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Di Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Chenlei Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Wei Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Lina Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hongbo Bi.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bi, H., Wang, K., Lu, D. et al. \(\hbox {C}^{2}\)Net: a complementary co-saliency detection network. Vis Comput 37, 911–923 (2021). https://doi.org/10.1007/s00371-020-01842-4

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-020-01842-4

Keywords

Search

Navigation