Abstract
The purpose of co-saliency detection is to find out the salient and common objects of related images. This paper proposes a novel reverse collaborative fusion model (RCFM) for co-saliency detection. The model is mainly composed of two parts: reverse message fusion module (RMFM) and collaborative consistency learning module (CCLM). Specifically, we first aggregate the features in high-level layers as global guidance by using the cascaded decoder (CD). Then, we propose repeated RMFMs on each side output to complete the complementary fusion of deep and shallow information. Then, we fuse multi-scale feature maps as initial co-saliency maps. Finally, the CCLM extracts the collaborative information between images to improve the quality of the initial co-saliency map to obtain the final co-saliency map. The model fully considers the semantic features of high-level and the boundary features of low-level, thereby correcting some deviation predictions and improving the accuracy of co-saliency detection. Compared to the state-of-the-art approaches, experimental results demonstrate that our proposed approach achieves the best performance on four evaluation indicators of three datasets.
Similar content being viewed by others
References
Achanta, R., Hemami, S., Estrada, F., Susstrunk, S.: Frequency-tuned salient region detection. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1597–1604. IEEE (2009)
Arbelaez, P., Maire, M., Fowlkes, C., Malik, J.: Contour detection and hierarchical image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 33(5), 898–916 (2010)
Bai, C., Chen, J., Huang, L., Kpalma, K., Chen, S.: Saliency-based multi-feature modeling for semantic image retrieval. J. Vis. Commun. Image Represent. 50, 199–204 (2018)
Batra, D., Kowdle, A., Parikh, D., Luo, J., Chen, T.: icoseg: Interactive co-segmentation with intelligent scribble guidance. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 3169–3176. IEEE (2010)
Bi, H., Wang, K., Lu, D., Wu, C., Wang, W., Yang, L: C (2) net: a complementary co-saliency detection network. In: Visual Computer (2020)
Borji, A., Cheng, M.-M., Jiang, H., Li, J.: Salient object detection: a benchmark. IEEE Trans. Image Process. 24(12), 5706–5722 (2015)
Borji, A., Frintrop, S., Sihite, D.N., Itti, L.: Adaptive object tracking by learning background context. In: 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, pp. 23–30. IEEE (2012)
Cao, X., Tao, Z., Zhang, B., Huazhu, F., Feng, W.: Self-adaptively weighted co-saliency detection via rank constraint. IEEE Trans. Image Process. 23(9), 4175–4186 (2014)
Chen, J., Bai, C., Huang, L., Liu, Z., Chen, S.: Visual saliency fusion based multi-feature for semantic image retrieval. In: CCF Chinese Conference on Computer Vision, pp. 126–136. Springer (2017)
Chen, Y.-L., Hsu, C.-T.: Implicit rank-sparsity decomposition: applications to saliency/co-saliency detection. In: 2014 22nd International Conference on Pattern Recognition, pp. 2305–2310. IEEE (2014)
Cheng, M.-M., Mitra, N.J., Huang, X., Hu, S.-M.: Salientshape: group saliency in image collections. Vis. Comput. 30(4), 443–453 (2014)
Cheng, M.-M., Mitra, N., Huang, X., Torr, P.H.S., Hu, S.-M.: Global contrast based salient region detection. IEEE Trans. Pattern Anal. Mach. Intell. 37(3), 569–582 (2014)
Cong, R., Lei, J., Huazhu, F., Cheng, M.-M., Lin, W., Huang, Q.: Review of visual saliency detection with comprehensive information. IEEE Trans Circuits Syst. Video Technol. 29(10), 2941–2959 (2018)
Fan, D.-P., Li, T., Lin, Z., Ji, G.-P., Zhang, D., Cheng, M.-M., Fu, H., Shen, J.: Re-thinking co-salient object detection. Preprint arXiv:2007.03380
Fan, D.-P., Cheng, M.-M., Liu, J.-J., Gao, S.-H., Hou, Q., Borji, A.: Salient objects in clutter: bringing salient object detection to the foreground. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 3–19 (2018)
Fan, D.-P., Cheng, M.-M., Liu, Y., Li, T., Borji, A.: Structure-measure: a new way to evaluate foreground maps. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 4548–4557 (2017)
Fan, D.-P., Gong, C., Cao, Y., Ren, B., Cheng, M.-M., Borji, A.: Enhanced-alignment measure for binary foreground map evaluation. Preprint arXiv:1805.10421
Fan, D.-P., Lin, Z., Ji, G.-P., Zhang, D., Fu, H., Cheng, M.-M.: Taking a deeper look at co-salient object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2919–2929 (2020)
Fan, D.-P., Zhai, Y., Borji, A., Yang, J., Shao, L.: Bbs-net: Rgb-d salient object detection with a bifurcated backbone strategy network. In: European Conference on Computer Vision, pp. 275–292. Springer (2020)
Fu, H., Cao, X., Tu, Z.: Cluster-based co-saliency detection. IEEE Trans. Image Process. 22(10), 3766–3778 (2013)
Fu, H., Xu, D., Zhang, B., Lin, S., Ward, R.K.: Object-based multiple foreground video co-segmentation via multi-state selection graph. IEEE Trans. Image Process. 24(11), 3415–3424 (2015)
Fu, K., Gong, C., Gu, I.Y.-H., Yang, J.: Normalized cut-based saliency detection by adaptive multi-level region merging. IEEE Trans. Image Process. 24(12), 5671–5683 (2015)
Gao, S.-H., Tan, Y.-Q., Cheng, M.-M., Lu, C., Chen, Y., Yan, S.: Highly efficient salient object detection with 100k parameters (2020). Preprint arXiv:2003.05643
Ge, C., Keren, F., Liu, F., Bai, L., Yang, J.: Co-saliency detection via inter and intra saliency propagation. Signal Process. Image Commun. 44, 69–83 (2016)
Gong, C., Tao, D., Liu, W., Maybank, S.J., Fang, M., Fu, K., Yang, J.: Saliency propagation from simple to difficult. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2531–2539 (2015)
Han, J., Cheng, G., Li, Z., Zhang, D.: A unified metric learning-based framework for co-saliency detection. IEEE Trans. Circuits Syst. Video Technol. 28(10), 2473–2483 (2017)
Han, J., Quan, R., Zhang, D., Nie, F.: Robust object co-segmentation using background prior. IEEE Trans. Image Process. 27(4), 1639–1651 (2017)
Hsu, K.-J., Tsai, C.-C., Lin, Y.-Y., Qian X., Chuang, Y.-Y.: Unsupervised cnn-based co-saliency detection with graphical optimization. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 485–501 (2018)
Itti, L., Koch, C., Niebur, E.: A model of saliency-based visual attention for rapid scene analysis. IEEE Trans. Pattern Anal. Mach. Intell. 20(11), 1254–1259 (1998)
Jeong, D., Hwang, I., Cho, N.I.: Co-salient object detection based on deep saliency networks and seed propagation over an integrated graph. IEEE Trans. Image Process. 27(12), 5866–5879 (2018)
Jiang, B., Jiang, X., Zhou, A., Tang, J., Luo, B.: A unified multiple graph learning and convolutional network model for co-saliency estimation. In: Proceedings of the 27th ACM International Conference on Multimedia, pp. 1375–1382 (2019)
Klein, D.A, Frintrop, S.: Center-surround divergence of feature statistics for salient object detection. In: 2011 International Conference on Computer Vision, pp. 2214–2219. IEEE (2011)
Lee, G., Tai, Y.-W., Kim, J.: Eld-net: an efficient deep learning architecture for accurate saliency detection. IEEE Trans. Pattern Anal. Mach. Intell. 40(7), 1599–1610 (2017)
Li, B., Sun, Z., Tang, L., Sun, Y., Shi. J.: Detecting robust co-saliency with recurrent co-attention neural network. In: IJCAI, pp. 818–825 (2019)
Li, G., Yu, Y.: Visual saliency based on multiscale deep features. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5455–5463 (2015)
Li, H., Meng, F., Ngan, K.N.: Co-salient object detection from multiple images. IEEE Trans. Multimedia 15(8), 1869–1909 (2013)
Li, H., Ngan, K.N.: A co-saliency model of image pairs. IEEE Trans. Image Process. 20(12), 3365–3375 (2011)
Li, H., Chen, J., Huchuan, L., Chi, Z.: Cnn for saliency detection with low-level feature integration. Neurocomputing 226, 212–220 (2017)
Li, L., Liu, Z., Zhang, J.: Unsupervised image co-segmentation via guidance of simple images. Neurocomputing 275, 1650–1661 (2018)
Li, L., Liu, Z., Zou, W., Zhang, X., Le Meur, O.: Co-saliency detection based on region-level fusion and pixel-level refinement. In: 2014 IEEE International Conference on Multimedia and Expo (ICME), pp 1–6. IEEE (2014)
Li, X., Lu, H., Zhang, L., Ruan, X., Yang, M.-H.: Saliency detection via dense and sparse reconstruction. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2976–2983 (2013)
Li, Y., Keren, F., Liu, Z., Yang, J.: Efficient saliency-model-guided visual co-saliency detection. IEEE Signal Process. Lett. 22(5), 588–592 (2014)
Liu, Z., Zou, W., Le Meur, O.: Saliency tree: a novel saliency detection framework. IEEE Trans. Image Process. 23(5), 1937–1952 (2014)
Liu, Z., Zou, W., Li, L., Shen, L., Le Meur, O.: Co-saliency detection based on hierarchical segmentation. IEEE Signal Process. Lett. 21(1), 88–92 (2013)
Ma, Y.-F., Zhang, H.-J.: Contrast-based image attention analysis by using fuzzy growing. In: Proceedings of the 11th ACM International Conference on Multimedia, pp. 374–381 (2003)
Mahadevan, V., Vasconcelos, N.: Saliency-based discriminant tracking. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1007–1013. IEEE (2009)
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al.: Pytorch: an imperative style, high-performance deep learning library. In: Advances in Neural Information Processing Systems, pp. 8026–8037 (2019)
Peng, H., Li, B., Ling, H., Hu, W., Xiong, W., Maybank, S.J.: Salient object detection via structured matrix decomposition. IEEE Trans. Pattern Anal. Mach. Intell. 39(4), 818–832 (2016)
Ren, J., Liu, Z., Zhou, X., Sun, G., Bai, C.: Saliency integration driven by similar images. J. Vis. Commun. Image Represent. 50, 227–236 (2018)
Shen, X., Wu, Y.: A unified approach to salient object detection via low rank matrix recovery. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 853–860. IEEE (2012)
Simonyan, K., Zisserman A.: Very deep convolutional networks for large-scale image recognition (2014). Preprint arXiv:1409.1556
Tang, K., Joulin, A., Li, L.-J., Li, F.-F.: Co-localization in real-world images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1464–1471 (2014)
Toshev, A., Shi, J., Daniilidis, K.: Image matching via saliency region correspondences. In: 2007 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8. IEEE (2007)
Tsai, C.-C., Li, W., Hsu, K.-J., Qian, X., Lin, Y.-Y.: Image co-saliency detection and co-segmentation via progressive joint optimization. IEEE Trans. Image Process. 28(1), 56–71 (2018)
Wei, L., Zhao, S., Bourahla, O. E. F., Li, X., Wu, F.: Group-wise deep co-saliency detection (2017). Preprint arXiv:1707.07381
Wei, L., Zhao, S., El Farouk, O., Bourahla, X.L., Fei, W., Zhuang, Y.: Deep group-wise fully convolutional network for co-saliency detection with graph propagation. IEEE Trans. Image Process. 28(10), 5052–5063 (2019)
Winn, J., Criminisi, A., Minka, T.: Object categorization by learned universal visual dictionary. In: 10th IEEE International Conference on Computer Vision (ICCV’05), vol. 1, vol. 2, pp. 1800–1807. IEEE (2005)
Woo, S., Park, J., Lee, J.Y., Kweon, I.S.: Cbam: Convolutional block attention module. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 3–19 (2018)
Wu, Z., Su, L., Huang, Q.: Cascaded partial decoder for fast and accurate salient object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3907–3916 (2019)
Yan, Q., Xu, L., Shi, J., Jia, J.: Hierarchical saliency detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1155–1162 (2013)
Yao, X., Han, J., Zhang, D., Nie, F.: Revisiting co-saliency detection: a novel approach based on two-stage multi-view spectral rotation co-clustering. IEEE Trans. Image Process. 26(7), 3196–3209 (2017)
Ye, L., Liu, Z., Li, J., Zhao, W.-L., Shen, L.: Co-saliency detection via co-salient object discovery and recovery. IEEE Signal Process. Lett. 22(11), 2073–2077 (2015)
Zha, Z.-J., Wang, C., Liu, D., Xie, H., Zhang, Y.: Robust deep co-saliency detection with group semantic and pyramid attention. In: IEEE Transactions on Neural Networks and Learning Systems (2020)
Zhai, Y., Shah, M.: Visual attention detection in video sequences using spatiotemporal cues. In: Proceedings of the 14th ACM International Conference on Multimedia, pp. 815–824 (2006)
Zhang, D., Han, J., Han, J., Shao, L.: Cosaliency detection based on intrasaliency prior transfer and deep intersaliency mining. IEEE Trans. Neural Netw. Learn. Syst. 27(6), 1163–1176 (2015)
Zhang, D., Han, J., Li, C., Wang, J: Co-saliency detection via looking deep and wide. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2994–3002 (2015)
Zhang, D., Han, J., Li, C., Wang, J., Li, X.: Detection of co-salient objects by looking deep and wide. Int. J. Comput. Vis. 120(2), 215–232 (2016)
Zhang, D., Meng, D., Han, J.: Co-saliency detection via a self-paced multiple-instance learning framework. IEEE Trans. Pattern Anal. Mach. Intell. 39(5), 865–878 (2016)
Zhang, F., Bo, D., Zhang, L.: Saliency-guided unsupervised feature learning for scene classification. IEEE Trans. Geosci. Remote Sens. 53(4), 2175–2184 (2014)
Zhang, K., Li, T., Liu, B.,Liu, Q.: Co-saliency detection via mask-guided fully convolutional networks with multi-scale label smoothing. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3095–3104 (2019)
Zhang, K., Li, T., Shen, S., Liu, B., Chen, J., Liu, Q.: Adaptive graph convolutional network with attention graph clustering for co-saliency detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9050–9059 (2020)
Zhang, P., Wang, D., Lu, H., Wang, H., Yin, B.: Learning uncertain convolutional features for accurate saliency detection. In: Proceedings of the IEEE International Conference on computer vision, pp. 212–221 (2017)
Zhang, Y., Li, L., Cong, R., Guo, X., Xu, H., Zhang, J.: Co-saliency detection via hierarchical consistency measure. In: 2018 IEEE International Conference on Multimedia and Expo (ICME), pp. 1–6. IEEE (2018)
Zhao, J.X., Liu, J.J., Fan, D.P., Cao, Y., Yang, J., Cheng, M.M.: Egnet: Edge guidance network for salient object detection. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 8779–8788 (2019)
Zhao, R., Ouyang, W., Wang, X.: Unsupervised salience learning for person re-identification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3586–3593 (2013)
Zhou, X., Liu, Z., Sun, G., Wang, X.: Adaptive saliency fusion based on quality assessment. Multimedia Tools Appl. 76(22), 23187–23211 (2017)
Zhou, X., Liu, Z., Sun, G., Ye, L., Wang, X.: Improving saliency detection via multiple kernel boosting and adaptive fusion. IEEE Signal Process. Lett. 23(4), 517–521 (2016)
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Wang, X., Wang, W., Bi, H. et al. Reverse collaborative fusion model for co-saliency detection. Vis Comput 38, 3911–3921 (2022). https://doi.org/10.1007/s00371-021-02231-1
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00371-021-02231-1