Skip to main content
SpringerLink
Log in

Weakly supervised multi-scale recurrent convolutional neural network for co-saliency detection and co-segmentation

  • IAPR-MedPRAI
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

A new approach involving multi-scale recurrent convolutional neural network (RCNN) has been proposed for co-saliency object detection. The proposed approach involves careful separation of foreground and background superpixel regions from a single image taken from a related group of images in order to train an RCNN to extract the common salient object regions. The one-dimensional convolutional neural network (CNN) is trained using superpixels extracted from several multi-scaled images derived from a single image in every group. The output of the CNN is fed into the recurrent neural network to classify the common object superpixel properties from the remaining images. The superpixel feature training-based RCNN approach addresses two challenges: It requires a small training dataset of about 38 representative images. Further, the use of 1-dimensional superpixel features to train the RCNN results in faster training. The proposed approach delivers accurate identification and segmentation of the common salient object from an image group even under extreme background conditions and object pose variations. The approach has been extensively evaluated using public domain datasets, such as imagepair, iCoseg-sub and iCoseg. The proposed approach delivers higher accuracy, F-measure and lower mean absolute error compared to several state-of-the-art approaches.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  1. Achanta R, Shaji A, Smith K, Lucchi A, Fua P, Susstrunk S (2012) SLIC superpixels compared to state-of-the-art superpixel methods. IEEE Trans Pattern Anal Mach Intell 34(11):2274–2282

    Google Scholar 

  2. Al-Rfou R, Alain G, Almahairi A, Angermueller C, Bahdanau D, Ballas N, Bastien F, Bayer J, Belikov A, Belopolsky A et al (2016) Theano: A python framework for fast computation of mathematical expressions. ArXiv preprint arXiv:1605.02688

  3. Batra D, Kowdle A, Parikh D, Luo J, Chen T (2010) iCoseg: interactive co-segmentation with intelligent scribble guidance. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp 3169–3176

  4. Borji A, Cheng MM, Hou Q, Jiang H, Li J (2014) Salient object detection: a survey. ArXiv preprint arXiv:1411.5878

  5. Chen M, Velasco-Forero S, Tsang I, Cham TJ (2015) Objects co-segmentation: propagated from simpler images. In: Proceedings of the IEEE international conference on acoustics, speech and signal processing (ICASSP), pp 1682–1686

  6. Cheng MM, Warrell J, Lin WY, Zheng S, Vineet V, Crook N (2013) Efficient salient region detection with soft image abstraction. In: Proceedings of the IEEE international conference on computer vision (ICCV), pp 1529–1536

  7. Chollet F et al (2015) Keras: deep learning library for Theano and TensorFlow. https://keras.io/k, 7(8):T1

  8. Faktor A, Irani M (2013) Co-segmentation by composition. In: Proceedings of the IEEE international conference on computer vision (ICCV), pp 1297–1304

  9. Fu H, Cao X, Tu Z (2013) Cluster-based co-saliency detection. IEEE Trans Image Process 22(10):3766–3778

    Article  MathSciNet  Google Scholar 

  10. Jacobs DE, Goldman DB, Shechtman E (2010) Cosaliency: where people look when comparing images. In: Proceedings of the 23rd annual ACM symposium on user interface software and technology, pp 219–228

  11. Jiang H, Wang J, Yuan Z, Wu Y, Zheng N, Li S (2013) Salient object detection: a discriminative regional feature integration approach. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp 2083–2090

  12. Joulin A, Bach F, Ponce J (2010) Discriminative clustering for image co-segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp 1943–1950

  13. Joulin A, Bach F, Ponce J (2012) Multi-class cosegmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp 542–549

  14. Kim G, Xing EP, Fei-Fei L, Kanade T (2011) Distributed cosegmentation via submodular optimization on anisotropic diffusion. In: Proceedings of the IEEE international conference on computer vision (ICCV), pp 169–176

  15. Kim J, Han D, Tai YW, Kim J (2014) Salient region detection via high-dimensional color transform. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp 883–890

  16. Kingma DP, Ba J (2014) Adam: a method for stochastic optimization. ArXiv preprint arXiv:1412.6980

  17. Kompella A, Kulkarni RV (2018) Co-saliency detection via extremely weakly supervised convolutional neural network. In: 2018 IEEE symposium series on computational intelligence (SSCI). IEEE, pp 447–454

  18. Li G, Yu Y (2016) Visual saliency detection based on multiscale deep CNN features. IEEE Trans Image Process 25(11):5012–5024

    Article  MathSciNet  Google Scholar 

  19. Li H, Ngan KN (2011) A co-saliency model of image pairs. IEEE Trans Image Process 20(12):3365–3375

    Article  MathSciNet  Google Scholar 

  20. Li M, Dong S, Zhang K, Gao Z, Wu X, Zhang H, Yang G, Li S (2018) Deep learning intra-image and inter-images features for co-saliency detection. In: Proceedings of the 29th British machine vision conference (BMVC), pp 1–13

  21. Li W, Jafari OH, Rother C (2018) Deep object co-segmentation. ArXiv preprint arXiv:1804.06423

  22. Li Y, Fu K, Liu Z, Yang J (2015) Efficient saliency-model-guided visual co-saliency detection. IEEE Signal Process Lett 22(5):588–592

    Article  Google Scholar 

  23. Liu Z, Zou W, Li L, Shen L, Le Meur O (2014) Co-saliency detection based on hierarchical segmentation. IEEE Signal Process Lett 21(1):88–92

    Article  Google Scholar 

  24. Mukherjee P, Lall B, Lattupally S (2018) Object cosegmentation using deep siamese network. ArXiv preprint arXiv:1803.02555

  25. Papushoy A, Bors AG (2015) Image retrieval based on query by saliency content. Digital Signal Process 36:156–173

    Article  MathSciNet  Google Scholar 

  26. Rother C, Minka T, Blake A, Kolmogorov V (2006) Cosegmentation of image pairs by histogram matching-incorporating a global constraint into mrfs. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), vol 1, pp 993–1000

  27. Rubinstein M, Joulin A, Kopf J, Liu C (2013) Unsupervised joint object discovery and segmentation in internet images. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)

  28. Tsai CC, Li W, Hsu KJ, Qian X, Lin YY (2019) Image co-saliency detection and co-segmentation via progressive joint optimization. IEEE Trans Image Process 28(1):56–71

    Article  MathSciNet  Google Scholar 

  29. Vicente S, Rother C, Kolmogorov V (2011) Object cosegmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp 2217–2224

  30. Wang S, Zhang H, Wang H (2017) Object co-segmentation via weakly supervised data fusion. Comput Vis Image Underst 155:43–54

    Article  Google Scholar 

  31. Wei L, Zhao S, Bourahla OEF, Li X, Wu F (2017) Group-wise deep co-saliency detection. ArXiv preprint arXiv:1707.07381

  32. Yan Q, Xu L, Shi J, Jia J (2013) Hierarchical saliency detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp 1155–1162

  33. Yang B, Yu H, Hu R (2015) Unsupervised regions based segmentation using object discovery. J Visual Commun Image Represent 31:125–137

    Article  Google Scholar 

  34. Yang C, Zhang L, Lu H, Ruan X, Yang MH (2013) Saliency detection via graph-based manifold ranking. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp 3166–3173

  35. Ye L, Liu Z, Li J, Zhao WL, Shen L (2015) Co-saliency detection via co-salient object discovery and recovery. IEEE Signal Process Lett 22(11):2073–2077

    Article  Google Scholar 

  36. Zhang D, Fu H, Han J, Wu F (2016) A review of co-saliency detection technique: fundamentals, applications, and challenges. ArXiv preprint arXiv:1604.07090

  37. Zhang D, Han J, Han J, Shao L (2016) Cosaliency detection based on intrasaliency prior transfer and deep intersaliency mining. IEEE Trans Neural Netw Learn Syst 27(6):1163–1176

    Article  MathSciNet  Google Scholar 

  38. Zhang D, Han J, Li C, Wang J (2015) Co-saliency detection via looking deep and wide. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp 2994–3002

  39. Zhang D, Meng D, Li C, Jiang L, Zhao Q, Han J (2015) A self-paced multiple-instance learning framework for co-saliency detection. In: Proceedings of the IEEE international conference on computer vision (ICCV), pp 594–602

  40. Zhu W, Liang S, Wei Y, Sun J (2014) Saliency optimization from robust background detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp 2814–2821

Download references

Acknowledgements

Authors gratefully acknowledge the support received from M S Ramaiah University of Applied Sciences, Bengaluru, India.

Author information

Authors and Affiliations

  1. Center for Machine Learning and Computational Intelligence, M S Ramaiah University of Applied Sciences, Bangalore, Karnataka State, India

    Aditya Kompella

  2. Department of Electronics and Communication Engineering, M S Ramaiah University of Applied Sciences, Bangalore, Karnataka State, India

    Raghavendra V. Kulkarni

Authors
  1. Aditya Kompella
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Raghavendra V. Kulkarni
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Aditya Kompella.

Ethics declarations

Conflict of interest

The authors declare no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kompella, A., Kulkarni, R.V. Weakly supervised multi-scale recurrent convolutional neural network for co-saliency detection and co-segmentation. Neural Comput & Applic 32, 16571–16588 (2020). https://doi.org/10.1007/s00521-019-04265-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-019-04265-y

Keywords

Search

Navigation