Skip to main content
SpringerLink
Log in

Iterative algorithm for interactive co-segmentation using semantic information propagation

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

This paper introduces a novel iterative approach for interactive single or multiple foreground co-segmentation using semantic information. A quadratic cost function based on a graph model is proposed. The cost function includes a ‘smoothness’ and a ‘label-information’ terms. The ‘label-information’ term propagates the feature-level and contextual information. This information is updated based on the features and neighborhood patterns of all the images after each iteration. The approach can be easily implemented with a few scribbles on a few random images. The paper also proposes a model called Neighborhood Pattern Model (NPM) for contextual information. Along with feature level information, NPM helps to give semantic meanings to the labels (i.e., foreground(s) and background). Moreover, in the case of insufficient features (i.e., same features for different labels), NPM can be effective to distinct the labels. Experimental results on two benchmark datasets, iCoseg and FlickrMFC, illustrate the better performance of the proposed approach over the current state-of-the-art co-segmentation methods.

Workflow of the proposed algorithm. The left images are samples of the ’Hot-Balloons’ group in iCoseg dataset [1]. It is the only image of the group which is scribbled by user. Green and red scribbles indicate label1 (i.e., background) and label2 (i.e., foreground), respectively. The final results are illustrated in the right

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

References

  1. Batra D, Kowdle A, Parikh D, Luo J, Chen T (2010) icoseg: interactive co-segmentation with intelligent scribble guidance. In: Conference on computer vision and pattern recognition (CVPR). IEEE, pp 3169–3176

  2. Blake A, Rother C, Brown M, Perez P, Torr P (2004) Interactive image segmentation using an adaptive gmmrf model. Comput Vis-ECCV 2004:428–441

    MATH  Google Scholar 

  3. Boyd S, Vandenberghe L (2004) Convex optimization. Cambridge University Press

  4. Boykov Y, Funka-Lea G (2006) Graph cuts and efficient nd image segmentation. computer vision. Int J 70 (2):109–131

    Google Scholar 

  5. Cao L, Fei-Fei L (2007) Spatially coherent latent topic model for concurrent segmentation and classification of objects and scenes. In: International Conference on computer vision (ICCV). IEEE, pp 1–8

  6. Chai Y, Lempitsky VS, Zisserman A (2011) Bicos: a bi-level co-segmentation method for image classification. In: International Conference on computer vision (ICCV). IEEE, pp 2579–2586

  7. Chai Y, Rahtu E, Lempitsky V, Van Gool L, Zisserman A (2012) Tricos: a tri-level class-discriminative co-segmentation method for image classification. In: European Conference on computer vision (ECCV). Springer, pp 794–807

  8. Chang HS, Wang YCF (2015) Optimizing the decomposition for multiple foreground cosegmentation. Comput Vis Image Underst 141:18–27

    Article  Google Scholar 

  9. Collins MD, Xu J, Grady L, Singh V (2012) Random walks based multi-image segmentation: quasiconvexity results and gpu-based solutions. In: Conference on computer vision and pattern recognition (CVPR). IEEE, pp 1656–1663

  10. Comaniciu D, Meer P (2002) Mean shift: a robust approach toward feature space analysis. IEEE Trans Pattern Anal Mach Intell 24(5):603–619

    Article  Google Scholar 

  11. Cui J, Yang Q, Wen F, Wu Q, Zhang C, Van Gool L, Tang X (2008) Transductive object cutout. In: Conference on computer vision and pattern recognition (CVPR). IEEE, pp 1–8

  12. Dong X, Shen J, Shao L, Yang MH (2015) Interactive cosegmentation using global and local energy optimization. IEEE Trans Image Process 24(11):3966–3977

    Article  MathSciNet  Google Scholar 

  13. Eslami A, Karamalis A, Katouzian A, Navab N (2013) Segmentation by retrieval with guided random walks: application to left ventricle segmentation in mri. Med Image Anal 17(2):236–253

    Article  Google Scholar 

  14. Fathi A, Naghsh-Nilchi AR (2013) Automatic wavelet-based retinal blood vessels segmentation and vessel diameter estimation. Biomed Signal Process Control 8(1):71–80

    Article  Google Scholar 

  15. Fu H, Xu D, Lin S, Liu J (2015) Object-based rgbd image co-segmentation with mutex constraint. In: Conference on Computer vision and pattern recognition (CVPR). IEEE, pp 4428–4436

  16. Fu H, Xu D, Zhang B, Lin S, Ward RK (2015) Object-based multiple foreground video co-segmentation via multi-state selection graph. IEEE Trans Image Process 24(11):3415–3424

    Article  MathSciNet  Google Scholar 

  17. Hochbaum DS, Singh V (2009) An efficient algorithm for co-segmentation. In: International Conference on computer vision (ICCV). IEEE, pp 269–276

  18. Joulin A, Bach F, Ponce J (2010) Discriminative clustering for image co-segmentation. In: Conference on Computer vision and pattern recognition (CVPR). IEEE, pp 1943–1950

  19. Joulin A, Bach F, Ponce J (2012) Multi-class cosegmentation. In: Conference on Computer vision and pattern recognition (CVPR). IEEE, pp 542–549

  20. Kamranian Z, Monadjemi SA, Nematbakhsh N (2013) A novel free format persian/arabic handwritten zip code recognition system. Comput Electric Eng 39(7):1970–1979

    Article  Google Scholar 

  21. Kamranian Z, Tombari F, Nilchi ARN, Monadjemi A, Navab N (2018) J Vis Commun Image Represent. Elsevier

  22. Kim E, Li H, Huang X (2012) A hierarchical image clustering cosegmentation framework. In: Conference on Computer vision and pattern recognition (CVPR). IEEE, pp 686–693

  23. Kim G, Xing EP (2012) On multiple foreground cosegmentation. In: Conference on Computer vision and pattern recognition (CVPR). IEEE, pp 837–844

  24. Kim G, Xing EP, Fei-Fei L, Kanade T (2011) Distributed cosegmentation via submodular optimization on anisotropic diffusion. In: International Conference on computer vision (ICCV). IEEE, pp 169–176

  25. Kim TH, Lee KM, Lee SU (2010) Nonparametric higher-order learning for interactive segmentation. In: Conference on Computer vision and pattern recognition (CVPR). IEEE, pp 3201–3208

  26. Lee C, Jang WD, Sim JY, Kim CS (2015) Multiple random walkers and their application to image cosegmentation. In: Conference on Computer vision and pattern recognition (CVPR). IEEE, pp 3837–3845

  27. Li K, Zhang J, Tao W (2016) Unsupervised co-segmentation for indefinite number of common foreground objects. IEEE Trans Image Process 25(4):1898–1909

    Article  MathSciNet  Google Scholar 

  28. Liu Z, Zhu J, Bu J, Chen C (2014) Object cosegmentation by nonrigid mapping. Neurocomputing 135:107–116

    Article  Google Scholar 

  29. Ma T, Latecki LJ (2013) Graph transduction learning with connectivity constraints with application to multiple foreground cosegmentation. In: Conference on Computer vision and pattern recognition (CVPR). IEEE, pp 1955–1962

  30. Meng F, Cai J, Li H (2016) Cosegmentation of multiple image groups. Comput Vis Image Underst 146:67–76

    Article  Google Scholar 

  31. Meyer CD (2000) Matrix analysis and applied linear algebra. Siam

  32. Mukherjee L, Singh V, Dyer CR (2009) Half-integrality based algorithms for cosegmentation of images. In: Conference on Computer vision and pattern recognition (CVPR). IEEE, pp 2028–2035

  33. Mukherjee L, Singh V, Peng J (2011) Scale invariant cosegmentation for image groups. In: Conference on Computer vision and pattern recognition (CVPR). IEEE, pp 1881–1888

  34. Quan R, Han J, Zhang D, Nie F (2016) Object co-segmentation via graph optimized-flexible manifold ranking. In: Conference on Computer vision and pattern recognition (CVPR). IEEE, pp 687–695

  35. Rother C, Kolmogorov V, Blake A (2004) Grabcut: interactive foreground extraction using iterated graph cuts. In: ACM Transactions on graphics (TOG), vol 23. ACM, pp 309–314

  36. Rother C, Minka T, Blake A, Kolmogorov V (2006) Cosegmentation of image pairs by histogram matching-incorporating a global constraint into mrfs. In: Conference on Computer vision and pattern recognition (CVPR). IEEE, pp 993–1000

  37. Roy AG, Conjeti S, Karri SPK, Sheet D, Katouzian A, Wachinger C, Navab N (2017) Relaynet: retinal layer and fluid segmentation of macular optical coherence tomography using fully convolutional network. arXiv:http://arXiv.org/abs/170402161

  38. Rubinstein M, Joulin A, Kopf J, Liu C (2013) Unsupervised joint object discovery and segmentation in internet images. In: Conference on Computer vision and pattern recognition (CVPR). IEEE, pp 1939–1946

  39. Rubio JC, Serrat J, López A, Paragios N (2012) Unsupervised co-segmentation through region matching. In: Conference on Computer vision and pattern recognition (CVPR). IEEE, pp 749–756

  40. Rupprecht C, Peter L, Navab N (2015) Image segmentation in twenty questions. In: Conference on Computer vision and pattern recognition (CVPR). pp 3314–3322

  41. Sadeghian H, Villani L, Kamranian Z, Karami A (2015) Visual servoing with safe interaction using image moments. In: International Conference on intelligent robots and systems (IROS). IEEE, pp 5479–5485

  42. Vicente S, Kolmogorov V, Rother C (2010) Cosegmentation revisited: models and optimization. In: European Conference on computer vision (ECCV). Springer, pp 465–479

  43. Vicente S, Rother C, Kolmogorov V (2011) Object cosegmentation. In: Conference on computer vision and pattern recognition (CVPR). IEEE, pp 2217–2224

  44. Wang L, Hua G, Xue J, Gao Z, Zheng N (2014) Joint segmentation and recognition of categorized objects from noisy web image collection. IEEE Trans Image Process 23(9):4070–4086

    Article  MathSciNet  Google Scholar 

  45. Wang W, Shen J (2016) Higher-order image co-segmentation. IEEE Trans Multimed 18(6):1011–1021

    Article  Google Scholar 

  46. Wang W, Shen J, Li X, Porikli F (2015) Robust video object cosegmentation. IEEE Trans Image Process 24(10):3137–3148

    Article  MathSciNet  Google Scholar 

  47. Wei W, Yang XL, Zhou B, Feng J, Shen PY (2012) Combined energy minimization for image reconstruction from few views. Math Probl Eng, 2012

  48. Wei W, Fan X, Song H, Wang H (2017) Video tamper detection based on multi-scale mutual information. Multimed Tools Appl, 1–18

  49. Wiesent K, Barth K, Navab N, Durlak P, Brunner T, Schuetz O, Seissler W (2000) Enhanced 3-d-reconstruction algorithm for c-arm systems suitable for interventional procedures. IEEE Trans Med Imag 19 (5):391–403

    Article  Google Scholar 

  50. Zhu H, Lu J, Cai J, Zheng J, Thalmann NM (2014) Multiple foreground recognition and cosegmentation: an object-oriented crf model with robust higher-order potentials. In: Winter Conference on applications of computer vision (WACV). IEEE, pp 485–492

Download references

Author information

Authors and Affiliations

  1. Department of Artificial Intelligence, Faculty of Computer Engineering, University of Isfahan, Isfahan, 8174673441, Iran

    Zahra Kamranian, Ahmad Reza Naghsh Nilchi & Amirhassan Monadjemi

  2. Computer Aided Medical Procedures and Augmented Reality, Technische Universität München, Munich, Germany

    Nassir Navab

Authors
  1. Zahra Kamranian
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ahmad Reza Naghsh Nilchi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Amirhassan Monadjemi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Nassir Navab
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ahmad Reza Naghsh Nilchi.

Ethics declarations

Conflict of interests

Authors Zahra Kamranian, Ahmad Reza Naghsh Nilchi, Amirhassan Monadjemi and Nassir Navab declare that they have no conflict of interest regarding the publication of this paper.

Additional information

Ethical approval

This article does not contain any studies with human participants or animals performed by any of the authors.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kamranian, Z., Naghsh Nilchi, A.R., Monadjemi, A. et al. Iterative algorithm for interactive co-segmentation using semantic information propagation. Appl Intell 48, 5019–5036 (2018). https://doi.org/10.1007/s10489-018-1221-3

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-018-1221-3

Keywords

Search

Navigation