Skip to main content
SpringerLink
Log in

Learning contextual superpixel similarity for consistent image segmentation

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

This paper addresses the problem of image segmentation by iterative region aggregations starting from an initial superpixel decomposition. Classical approaches for this task compute superpixel similarity using distance measures between superpixel descriptor vectors. This usually poses the well-known problem of the semantic gap and fails to properly aggregate visually non-homogeneous superpixels that belong to the same high-level object. This work proposes to use random forests to learn the merging probability between adjacent superpixels in order to overcome the aforementioned issues. Compared to existing works, this approach learns the fusion rules without explicit similarity measure computation. We also introduce a new superpixel context descriptor to strengthen the learned characteristics towards better similarity prediction. Image segmentation is then achieved by iteratively merging the most similar superpixel pairs selected using a similarity weighting objective function. Experimental results of our approach on four datasets including DAVIS 2017 and ISIC 2018 show its potential compared to state-of-the-art approaches.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

Notes

  1. https://challenge2018.isic-archive.com/visitedon25/11/2018

References

  1. Achanta R, Shaji A, Smith K, Lucchi A, Fua P, Süsstrunk S (2012) Slic superpixels compared to state-of-the-art superpixel methods. IEEE Transactions on Pattern Analysis and Machine Intelligence 34(11):2274–2282

    Article  Google Scholar 

  2. Amit Y, Geman D (1997) Shape quantization and recognition with randomized trees. Neural Computation 9(7):1545–1588

    Article  Google Scholar 

  3. Audebert N, Boulch A, Randrianarivo H, Le Saux B, Ferecatu M, Lefèvre S., Marlet R (2017) Deep learning for urban remote sensing. In: Urban Remote Sensing Event (JURSE), 2017 Joint, IEEE, pp 1–4

  4. Bergstra J, Bengio Y (2012) Random search for hyper-parameter optimization. J Mach Learn Res 13(Feb):281–305

    MathSciNet  MATH  Google Scholar 

  5. Bhatti AH, Rahman AU, Butt AA (2019) Unsupervised video object segmentation using conditional random fields. Image and Video Processing 13(1):9–16

    Article  Google Scholar 

  6. Bosch A, Zisserman A, Munoz X (2007) Image classification using random forests and ferns. In: 2007. ICCV 2007. IEEE 11th International Conference on Computer Vision, IEEE, pp 1–8

  7. Brahim K, Kalboussi R, Abdellaoui M, Douik A (2019) Spatio-temporal saliency detection using objectness measure. Signal, Image and Video Processing, pp 1–8

    Article  Google Scholar 

  8. Breiman L (2001) Random forests. Mach learn 45(1):5–32

    Article  Google Scholar 

  9. Breiman L (2002) Manual on setting up, using, and understanding random forests v3. 1. Statistics Department University of California Berkeley, CA, USA

  10. Chaibou MS, Conze P-H, Kalti K, Solaiman B, Mahjoub MA (2017) Adaptive strategy for superpixel-based region-growing image segmentation. J Electron Imaging 26(6):061605

    Article  Google Scholar 

  11. Chen C, Li S, Qin H, Pan Z, Yang G (2018) Bilevel feature learning for video saliency detection. IEEE Trans Multimedia 20(12):3324–3336

    Article  Google Scholar 

  12. Chen C, Li S, Wang Y, Qin H, Hao A (2017) Video saliency detection via spatial-temporal fusion and low-rank coherency diffusion. IEEE Trans Image Process 26(7):3156–3170

    Article  MathSciNet  Google Scholar 

  13. Conze P-H, Noblet V, Rousseau F, Heitz F, De Blasi V, Memeo R, Pessaux P (2017) Scale-adaptive supervoxel-based random forests for liver tumor segmentation in dynamic contrast-enhanced ct scans. International Journal of Computer Assisted Radiology and Surgery 12(2):223–233

    Article  Google Scholar 

  14. Duffner S, Garcia C (2013) Pixeltrack: a fast adaptive algorithm for tracking non-rigid objects. In: Proceedings of the IEEE International Conference on Computer Vision, pp 2480–2487

  15. Freixenet J, Muñoz X, Raba D, Martí J, Cufí X (2002) Yet another survey on image segmentation: Region and boundary information integration. In: European Conference on Computer Vision, Springer, pp 408–422

    Chapter  Google Scholar 

  16. Fukuchi K, Miyazato K, Kimura A, Takagi S, Yamato J (2009) Saliency-based video segmentation with graph cuts and sequentially updated priors. In: 2009 IEEE International Conference on Multimedia and Expo, IEEE, pp 638–641

  17. Godec M, Roth PM, Bischof H (2013) Hough-based tracking of non-rigid objects. Comput Vis Image Underst 117(10):1245–1256

    Article  Google Scholar 

  18. Granitto PM, Furlanello C, Biasioli F, Gasperi F (2006) Recursive feature elimination with random forest for ptr-ms analysis of agroindustrial products. Chemometr Intell Lab Syst 83(2):83–90

    Article  Google Scholar 

  19. Guyon I, Weston J, Barnhill S, Vapnik V (2002) Gene selection for cancer classification using support vector machines. Mach learn 46(1-3):389–422

    Article  Google Scholar 

  20. Haller E, Leordeanu M (2017) Unsupervised object segmentation in video by efficient selection of highly probable positive features. In: Proceedings of the IEEE International Conference on Computer Vision, pp 5085–5093

  21. Hamming R (1950) The bell system technical journal. Bell Syst Tech J 26 (2):147–160

    Article  MathSciNet  Google Scholar 

  22. Haralick RM, Shapiro LG (1985) Image segmentation techniques. Computer Vision, Graphics, and Image Processing 29(1):100–132

    Article  Google Scholar 

  23. Hsu C-Y, Ding J-J (2013) Efficient image segmentation algorithm using slic superpixels and boundary-focused region merging. In: Communications and Signal Processing (ICICS) 2013 9th international conference on Information, IEEE, pp 1–5

  24. Lepetit V, Fua P (2006) Keypoint recognition using randomized trees. IEEE Transactions on Pattern Analysis and Machine Intelligence 28(9):1465–1479

    Article  Google Scholar 

  25. Li F, Kim T, Humayun A, Tsai D, Rehg JM (2013) Video segmentation by tracking many figure-ground segments. In: Proceedings of the IEEE International Conference on Computer Vision, pp 2192– 2199

  26. Louppe G, Wehenkel L, Sutera A, Geurts P (2013) Understanding variable importances in forests of randomized trees. In: Advances in Neural Information Processing Systems, pp 431–439

  27. Martin D, Fowlkes C, Tal D, Malik J (2001) A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. Proc. 8th Int’l Conf Computer Vision 2:416–423

    Article  Google Scholar 

  28. Meilă M (2007) Comparing clusterings — an information based distance. J Multivar Anal 98(5):873–895

    Article  MathSciNet  Google Scholar 

  29. Oneata D, Revaud J, Verbeek J, Schmid C (2014) Spatio-temporal object detection proposals. In: European Conference on Computer Vision, Springer, pp 737–752

    Chapter  Google Scholar 

  30. Ozuysal M, Fua P, Lepetit V (2007) Fast keypoint recognition in ten lines of code. In: 2007. CVPR’07. IEEE Conference on Computer Vision and Pattern Recognition, IEEE, pp 1–8

  31. Pauly O (2012) Random forests for medical applications. PhD thesis, Technische Universität München

  32. Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, Blondel M, Prettenhofer P, Weiss R, Dubourg V, Vanderplas J, Passos A, Cournapeau D, Brucher M, Perrot M, Duchesnay E (2011) Scikit-learn: machine learning in python. J Mach Learn Res 12:2825–2830

    MathSciNet  MATH  Google Scholar 

  33. Pont-Tuset J, Perazzi F, Caelles S, Arbeláez P, Sorkine-Hornung A, Van Gool L. (2017) The 2017 davis challenge on video object segmentation. arXiv:1704.00675

  34. Ren X, Malik J (2003) Learning a classification model for segmentation. In: ICCV, vol 1, pp 10–17

  35. Sangsefidi N, Foruzan AH, Dolati A (2017) Balancing the data term of graph-cuts algorithm to improve segmentation of hepatic vascular structures. Computers in Biology and Medicine

  36. Santana TM, Machado AM, Araújo AdA, dos Santos JA (2016) Star: a contextual description of superpixels for remote sensing image classification. In: Iberoamerican Congress on Pattern Recognition, Springer, pp 300–308

  37. Shi J, Malik J (2000) Normalized cuts and image segmentation. IEEE Trans Pattern Anal Machine Intelligence 22(8):888–905

    Article  Google Scholar 

  38. Silva RE (2017) An alternative approach to counting minimum (s; t)-cuts in planar graphs

  39. Son J, Jung I, Park K, Han B (2015) Tracking-by-segmentation with online gradient boosting decision tree. In: Proceedings of the IEEE International Conference on Computer Vision, pp 3056–3064

  40. Stutz D, Hermans A, Leibe B (2016) Superpixels: an evaluation of the state-of-the-art. CoRR, arXiv:http://arxiv.org/1612.01601

  41. Tilquin F, Conze P-H, Pessaux P, Lamard M, Quellec G, Noblet V, Heitz F (2018) Robust supervoxel matching combining mid-level spectral and context-rich features. In: International Workshop on Patch-based Techniques in Medical Imaging, Springer, pp 39–47

  42. Unnikrishnan R, Pantofaru C, Hebert M (2007) Toward objective evaluation of image segmentation algorithms. IEEE transactions on pattern analysis and machine intelligence, 29(6)

    Article  Google Scholar 

  43. Vargas JE, Falcão AX, Dos Santos J, Esquerdo JCDM, Coutinho AC, Antunes J (2015) Contextual superpixel description for remote sensing image classification. In: IEEE International Geoscience and Remote Sensing Symposium (IGARSS), IEEE 2015, pp 1132–1135

  44. Vasconcelos MJM, Tavares JMR (2015) Human motion segmentation using active shape models. In: Computational and Experimental Biomedical Sciences: Methods and Applications, Springer, pp 237–246

  45. Wang S, Lu H, Yang F, Yang M-H (2011) Superpixel tracking. In: 2011 International Conference on Computer Vision, IEEE, pp 1323–1330

  46. Yang Y, Wang Y, Xue X (2016) A novel spectral clustering method with superpixels for image segmentation. Optik-International Journal for Light and Electron Optics 127(1):161–167

    Article  Google Scholar 

  47. Yeo D, Son J, Han B, Hee Han J (2017) Superpixel-based tracking-by-segmentation using markov chains. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 1812–1821

  48. Yin P, Criminisi A, Winn J, Essa I (2007) Tree-based classifiers for bilayer video segmentation. In: 2007. CVPR’07. IEEE Conference on Computer Vision and Pattern Recognition, IEEE, pp 1–8

  49. Yin S, Qian Y, Gong M (2017) Unsupervised hierarchical image segmentation through fuzzy entropy maximization. Pattern Recognition

  50. Yu H, Zhang X, Wang S, Hou B (2013) Context-based hierarchical unequal merging for sar image segmentation. IEEE Transactions on Geoscience and Remote Sensing 51(2):995–1009

    Article  Google Scholar 

  51. Zhang D, Javed O, Shah M (2013) Video object segmentation through spatially accurate and temporally dense extraction of primary object regions (open access). Technical report University of Central Florida Orlando United States

  52. Zhang Y, He K (2017) Multi-scale gaussian segmentation via graph cuts. DEStech Transactions on Computer Science and Engineering (csae)

Download references

Author information

Authors and Affiliations

  1. Université de Sousse, Ecole Nationale d’Ingénieurs de Sousse, LATIS - Laboratory of Advanced Technology and Intelligent Systems, 4023, Sousse, Tunisie

    Mahaman Sani Chaibou, Karim Kalti & Mohamed Ali Mahjoub

  2. Institut Supérieur d’Informatique et des Techniques de Communication, Université de Sousse, 4011, Hammam Sousse, Tunisie

    Mahaman Sani Chaibou

  3. IMT Atlantique, Technopôle Brest-Iroise, CS 83818, 29238, Brest Cedex 03, France

    Mahaman Sani Chaibou, Pierre-Henri Conze & Basel Solaiman

  4. LaTIM UMR 1101, Inserm, IBRBS, 22 rue Camille Desmoulins, 29238, Brest, France

    Pierre-Henri Conze

  5. Faculté des sciences de Monatir, Université de Monastir, 5019, Monastir, Tunisie

    Karim Kalti

Authors
  1. Mahaman Sani Chaibou
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Pierre-Henri Conze
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Karim Kalti
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Mohamed Ali Mahjoub
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Basel Solaiman
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mahaman Sani Chaibou.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chaibou, M.S., Conze, PH., Kalti, K. et al. Learning contextual superpixel similarity for consistent image segmentation. Multimed Tools Appl 79, 2601–2627 (2020). https://doi.org/10.1007/s11042-019-08391-6

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-019-08391-6

Keywords

Search

Navigation