Skip to main content
SpringerLink
Log in

Weakly-supervised instance co-segmentation via tensor-based salient co-peak search

  • Research Article
  • Published:
Frontiers of Computer Science Aims and scope Submit manuscript

Abstract

Instance co-segmentation aims to segment the co-occurrent instances among two images. This task heavily relies on instance-related cues provided by co-peaks, which are generally estimated by exhaustively exploiting all paired candidates in point-to-point patterns. However, such patterns could yield a high number of false-positive co-peaks, resulting in over-segmentation whenever there are mutual occlusions. To tackle with this issue, this paper proposes an instance co-segmentation method via tensor-based salient co-peak search (TSCPS-ICS). The proposed method explores high-order correlations via triple-to-triple matching among feature maps to find reliable co-peaks with the help of co-saliency detection. The proposed method is shown to capture more accurate intra-peaks and inter-peaks among feature maps, reducing the false-positive rate of co-peak search. Upon having accurate co-peaks, one can efficiently infer responses of the targeted instance. Experiments on four benchmark datasets validate the superior performance of the proposed method.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Chen H Y, Lin Y Y, Chen B Y. Co-segmentation guided Hough transform for robust feature matching. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 37(12): 2388–2401

    Article  Google Scholar 

  2. Subramaniam A, Nambiar A, Mittal A. Co-segmentation inspired attention networks for video-based person re-identification. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019, 562–572

  3. Mustafa A, Hilton A. Semantically coherent co-segmentation and reconstruction of dynamic scenes. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017, 5583–5592

  4. Rother C, Minka T, Blake A, Kolmogorov V. Cosegmentation of image pairs by histogram matching - incorporating a global constraint into MRFs. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. 2006, 993–1000

  5. Wang F, Huang Q, Guibas L J. Image co-segmentation via consistent functional maps. In: Proceedings of the IEEE International Conference on Computer Vision. 2013, 849–856

  6. Taniai T, Sinha S N, Sato Y. Joint recovery of dense correspondence and cosegmentation in two images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016, 4246–4255

  7. Li L, Liu Z, Zhang J. Unsupervised image co-segmentation via guidance of simple images. Neurocomputing, 2018, 275: 1650–1661

    Article  Google Scholar 

  8. Tao G, Li H, Huang J, Han C, Chen J, Ruan G, Huang W, Hu Y, Dan T, Zhang B, He S, Liu L, Cai H. SeqSeg: A sequential method to achieve nasopharyngeal carcinoma segmentation free from background dominance. Medical Image Analysis, 2022, 78: 102381

    Article  Google Scholar 

  9. Li Y, Dan T, Li H, Chen J, Peng H, Liu L, Cai H. NPCNet: Jointly segment primary nasopharyngeal carcinoma tumors and metastatic lymph nodes in MR images. IEEE Transactions on Medical Imaging, 2022, 41(7): 1639–1650

    Article  Google Scholar 

  10. Li Y, Peng H, Dan T, Hu Y, Tao G, Cai H. Coarse-to-fine nasopharyngeal carcinoma segmentation in MRI via multi-stage rendering. In: Proceedings of the IEEE International Conference on Bioinformatics and Biomedicine. 2020, 623–628

  11. Han J, Quan R, Zhang D, Nie F. Robust object co-segmentation using background prior. IEEE Transactions on Image Processing, 2018, 27(4): 1639–1651

    Article  MathSciNet  Google Scholar 

  12. Li W, Hosseini Jafari O, Rother C. Deep object co-segmentation. In: Proceedings of the 14th Asian Conference on Computer Vision. 2018, 638–653

  13. Zhang K, Chen J, Liu B, Liu Q. Deep object co-segmentation via spatial-semantic network modulation. In: Proceedings of the 34th AAAI Conference on Artificial Intelligence. 2020, 12813–12820

  14. Hsu K J, Lin Y Y, Chuang Y Y. DeepCO3: Deep instance co-segmentation by co-peak search and co-saliency detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019, 8838–8847

  15. Papandreou G, Chen L C, Murphy K P, Yuille A L. Weakly-and semisupervised learning of a deep convolutional network for semantic image segmentation. In: Proceedings of the IEEE International Conference on Computer Vision. 2015, 1742–1750

  16. Lin D, Dai J, Jia J, He K, Sun J. ScribbleSup: Scribble-supervised convolutional networks for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016, 3159–3167

  17. Roy A, Todorovic S. Combining bottom-up, top-down, and smoothness cues for weakly supervised image segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017, 7282–7291

  18. Lai B, Gong X. Saliency guided dictionary learning for weakly-supervised image parsing. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016, 3630–3639

  19. Kolesnikov A, Lampert C H. Seed, expand and constrain: three principles for weakly-supervised image segmentation. In: Proceedings of the 14th European Conference on Computer Vision. 2016, 695–711

  20. Zitnick C L, Dollár P. Edge boxes: Locating object proposals from edges. In: Proceedings of the 13th European Conference on Computer Vision. 2014, 391–405

  21. Pont-Tuset J, Arbeláez P, Barron J T, Marques F, Malik J. Multiscale combinatorial grouping for image segmentation and object proposal generation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(1): 128–140

    Article  Google Scholar 

  22. Wan F, Wei P, Jiao J, Han Z, Ye Q. Min-entropy latent model for weakly supervised object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2018, 1297–1306

  23. Wei Y, Feng J, Liang X, Cheng M M, Zhao Y, Yan S. Object region mining with adversarial erasing: A simple classification to semantic segmentation approach. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017, 6488–6496

  24. Khan R A, Meyer A, Konik H, Bouakaz S. Saliency-based framework for facial expression recognition. Frontiers of Computer Science, 2019, 13(1): 183–198

    Article  Google Scholar 

  25. Li T, Zhang K, Shen S, Liu B, Liu Q, Li Z. Image co-saliency detection and instance co-segmentation using attention graph clustering based graph convolutional network. IEEE Transactions on Multimedia, 2021, 24: 492–505

    Article  Google Scholar 

  26. Meng F, Luo K, Li H, Wu Q, Xu X. Weakly supervised semantic segmentation by a class-level multiple group cosegmentation and foreground fusion strategy. IEEE Transactions on Circuits and Systems for Video Technology, 2020, 30(12): 4823–4836

    Article  Google Scholar 

  27. Gong X, Liu X, Li Y, Li H. A novel co-attention computation block for deep learning based image co-segmentation. Image and Vision Computing, 2020, 101: 103973

    Article  Google Scholar 

  28. Wei X S, Zhang C L, Li Y, Xie C W, Wu J, Shen C, Zhou Z H. Deep descriptor transforming for image co-localization. In: Proceedings of the 26th International Joint Conference on Artificial Intelligence. 2017, 3048–3054

  29. Wei X S, Zhang C L, Wu J, Shen C, Zhou Z H. Unsupervised object discovery and co-localization by deep descriptor transformation. Pattern Recognition, 2019, 88: 113–126

    Article  Google Scholar 

  30. Zhou Y, Zhu Y, Ye Q, Qiu Q, Jiao J. Weakly supervised instance segmentation using class peak response. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2018, 3791–3800

  31. Fan R, Cheng M M, Hou Q, Mu T J, Wang J, Hu S M. S4Net: single stage salient-instance segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019, 6096–6105

  32. Li G, Xie Y, Lin L, Yu Y. Instance-level salient object segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017, 247–256

  33. Aumüller M, Bernhardsson E, Faithfull A. ANN-benchmarks: a benchmarking tool for approximate nearest neighbor algorithms. In: Proceedings of the 10th International Conference on Similarity Search and Applications. 2017, 34–49

  34. Zass R, Shashua A. Probabilistic graph and hypergraph matching. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2008, 1–8

  35. Zhang D, Han J, Zhang Y. Supervision by fusion: Towards unsupervised learning of deep salient object detector. In: Proceedings of the IEEE International Conference on Computer Vision. 2017, 4068–4076

  36. Tang K, Joulin A, Li L J, Li F F. Co-localization in real-world images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2014, 1464–1471

  37. Cho M, Kwak S, Schmid C, Ponce J. Unsupervised object discovery and localization in the wild: part-based matching with bottom-up region proposals. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2015, 1201–1210

  38. Collins E, Achanta R, Susstrunk S. Deep feature factorization for concept discovery. In: Proceedings of the 15th European Conference on Computer Vision. 2018, 352–368

  39. Luo Z, Mishra A, Achkar A, Eichel J, Li S, Jodoin P M. Non-local deep features for salient object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017, 6593–6601

  40. Li X, Yang F, Cheng H, Liu W, Shen D. Contour knowledge transfer for salient object detection. In: Proceedings of the 15th European Conference on Computer Vision. 2018, 370–385

  41. Liu N, Zhao W, Shao L, Han J. SCG: Saliency and contour guided salient instance segmentation. IEEE Transactions on Image Processing, 2021, 30: 5862–5874

    Article  Google Scholar 

  42. Wang X, Feng J, Hu B, Ding Q, Ran L, Chen X, Liu W. Weakly-supervised instance segmentation via class-agnostic learning with salient images. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021, 10220–10230

Download references

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China (Grant Nos. U21A20520, 62172112), the Key-Area Research and Development of Guangdong Province (2022A0505050014, 2020B1111190001), the National Key Research and Development Program of China (2022YFE0112200), and the Key-Area Research and Development Program of Guangzhou City (202206030009).

Author information

Authors and Affiliations

  1. School of Computer Science, Guangdong Polytechnic Normal University, Guangzhou, 510665, China

    Wuxiu Quan & Yue Zhang

  2. School of Computer Science and Engineering, South China University of Technology, Guangzhou, 510006, China

    Wuxiu Quan, Yu Hu, Tingting Dan, Junyu Li & Hongmin Cai

Authors
  1. Wuxiu Quan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yu Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Tingting Dan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Junyu Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yue Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Hongmin Cai
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yue Zhang.

Additional information

Wuxiu Quan is currently pursuing the MS degree with the School of Computer Science and Engineering, South China University of Technology, China. His current research interests include deep learning, multiview clustering.

Yu Hu is currently pursuing the PhD degree with School of Computer Science and Engineering, South China University of Technology, China. His research interests include machine learning, multiview clustering, and deep clustering.

Tingting Dan is currently pursuing the PhD degree with the School of Computer Science and Engineering, South China University of Technology, China. Her current research interests cover image processing and manifold learning.

Junyu Li is currently pursuing the PhD degree in Computer Science and Engineering from South China University of Technology, China. His research interests include machine learning and image processing.

Yue Zhang received the PhD degree in Computer Science from Hong Kong Baptist University, China in 2017. She is an Associate Professor with the School of Computer Science, Guangdong Polytechnic Normal University, China. Her research interests include bioinformatics and big data mining.

Hongmin Cai is a Professor at the School of Computer Science and Engineering, South China University of Technology, China. He received the BS and MS degrees from the Harbin Institute of Technology, China in 2001 and 2003, respectively, and the PhD from Hong Kong University, China in 2007. His areas of research interests include biomedical image processing and omics data integration.

Electronic supplementary material

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Quan, W., Hu, Y., Dan, T. et al. Weakly-supervised instance co-segmentation via tensor-based salient co-peak search. Front. Comput. Sci. 18, 182305 (2024). https://doi.org/10.1007/s11704-022-2468-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11704-022-2468-8

Keywords

Search

Navigation