Skip to main content
SpringerLink
Log in

TPN: Triple parts network for few-shot instance segmentation

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Few-shot instance segmentation aims to segment unseen objects in a so-called query image, given only one close-up illustration named support image. Recently, many few-shot instance segmentation methods are based on Mask R-CNN, and do not take advantage of the information embedding in support images. In this work, we propose Triple Parts Network (TPN) based on Mask R-CNN for few-shot instance segmentation consisting of three key modules, i.e., Attention Fusion Module (AFM), Cosine Similarity based Classifier with Circle Loss (CSC), and Cross-Local Module (CLM). AFM generates proposals targeting at the object in the support image. CSC uses cosine similarity with circle loss to classify objects. CLM is applied to the mask branch to establish a full relation between support feature maps and query feature maps. The experiments are conducted on the Microsoft COCO and PASCAL VOC benchmarks, and the results show that our TPN has better performance than other state-of-the-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Algorithm 1
Fig. 1
Fig. 2
Fig. 3
Fig. 4
Algorithm 2
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Bai M, Urtasun R (2017) Deep Watershed Transform for Instance Segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition, 5221–5229

  2. Bolya D, Zhou C, Xiao F, Lee Y J (2019) YOLACT: Real-time instance segmentation. In: IEEE International Conference on Computer Vision, 9157–9166

  3. Bolya D, Zhou C, Xiao F, Lee YJ (2020) YOLACT++: Better Real-time Instance Segmentation. IEEE Trans Pattern Anal Mach Intell 44(2):1108–1121

    Article  Google Scholar 

  4. De Brabandere B, Neven D, Van Gool L (2017). Semantic Instance Segmentation with a Discriminative Loss Function. arXiv:1708.02551

  5. Chen K, Wang J, Pang J, Cao Y, Xiong Y, Li X, Sun S, Feng W, Liu Z, Xu J (2019). MMDetection: Open MMLab Detection Toolbox and Benchmark. arXiv:1906.07155

  6. Chen H, Wang Y, Wang G, Qiao Y (2018) Lstd: A Low-Shot Transfer Detector for Object Detection. In: Association for the Advancement of Artificial Intelligence, 2836–2843

  7. Choi D, Ye-Bin M, Kim J, Oh T-H (2022) FoxInst: A Frustratingly Simple Baseline for Weakly Few-shot Instance Segmentation. In: International Conference on Learning Representations

  8. Doersch C, Gupta A, Zisserman A (2020). CrossTransformers: spatially-aware few-shot transfer. In: Neural Information Processing Systems, 21981–21993

  9. Dong N, Xing E P (2018) Few-Shot Semantic Segmentation with Prototype Learning. In: British Machine Vision Conference

  10. Dong X, Zheng L, Ma F, Yang Y, Meng D (2018) Few-Example Object Detection with Model Communication. IEEE Trans Pattern Anal Mach Intell 41(7):1641–1654

    Article  Google Scholar 

  11. Everingham M, Van Gool L, Williams CK, Winn J, Zisserman A (2010) The Pascal Visual Object Classes (VOC) Challenge. Int J Comput Vision 88(2):303–338

    Article  Google Scholar 

  12. Fan Z, Yu J-G, Liang Z, Ou J, Gao C, Xia G-S, Li Y (2020) FGN: Fully Guided Network for Few-Shot Instance Segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition, 9172–9181

  13. Fan Q, Zhuo W, Tang C-K, Tai Y-W (2020) Few-Shot Object Detection with Attention-RPN and Multi-Relation Detector. In: IEEE Conference on Computer Vision and Pattern Recognition, 4013–4022

  14. Finn C, Abbeel P, Levine S (2017) Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks. In: International Conference on Machine Learning, 1126–1135

  15. Ganea D A, Boom B, Poppe R (2021) Incremental Few-Shot Instance Segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition, 1185–1194

  16. Gidaris S, Komodakis N (2019) Generating Classification Weights with GNN Denoising Autoencoders for Few-Shot Learning. In: IEEE Conference on Computer Vision and Pattern Recognition, 21–30

  17. Gu K, Liu H, Xia Z, Qiao J, Lin W, Thalmann D (2021) PM2.5 Monitoring: Use Information Abundance Measurement and Wide and Deep Learning. IEEE Trans Neural Networks Learn Syst 32(10):4278–4290

    Article  Google Scholar 

  18. Gu K, Xia Z, Qiao J (2020) Stacked Selective Ensemble for PM2.5 Forecast. IEEE Trans Instrum Meas 69(3):660–671

    Article  Google Scholar 

  19. He K, Gkioxari G, Dollár P, Girshick R (2017) Mask R-CNN. In: IEEE International Conference on Computer Vision, 2961–2969

  20. He K, Zhang X, Ren S, Sun J (2016) Deep Residual Learning for Image Recognition. In: IEEE Conference on Computer Vision and Pattern Recognition, 770–778

  21. Kang B, Liu Z, Wang X, Yu F, Feng J, Darrell T (2019) Few-shot Object Detection via Feature Reweighting. In: IEEE International Conference on Computer Vision, 8420–8429

  22. Karlinsky L, Shtok J, Harary S, Schwartz E, Aides A, Feris R, Giryes R, Bronstein A M (2019) RepMet: Representative-based metric learning for classification and few-shot object detection. In: IEEE Conference on Computer Vision and Pattern Recognition, 5197–5206

  23. Kim J, Kim T, Kim S, Yoo C D (2019) Edge-labeling Graph Neural Network for Few-shot Learning. In: IEEE Conference on Computer Vision and Pattern Recognition, 11–20

  24. Koch G, Zemel R, Salakhutdinov R (2015) Siamese Neural Networks for One-shot Image Recognition. In: International Conference on Machine Learning Deep Learning Workshop

  25. Lin T-Y, Dollár P, Girshick R, He K, Hariharan B, Belongie S (2017) Feature Pyramid Networks for Object Detection. In: IEEE Conference on Computer Vision and Pattern Recognition, 2117–2125

  26. Lin T-Y, Maire M, Belongie S, Hays J, Perona P, Ramanan D, Dollár P, Zitnick C L (2014) Microsoft COCO: Common Objects in Context. In: European Conference on Computer Vision, 740–755

  27. Michaelis C, Ustyuzhaninov I, Bethge M, Ecker A S (2018). One-Shot Instance Segmentation. arXiv:1811.11507

  28. Nguyen K, Todorovic S (2021) FAPIS: A Few-shot Anchor-free Part-based Instance Segmenter. In: IEEE Conference on Computer Vision and Pattern Recognition, 11099–11108

  29. Rakelly K, Shelhamer E, Darrell T, Efros A, Levine S (2018). Conditional Networks for Few-Shot Semantic Segmentation. In: International Conference on Learning Representations Workshop

  30. Ravi S, Larochelle H (2016). Optimization as a Model for Few-Shot Learning. In: International Conference on Learning Representations

  31. Redmon J, Divvala S, Girshick R, Farhadi A (2016) You Only Look once: Unified, Real-Time Object Detection. In: IEEE Conference on Computer Vision and Pattern Recognition, 779–788

  32. Ren S, He K, Girshick R, Sun J (2015). Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. In: Neural Information Processing Systems, 91–99

  33. Santoro A, Bartunov S, Botvinick M, Wierstra D, Lillicrap T (2016) Meta-Learning with Memory-Augmented Neural Networks. In: International Conference on Machine Learning, 1842–1850

  34. Satorras V G, Estrach J B (2018) Few-Shot Learning with Graph Neural Networks. In: International Conference on Learning Representations

  35. Shaban A, Bansal S, Liu Z, Essa I, Boots B (2017) One-Shot Learning for Semantic Segmentation. In: British Machine Vision Conference

  36. Snell J, Swersky K, Zemel R (2017). Prototypical Networks for Few-shot Learning. In: Neural Information Processing Systems, 4077–4087

  37. Sun Y, Cheng C, Zhang Y, Zhang C, Zheng L, Wang Z, Wei Y (2020) Circle Loss: A Unified Perspective of Pair Similarity Optimization. In: IEEE Conference on Computer Vision and Pattern Recognition, 6398–6407

  38. Sung F, Yang Y, Zhang L, Xiang T, Torr P H, Hospedales T M (2018) Learning to Compare: Relation Network for Few-Shot Learning. In: IEEE Conference on Computer Vision and Pattern Recognition, 1199–1208

  39. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez A N, Kaiser Ł, Polosukhin I (2017) Attention is All You Need. In: Neural Information Processing Systems, 5998–6008

  40. Vinyals O, Blundell C, Lillicrap T, Wierstra D (2016). Matching Networks for One Shot Learning. In: Neural Information Processing Systems, 3630–3638

  41. Wang X, Girshick R, Gupta A, He K (2018) Non-local Neural Networks. In: IEEE Conference on Computer Vision and Pattern Recognition, 7794–7803

  42. Wang X, Kong T, Shen C, Jiang Y, Li L (2020) SOLO: Segmenting Objects by Locations. In: European Conference on Computer Vision, 649–665

  43. Wang X, Zhang R, Kong T, Li L, Shen C (2020). SOLOv2: Dynamic and Fast Instance Segmentation. In: Neural Information Processing Systems, 17721–17732

  44. Xie E, Sun P, Song X, Wang W, Liu X, Liang D, Shen C, Luo P (2020) PolarMask: Single Shot Instance Segmentation with Polar Representation. In: IEEE Conference on Computer Vision and Pattern Recognition, 12193–12202

  45. Yan X, Chen Z, Xu A, Wang X, Liang X, Lin L (2019) Meta R-CNN: Towards General Solver for Instance-level Low-shot Learning. In: IEEE International Conference on Computer Vision, 9577–9586

  46. Ye H-J, Hu H, Zhan D-C, Sha F (2020) Few-shot Learning via Embedding Adaptation with Set-to-Set Functions. In: IEEE Conference on Computer Vision and Pattern Recognition, 8808–8817

  47. Zhang C, Cai Y, Lin G, Shen C (2020) DeepEMD: Few-Shot Image Classification With Differentiable Earth Mover's Distance and Structured Classifiers. In: IEEE Conference on Computer Vision and Pattern Recognition, 12203–12213

  48. Zhang C, Lin G, Liu F, Guo J, Wu Q, Yao R (2019) Pyramid Graph Networks with Connection Attentions for Region-Based One-Shot Semantic Segmentation. In: IEEE International Conference on Computer Vision, 9587–9595

  49. Zhang X, Wei Y, Yang Y, Huang TS (2020) SG-One: Similarity Guidance Network for One-Shot Semantic Segmentation. IEEE Trans Cybern 50(9):3855–3865

    Article  Google Scholar 

Download references

Funding

This work was supported in part by the National Natural Science Foundation of China (Grant nos. 61971421 and 61976217).

Author information

Authors and Affiliations

  1. School of Computer Science and Technology, China University of Mining & Technology, Xuzhou, China

    Haotian Wang, Shibin Zhou, Xinzheng Xu & Guopeng Zhang

Authors
  1. Haotian Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shibin Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xinzheng Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Guopeng Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shibin Zhou.

Additional information

Publisher's note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, H., Zhou, S., Xu, X. et al. TPN: Triple parts network for few-shot instance segmentation. Multimed Tools Appl 82, 46439–46455 (2023). https://doi.org/10.1007/s11042-023-15624-2

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-15624-2

Keywords

Search

Navigation