Skip to main content
Springer Nature Link
Log in

Lite-SAM Is Actually What You Need for Segment Everything

  • Conference paper
  • First Online:
Computer Vision – ECCV 2024 (ECCV 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 15092))

Included in the following conference series:

  • 259 Accesses

Abstract

The Segment Anything model (SAM) has brought significant changes to the segmentation field with its superior performance, but its extensive computational resource requirements remain a limiting factor. Many works, such as MobileSAM, Edge-SAM, and MobileSAM-v2, have explored lightweight solutions. However, their use of traditional Grid Search sampling strategies or two-stage concatenation methods, which do not allow for end-to-end training, severely limit the performance of segment everything (SegEvery).

This paper introduces Lite-SAM, an efficient end-to-end solution for the SegEvery task designed to reduce computational costs and redundancy. Lite-SAM is composed of four main components: a streamlined CNN-Transformer hybrid encoder (LiteViT), an automated prompt proposal network (AutoPPN), a traditional prompt encoder, and a mask decoder. All these components are integrated within the SAM framework. Our LiteViT, a high-performance lightweight backbone network, has only 1.16M parameters, which is a 23\(\%\) reduction compared to the lightest existing backbone network Shufflenet. We also introduce AutoPPN, an innovative end-to-end method for prompt boxes and points generation. This is an improvement over traditional grid search sampling methods, and its unique design allows for easy integration into any SAM series algorithm, extending its usability.

We have thoroughly benchmarked Lite-SAM across a plethora of both public and private datasets. The evaluation encompassed a broad spectrum of universal metrics, including the number of parameters, SegEvery execution time, and accuracy. The findings reveal that Lite-SAM, operating with a lean 4.2M parameters, significantly outpaces its counterparts, demonstrating performance improvements of 43x, 31x, 20x, 21x, and 1.6x over SAM, MobileSAM, Edge-SAM, EfficientViT-SAM, and MobileSAM-v2 respectively, all the while maintaining competitive accuracy. This underscores Lite-SAM’s prowess in achieving an optimal equilibrium between performance and precision, thereby setting a new state-of-the-art (SOTA) benchmark in the domain.

J. Fu and Y. Yu—Equal contribution.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Arbelaez, P., Maire, M., Fowlkes, C., Malik, J.: Contour detection and hierarchical image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 33(5), 898–916 (2010)

    Article  Google Scholar 

  2. Bommasani, R., et al.: On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 (2021)

  3. Brown, T., et al.: Language models are few-shot learners. In: Advances in Neural Information Processing Systems, vol. 33, pp. 1877–1901 (2020)

    Google Scholar 

  4. Cai, H., Li, J., Hu, M., Gan, C., Han, S.: EfficientViT: multi-scale linear attention for high-resolution dense prediction (2023)

    Google Scholar 

  5. Canny, J.: A computational approach to edge detection (1986)

    Google Scholar 

  6. Dongcai, S.: Efficient graph based image segmentation. Image Processing (2004)

    Google Scholar 

  7. Dosovitskiy, A., et al.: An image is worth 16x16 words: transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)

  8. Gupta, A., Dollar, P., Girshick, R.: LVIS: a dataset for large vocabulary instance segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5356–5364 (2019)

    Google Scholar 

  9. He, K., Fan, H., Wu, Y., Xie, S., Girshick, R.: Momentum contrast for unsupervised visual representation learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729–9738 (2020)

    Google Scholar 

  10. Howard, A., et al.: Searching for MobileNetV3. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1314–1324 (2019)

    Google Scholar 

  11. Howard, A.G., et al.: MobileNets: efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 (2017)

  12. Jocher, G., Chaurasia, A., Qiu, J.: YOLO by Ultralytics (2023). https://github.com/ultralytics/ultralytics

  13. Kirillov, A., et al.: Segment anything. arXiv preprint arXiv:2304.02643 (2023)

  14. Law, H., Deng, J.: CornerNet: detecting objects as paired keypoints (2019)

    Google Scholar 

  15. Li, F., et al.: Semantic-SAM: segment and recognize anything at any granularity. arXiv preprint arXiv:2307.04767 (2023)

  16. Li, J., et al.: Next-ViT: next generation vision transformer for efficient deployment in realistic industrial scenarios. arXiv preprint arXiv:2207.05501 (2022)

  17. Li, Y., et al.: EfficientFormer: vision transformers at MobileNet speed. In: Advances in Neural Information Processing Systems, vol. 35, pp. 12934–12949 (2022)

    Google Scholar 

  18. Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollár, P.: Focal loss for dense object detection (2018)

    Google Scholar 

  19. Lin, T.-Y., et al.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10602-1_48

    Chapter  Google Scholar 

  20. Liu, X., Peng, H., Zheng, N., Yang, Y., Hu, H., Yuan, Y.: EfficientViT: memory efficient vision transformer with cascaded group attention. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14420–14430 (2023)

    Google Scholar 

  21. Ma, N., Zhang, X., Zheng, H.-T., Sun, J.: ShuffleNet V2: practical guidelines for efficient CNN architecture design. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) Computer Vision – ECCV 2018. LNCS, vol. 11218, pp. 122–138. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01264-9_8

    Chapter  Google Scholar 

  22. Martin, D., Fowlkes, C., Tal, D., Malik, J.: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: Proceedings Eighth IEEE International Conference on Computer Vision, ICCV 2001. vol. 2, pp. 416–423. IEEE (2001)

    Google Scholar 

  23. Mehta, S., Rastegari, M.: MobileViT: light-weight, general-purpose, and mobile-friendly vision transformer. arXiv preprint arXiv:2110.02178 (2021)

  24. Mehta, S., Rastegari, M.: MobileViT: light-weight, general-purpose, and mobile-friendly vision transformer (2022)

    Google Scholar 

  25. Pu, M., Huang, Y., Liu, Y., Guan, Q., Ling, H.: EDTER: edge detection with transformer (2022)

    Google Scholar 

  26. Qiao, Y., et al.: MP-FedCL: multi-prototype federated contrastive learning for edge intelligence. arXiv preprint arXiv:2304.01950 (2023)

  27. Radford, A., Narasimhan, K., Salimans, T., Sutskever, I., et al.: Improving language understanding by generative pre-training (2018)

    Google Scholar 

  28. Ren, T., et al.: Grounded SAM: assembling open-world models for diverse visual tasks (2024)

    Google Scholar 

  29. Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L.C.: MobileNetV2: inverted residuals and linear bottlenecks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4510–4520 (2018)

    Google Scholar 

  30. Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., Jégou, H.: Training data-efficient image transformers & distillation through attention. In: International Conference on Machine Learning, pp. 10347–10357. PMLR (2021)

    Google Scholar 

  31. Vasu, P.K.A., Gabriel, J., Zhu, J., Tuzel, O., Ranjan, A.: FastViT: a fast hybrid vision transformer using structural reparameterization. arXiv preprint arXiv:2303.14189 (2023)

  32. Wang, X., Zhang, X., Cao, Y., Wang, W., Shen, C., Huang, T.: SegGPT: segmenting everything in context (2023). https://arxiv.org/abs/2304.03284

  33. Wu, K., et al.: TinyViT: fast pretraining distillation for small vision transformers. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds.) ECCV 2022. LNCS, vol. 13681, pp. 68–85. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-19803-8_5

    Chapter  Google Scholar 

  34. Xie, S., Tu, Z.: Holistically-nested edge detection (2015)

    Google Scholar 

  35. Xiong, Y., et al.: EfficientSAM: leveraged masked image pretraining for efficient segment anything (2023)

    Google Scholar 

  36. Yang, Y., Wu, X., He, T., Zhao, H., Liu, X.: SAM3D: segment anything in 3D scenes (2023). https://arxiv.org/abs/2306.03908

  37. Yu, T., et al.: Inpaint anything: segment anything meets image inpainting (2023). https://arxiv.org/abs/2304.06790

  38. Yu, W., et al.: MetaFormer is actually what you need for vision (2022)

    Google Scholar 

  39. Zhang, C., et al.: Faster segment anything: towards lightweight SAM for mobile applications. arXiv preprint arXiv:2306.14289 (2023)

  40. Zhang, C., Han, D., Zheng, S., Choi, J., Kim, T.H., Hong, C.S.: MobileSAMv2: faster segment anything to everything (2023)

    Google Scholar 

  41. Zhang, C., et al.: One small step for generative AI, one giant leap for AGI: a complete survey on ChatGPT in AIGC era. arXiv preprint arXiv:2304.06488 (2023)

  42. Zhang, C., et al.: A complete survey on generative AI (AIGC): is ChatGPT from GPT-4 to GPT-5 all you need? arXiv preprint arXiv:2303.11717 (2023)

  43. Zhang, C., et al.: Dual temperature helps contrastive learning without many negative samples: towards understanding and simplifying MOCO. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14441–14450 (2022)

    Google Scholar 

  44. Zhang, C., Zhang, K., Zhang, C., Pham, T.X., Yoo, C.D., Kweon, I.S.: How does SimSiam avoid collapse without negative samples? A unified understanding with self-supervised contrastive learning. arXiv preprint arXiv:2203.16262 (2022)

  45. Zhang, X., Zhou, X., Lin, M., Sun, J.: ShuffleNet: an extremely efficient convolutional neural network for mobile devices. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6848–6856 (2018)

    Google Scholar 

  46. Zhao, X., et al.: Fast segment anything. arXiv preprint arXiv:2306.12156 (2023)

  47. Zhou, C., Li, X., Loy, C.C., Dai, B.: EdgeSAM: prompt-in-the-loop distillation for on-device deployment of SAM (2023)

    Google Scholar 

  48. Zhou, X., Wang, D., Krähenbühl, P.: Objects as points. arXiv preprint arXiv:1904.07850 (2019)

  49. Zou, X., et al.: Segment everything everywhere all at once (2023). https://arxiv.org/abs/2304.06718

Download references

Acknowledgements

This work is supported by Zhejiang Dahua Technology Co., Ltd. and Zhejiang University.

Author information

Authors and Affiliations

  1. Zhejiang Dahua Technology Co., Ltd., Hangzhou, China

    Jianhai Fu, Yuanjie Yu, Ningchuan Li, Yi Zhang, Qichao Chen & Jianping Xiong

  2. Zhejiang University, Hangzhou, China

    Yuanjie Yu, Jun Yin & Zhiyu Xiang

Authors
  1. Jianhai Fu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yuanjie Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ningchuan Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yi Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Qichao Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Jianping Xiong
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Jun Yin
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Zhiyu Xiang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Ningchuan Li or Zhiyu Xiang .

Editor information

Editors and Affiliations

  1. University of Birmingham, Birmingham, UK

    Aleš Leonardis

  2. University of Trento, Trento, Italy

    Elisa Ricci

  3. Technical University of Darmstadt, Darmstadt, Germany

    Stefan Roth

  4. Princeton University, Princeton, NJ, USA

    Olga Russakovsky

  5. Czech Technical University in Prague, Prague, Czech Republic

    Torsten Sattler

  6. École des Ponts ParisTech, Marne-la-Vallée, France

    Gül Varol

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 3222 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Fu, J. et al. (2025). Lite-SAM Is Actually What You Need for Segment Everything. In: Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., Varol, G. (eds) Computer Vision – ECCV 2024. ECCV 2024. Lecture Notes in Computer Science, vol 15092. Springer, Cham. https://doi.org/10.1007/978-3-031-72754-2_26

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-72754-2_26

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-72753-5

  • Online ISBN: 978-3-031-72754-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics