Abstract
The Segment Anything Model (SAM) has demonstrated remarkable capabilities of scaled-up segmentation models, enabling zero-shot generalization across a variety of domains. By leveraging large-scale foundational models as pre-trained models, it is a natural progression to fine-tune SAM for specific domains to further enhance performances. However, the adoption of foundational models in the medical domain presents a challenge due to the difficulty and expense of labeling sufficient data for adaptation within hospital systems. In this paper, we introduce an efficient and practical approach for fine-tuning SAM using a limited number of exemplars, making it suitable for such scenarios. Our approach combines two established techniques from the literature: an exemplar-guided synthesis module and the widely recognized Low-Rank Adaptation (LoRA) fine-tuning strategy, serving as data-level and model-level attempts respectively. Interestingly, our empirical findings suggest that SAM can be effectively aligned within the medical domain even with few labeled data. We validate our approach through experiments on brain tumor segmentation (BraTS) and multi-organ CT segmentation (Synapse). The comprehensive results underscore the feasibility and effectiveness of such an approach, paving the way for the practical application of SAM in the medical domain.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Bakas, S., et al.: Advancing the cancer genome atlas glioma MRI collections with expert segmentation labels and radiomic features. Sci. Data 4(1), 1–13 (2017)
Bakas, S., et al.: Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the brats challenge. arXiv preprint arXiv:1811.02629 (2018)
Bommasani, R., et al.: On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 (2021)
Brown, T., et al.: Language models are few-shot learners. Adv. Neural. Inf. Process. Syst. 33, 1877–1901 (2020)
Chaitanya, K., Erdil, E., Karani, N., Konukoglu, E.: Contrastive learning of global and local features for medical image segmentation with limited annotations. Adv. Neural. Inf. Process. Syst. 33, 12546–12558 (2020)
Chen, J., et al.: Transunet: transformers make strong encoders for medical image segmentation. arXiv preprint arXiv:2102.04306 (2021)
Chen, T., et al.: Sam fails to segment anything?–sam-adapter: adapting sam in underperformed scenes: camouflage, shadow, and more. arXiv preprint arXiv:2304.09148 (2023)
Chowdhery, A., et al.: Palm: scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 (2022)
En, Q., Guo, Y.: Exemplar learning for medical image segmentation. arXiv preprint arXiv:2204.01713 (2022)
Ghorbel, A., Aldahdooh, A., Albarqouni, S., Hamidouche, W.: Transformer based models for unsupervised anomaly segmentation in brain mr images. arXiv preprint arXiv:2207.02059 (2022)
Hu, E.J., et al.: Lora: low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 (2021)
Huang, Y., et al.: Segment anything model for medical images? arXiv preprint arXiv:2304.14660 (2023)
Ke, L., et al.: Segment anything in high quality. arXiv preprint arXiv:2306.01567 (2023)
Kirillov, A., et al.: Segment anything. arXiv preprint arXiv:2304.02643 (2023)
Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017)
Ma, J., He, Y., Li, F., Han, L., Chenyu, Y., Wang, B.: Segment anything in medical images. arXiv preprint arXiv:2304.12306 (2023)
Menze, B.H., et al.: The multimodal brain tumor image segmentation benchmark (brats). IEEE Trans. Med. Imaging 34(10), 1993–2024 (2014)
Oktay, O., et al.: Attention u-net: learning where to look for the pancreas. arXiv preprint arXiv:1804.03999 (2018)
Vaswani, A., et al.: Attention is all you need. Adv. Neural Inf. Process. Syst. 30 (2017)
Wang, S., et al.: Annotation-efficient deep learning for automatic medical image segmentation. Nat. Commun. 12(1), 5915 (2021)
Zhang, K., Liu, D.: Customized segment anything model for medical image segmentation. arXiv preprint arXiv:2304.13785 (2023)
Acknowledgement
The work described in this paper was partially supported by grants from the National Natural Science Fund (62201483) and the Research Grants Council of the Hong Kong Special Administrative Region, China (T45-401/22-N).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Feng, W., Zhu, L., Yu, L. (2024). Cheap Lunch for Medical Image Segmentation by Fine-Tuning SAM on Few Exemplars. In: Baid, U., et al. Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries. BrainLes SWITCH 2023 2023. Lecture Notes in Computer Science, vol 14668. Springer, Cham. https://doi.org/10.1007/978-3-031-76160-7_2
Download citation
DOI: https://doi.org/10.1007/978-3-031-76160-7_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-76159-1
Online ISBN: 978-3-031-76160-7
eBook Packages: Computer ScienceComputer Science (R0)
