Skip to main content
SpringerLink
Log in

Automated Bone Tumor Segmentation and Classification as Benign or Malignant Using Computed Tomographic Imaging

  • Original Paper
  • Published:
Journal of Digital Imaging Aims and scope Submit manuscript

Abstract

The purpose of this study was to pair computed tomography (CT) imaging and machine learning for automated bone tumor segmentation and classification to aid clinicians in determining the need for biopsy. In this retrospective study (March 2005–October 2020), a dataset of 84 femur CT scans (50 females and 34 males, 20 years and older) with definitive histologic confirmation of bone lesion (71% malignant) were leveraged to perform automated tumor segmentation and classification. Our method involves a deep learning architecture that receives a DICOM slice and predicts (i) a segmentation mask over the estimated tumor region, and (ii) a corresponding class as benign or malignant. Class prediction for each case is then determined via majority voting. Statistical analysis was conducted via fivefold cross validation, with results reported as averages along with 95% confidence intervals. Despite the imbalance between benign and malignant cases in our dataset, our approach attains similar classification performances in specificity (75%) and sensitivity (79%). Average segmentation performance attains 56% Dice score and reaches up to 80% for an image slice in each scan. The proposed approach establishes the first steps in developing an automated deep learning method on bone tumor segmentation and classification from CT imaging. Our approach attains comparable quantitative performance to existing deep learning models using other imaging modalities, including X-ray. Moreover, visual analysis of bone tumor segmentation indicates that our model is capable of learning typical tumor characteristics and provides a promising direction in aiding the clinical decision process for biopsy.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Data Availability

Data analyzed in this study is not available upon request.

References

  1. Mundy GR. Metastasis to bone: causes, consequences and therapeutic opportunities. Nature Reviews Cancer. 2002 Aug;2(8):584–93.

    Article  CAS  PubMed  Google Scholar 

  2. Coleman RE. Clinical features of metastatic bone disease and risk of skeletal morbidity. Clinical Cancer Research. 2006 Oct 15;12(20):6243s–9s.

    Article  PubMed  Google Scholar 

  3. Pockett RD, Castellano D, Mcewan P, Oglesby A, Barber BL, Chung K. The hospital burden of disease associated with bone metastases and skeletal-related events in patients with breast cancer, lung cancer, or prostate cancer in Spain. European Journal of Cancer Care. 2009 Aug 26;19(6):755–60.

    Article  Google Scholar 

  4. Cancer Incidence and Survival among Children and Adolescents: United States SEER Program 1975–1995 [Internet]. Test accounts; 1999. Available from: https://doi.org/10.1037/e407432005-001

  5. Siegel RL, Miller KD, Jemal A. Cancer statistics, 2019. CA Cancer J Clin 2019;69(1):7–34.

    Article  PubMed  Google Scholar 

  6. Lodwick GS. A probabilistic approach to the diagnosis of bone tumors. Radiol Clin North Am. 1965 Dec;3(3):487–97.

    Article  CAS  PubMed  Google Scholar 

  7. Resnick D, Kransdorf MJ. Tumors and tumor-like lesions of bone: radiographic principles. Bone and Joint Imaging. 2005;1109–19.

  8. Lietman SA, Joyce MJ. Bone sarcomas: overview of management, with a focus on surgical treatment considerations. Cleveland Clinic Journal of Medicine. 2010 Feb 23;77(Suppl_1):S8–12.

  9. Vanel D, Rimondi E, Vanel M, Gambarotti M, Alberghini M. Solitary bone lesions: which ones to worry about? Cancer Imaging. 2012;12(2):409–13.

    Article  PubMed  PubMed Central  Google Scholar 

  10. Pommersheim WJ, Chew FS. Imaging, diagnosis, and staging of bone tumors: a primer. Seminars in Roentgenology. 2004 Jul;39(3):361–72.

    Article  PubMed  Google Scholar 

  11. Ofluoglu O, Boriani S, Gasbarrini A, De Iure F, Donthineni R. Diagnosis and planning in the management of musculoskeletal tumors: surgical perspective. Seminars in Interventional Radiology. 2010 May 18;27(02):185–90.

    Article  PubMed  PubMed Central  Google Scholar 

  12. Hwang S, Lefkowitz RA, Landa J, Zheng J, Moskowitz CS, Maybody M, et al. Percutaneous CT-guided bone biopsy: diagnosis of malignancy in lesions with initially indeterminate biopsy results and CT features associated with diagnostic or indeterminate results. American Journal of Roentgenology. 2011 Dec;197(6):1417–25.

    Article  PubMed  Google Scholar 

  13. Zamora T, Urrutia J, Schweitzer D, Amenabar PP, Botello E. Do orthopedic oncologists agree on the diagnosis and treatment of cartilage tumors of the appendicular skeleton? Clinical Orthopedics and Related Research®. 2017 Feb 15;475(9):2176–86.

  14. Rozenberg A, Kenneally BE, Abraham JA, Strogus K, Roedl JB, Morrison WB, et al. Clinical impact of second-opinion musculoskeletal subspecialty interpretations during a multidisciplinary orthopedic oncology conference. Journal of the American College of Radiology. 2017 Jul;14(7):931–6.

    Article  PubMed  Google Scholar 

  15. Rozenberg A, Kenneally BE, Abraham JA, Strogus K, Roedl JB, Morrison WB, et al. Second opinions in orthopedic oncology imaging: can fellowship training reduce clinically significant discrepancies? Skeletal Radiology. 2018 Jul 12;48(1):143–7.

    Article  PubMed  Google Scholar 

  16. Caracciolo JT, Temple HT, Letson GD, Kransdorf MJ. A modified Lodwick-Madewell grading system for the evaluation of lytic bone lesions. AJR Am J Roentgenol 2016;207(1):150–156.

    Article  PubMed  Google Scholar 

  17. Mintz DN, Hwang S. Bone tumor imaging, then and now: review article. HSS J 2014;10(3):230–239.

    Article  PubMed  PubMed Central  Google Scholar 

  18. Redondo A, Bagué S, Bernabeu D, et al. Malignant bone tumors (other than Ewing’s): clinical practice guidelines for diagnosis, treatment and follow-up by Spanish Group for Research on Sarcomas (GEIS). Cancer Chemother Pharmacol 2017;80(6):1113–1131.

    Article  PubMed  PubMed Central  Google Scholar 

  19. Yushkevich PA, Gao Y, Gerig G. ITK-SNAP: an interactive tool for semi-automatic segmentation of multi-modality biomedical images. In Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) 2016 (pp. 3342–3345).

  20. Cheng B, Misra I, Schwing AG, Kirillov A, Girdhar R. Masked-attention Mask Transformer for universal image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2022 (pp. 1290–1299).

  21. Zhang Y, Yang Q. An overview of multi-task learning. Natl Sci Rev. 2018;5(1):30-43.

    Article  Google Scholar 

  22. Liu Z, Lin Y, Cao Y, Hu H, Wei Y, Zhang Z, Lin S, Guo B. Swin transformer: hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision 2021 (pp. 10012–10022).

  23. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I. Attention is all you need. Advances in Neural Information Processing Systems. 2017;30.

  24. Bengio, Yoshua, Aaron Courville, and Pascal Vincent. Representation learning: a review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence 35, no. 8 (2013): 1798-1828.

    Article  PubMed  Google Scholar 

  25. Lin, Tsung-Yi, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: common objects in context. In European Conference on Computer Vision, pp. 740–755. Springer, 2014.

  26. Çiçek Ö, Abdulkadir A, Lienkamp SS, Brox T, Ronneberger O. 3D U-Net: learning dense volumetric segmentation from sparse annotation. In International Conference on Medical Image Computing and Computer-Assisted Intervention 2016 Oct 17 (pp. 424–432). Springer, Cham.

  27. Kingma, Diederik P., and Jimmy Ba. Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).

  28. Krogh, Anders, and John Hertz. A simple weight decay can improve generalization. Advances in Neural Information Processing Systems 4 (1991).

  29. Pedregosa F, Varoquaux G, Gramfort A, et al. Scikit-learn: machine learning in Python. The Journal of machine Learning research 2011;12:2825-2830.

    Google Scholar 

  30. He, K., Gkioxari, G., Dollár, P. and Girshick, R., 2017. Mask r-cnn. In Proceedings of the IEEE International Conference on Computer Vision (pp. 2961–2969).

  31. F. N. Iandola, M. W. Moskewicz, S. Karayev, R. B. Girshick, T. Darrell, and K. Keutzer, DenseNet: implementing efficient ConvNet descriptor pyramids technical report, 2014.

  32. N. -H. Ho, H. -J. Yang, S. -H. Kim, S. T. Jung and S. -D. Joo, Regenerative semi-supervised bidirectional W-network-based knee bone tumor classification on radiographs guided by three-region bone segmentation, in IEEE Access, vol. 7, pp. 154277-154289, 2019, doi: https://doi.org/10.1109/ACCESS.2019.2949125.

    Article  Google Scholar 

  33. Bandyopadhyay O, Biswas A, Bhattacharya BB. Bone-cancer assessment and destruction pattern analysis in long-bone X-ray image. J Digit Imaging. 2019 Apr;32(2):300-313. doi: https://doi.org/10.1007/s10278-018-0145-0. PMID: 30367308

    Article  PubMed  Google Scholar 

  34. Xu, R., Kido, S., Suga, K. et al. Texture analysis on 18F-FDG PET/CT images to differentiate malignant and benign bone and soft-tissue lesions. Ann Nucl Med 28, 926–935 (2014). https://doi.org/https://doi.org/10.1007/s12149-014-0895-9

    Article  CAS  PubMed  Google Scholar 

  35. N. Moreau et al., Deep learning approaches for bone and bone lesion segmentation on 18FDG PET/CT imaging in the context of metastatic breast cancer, 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), 2020, pp. 1532–1535, doi: https://doi.org/10.1109/EMBC44109.2020.9175904.

  36. Hossain E, Hossain MF, Rahaman MA. An approach for the detection and classification of tumor cells from bone MRI using wavelet transform and KNN classifier. In International Conference on Innovation in Engineering and Technology (ICIET) 2018 Dec 27 (pp. 1–6). IEEE.

  37. Wu J, Guo Y, Gou F, Dai Z. A medical assistant segmentation method for MRI images of osteosarcoma based on DecoupleSegNet. International Journal of Intelligent Systems. 2022 Jul 6.

  38. Eweje FR, Bao B, Wu J, Dalal D, Liao WH, He Y, Luo Y, Lu S, Zhang P, Peng X, Sebro R. Deep learning for classification of bone lesions on routine MRI. EBioMedicine. 2021 Jun 1;68:103402.

    Article  Google Scholar 

  39. Liu X, Han C, Cui Y, Xie T, Zhang X, Wang X. Detection and segmentation of pelvic bones metastases in MRI images for patients with prostate cancer based on deep learning. Frontiers in Oncology. 2021 Nov 29:4984.

    Google Scholar 

  40. Georgeanu VA, Mămuleanu M, Ghiea S, Selișteanu D. Malignant bone tumors diagnosis using magnetic resonance imaging based on deep learning algorithms. Medicina. 2022 May 4;58(5):636.

    Article  PubMed  PubMed Central  Google Scholar 

  41. Costelloe CM, Madewell JE. Radiography in the initial diagnosis of primary bone tumors. American Journal of Roentgenology. 2013 Jan;200(1):3-7.

    Article  PubMed  Google Scholar 

  42. Catal Reis H, Bayram B, Seker DZ. Bone tumor segmentation by gradient operators from CT images. Selcuk International Scientific Conference on Applied Sciences (The Selçuk ISCAS 2016) 2016 Sep (Vol. 27, p. 30).

  43. Sun W, Liu S, Guo J, Liu S, Hao D, Hou F, Wang H, Xu W. A CT-based radiomics nomogram for distinguishing between benign and malignant bone tumors. Cancer imaging. 2021 Dec;21(1):1-0.

    Article  Google Scholar 

  44. Do N-T, Jung S-T, Yang H-J, Kim S-H. Multi-level Seg-U-Net model with global and patch-based X-ray images for knee bone tumor detection. Diagnostics. 2021; 11(4):691. https://doi.org/https://doi.org/10.3390/diagnostics11040691

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  45. von Schacky CE, Wilhelm NJ, Schäfer VS, Leonhardt Y, Gassert FG, Foreman SC, Gassert FT, Jung M, Jungmann PM, Russe MF, Mogler C. Multitask deep learning for segmentation and classification of primary bone tumors on radiographs. Radiology. 2021 Nov;301(2):398-406.

    Article  Google Scholar 

  46. He Y, Pan I, Bao B, et al. Deep learning-based classification of primary bone tumors on radiographs: a preliminary study. EBioMedicine 2020;62:103121.

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Funding

The research reported in this publication was supported by the National Cancer Institute of the National Institutes of Health under Award Number R43CA254835. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

Author information

Authors and Affiliations

  1. BioSensics LLC, 57 Chapel Street, Newton, MA, 02458, USA

    Ilkay Yildiz Potter, Aidin Vaziri & Ashkan Vaziri

  2. Beth Israel Deaconess Medical Center (BIDMC), Harvard Medical School, 330 Brookline Avenue, Boston, MA, 02215, USA

    Diana Yeritsyan, Sarah Mahar, Jim Wu & Ara Nazarian

Authors
  1. Ilkay Yildiz Potter
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Diana Yeritsyan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sarah Mahar
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jim Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ara Nazarian
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Aidin Vaziri
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Ashkan Vaziri
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All of the listed authors have participated actively in the entire study project. Ashkan Vaziri, Ara Nazarian, and Jim Wu developed the design and conduct of the study. Ara Nazarian and Jim Wu led the data collection. Diana Yeritsyan and Sarah Mahar aided in the data collection. Aidin Vaziri annotated imaging data. Ilkay Yildiz Potter performed data analysis. The first draft of the manuscript was written by Ilkay Yildiz Potter and all authors commented on previous versions of the manuscript. All authors participated in and approved the final submission.

Corresponding author

Correspondence to Ilkay Yildiz Potter.

Ethics declarations

Ethical Approval

This retrospective study was approved by the Institutional Review Board, in compliance with the Health Information Portability and Accountability Act, and all data was collected at the institution (BIDMC Division of Musculoskeletal Imaging & Intervention). Informed consent was obtained from all individual participants included in the study.

Competing Interests

Ilkay Yildiz Potter, Diana Yeritsyan, Sarah Mahar, and Aidin Vaziri declare that they have no financial interests. Ashkan Vaziri, Ara Nazarian, and Jim Wu received the research grant R43CA254835 as investigators. Ara Nazarian is also a consultant with BioSensics, LLC, on an unrelated project.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yildiz Potter, I., Yeritsyan, D., Mahar, S. et al. Automated Bone Tumor Segmentation and Classification as Benign or Malignant Using Computed Tomographic Imaging. J Digit Imaging 36, 869–878 (2023). https://doi.org/10.1007/s10278-022-00771-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10278-022-00771-z

Keywords

Search

Navigation