Skip to main content

Advertisement

SpringerLink
Log in

BUS-net: a bimodal ultrasound network for breast cancer diagnosis

  • Original Article
  • Published:
International Journal of Machine Learning and Cybernetics Aims and scope Submit manuscript

Abstract

Ultrasound (US) is a fast and non-invasive imaging approach, which is recommended by current practice guidelines as early breast cancer screening. B-mode and contrast-enhanced ultrasound(CEUS) are two US modalities proven to detect abnormal tissues. However, no one has been able to make full use of these two modal ultrasound data characteristically to complete the classification problem. This paper proposed a bimodal ultrasound network (BUS-Net) capable of simultaneously dealing with the B-mode US and CEUS video. In the CEUS branch, We use seven CEUS pathological characteristics as multiple labels instead of the traditional two labels (benign and malignant) to extract the pathological semantic representative features. The model can be more general and robust by transforming the binary learning task into a multi-class learning task. In the B-mode US branch, we use a group of shape descriptors to identify hard samples with abnormal morphology. A shape constraint loss term is proposed to impose the shape constraints in the training phase and enhance its distinguish ability for hard samples. Finally, the two modal ultrasound data features are fused to realize the classification of benign and malignant tumors. Our experiments show that the classification accuracy is significantly improved using our bimodal strategy. Compared with existing breast ultrasound classification methods, our method increased by an average of 3 percentage points in each evaluation index, and the TNR and AUC index both exceeded 92%. This also demonstrates that our approach can more accurately classify ultrasound images with more complex imaging. In general, the bimodal ultrasound network proposed in this paper, which integrates bimodal data features, further improves the classification ability of the model. The multi-label learning task of the CEUS branch enhances the robustness of the model. The shape constraint loss term of the B-mode US branch improves its ability to distinguish between hard samples. The algorithm in this paper has good clinical guidance value.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

References

  1. An S, Liu J, Gao Y, Zhao X, Hou L, Xie T (2012) Comparative study of contrastenhanced ultrasound qualitative and quantitative analysis for identifying benign and malignant breast tumor lumps. Chin J Ultrasonogr 021(6):492–495

    Google Scholar 

  2. Carreira J, Zisserman A, Vadis Q (2017) action recognition? a new model and the kinetics dataset. In : Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 6299–6308

  3. Chen C-M, Chou Y-H, Han K-C, Hung G-S, Cm T, Hong-Jen C, See-Ying C (2003) Breast lesions on sonograms: computer-aided diagnosis with nearly setting-independent features and artificial neural networks. Radiology 226(504–14):03

    Google Scholar 

  4. Chiao J-Y, Chen K-Y, Liao KY-K, Hsieh P-H, Huang T-C (2019) Detection and classification the breast tumors using mask r-cnn on sonograms. Medicine 98(19):e15200

    Article  Google Scholar 

  5. Cox K, Sever A, Jones S, Weeks J, Mills P, Devalia H, Fish D, Jones P (2013) Validation of a technique using microbubbles and contrast enhanced ultrasound (ceus) to biopsy sentinel lymph nodes (sln) in pre-operative breast cancer patients with a normal grey-scale axillary ultrasound. Eur J Surg Oncol (EJSO) 39(7):760–765

    Article  Google Scholar 

  6. Deng J, Guo J, Xue N, Zafeiriou S (2019) Arcface: additive angular margin loss for deep face recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 4690–4699

  7. Diba A, Fayyaz M, Sharma V, Karami AH, Arzani MM, Yousefzadeh R , Van Gool L (2017) Temporal 3d convnets: New architecture and transfer learning for video classification. arXiv preprintarXiv:1711.08200

  8. Fang L, Huang B-J, Ding H, Mao F, Li C-L, Zeng M-S, Zhou J-J, Chen Y, Wang W-P (2019) Contrast-enhanced ultrasound (ceus) for the diagnosis of hypoechoic hepatic hemangioma in clinical practice. Clin Hemorheol Microcirc 72(4):395–405

    Article  Google Scholar 

  9. Fernando KR, Tsokos CP (2021) Dynamically weighted balanced loss: class imbalanced learning and confidence calibration of deep neural networks. IEEE Trans Neural Netw Learn Syst. https://doi.org/10.1109/TNNLS.2020.3047335

    Article  Google Scholar 

  10. Fu S, Ruan Q, Wang W, Li Yu (2008) Adaptive anisotropic diffusion for ultrasonic image denoising and edge enhancement. Int J Comput Inf Eng 2(11):3973–3976

    Google Scholar 

  11. González-Luna FA, Hernández-López J, Gómez-Flores W (2019) A performance evaluation of machine learning techniques for breast ultrasound classification. In: 2019 16th International Conference on Electrical Engineering, Computing Science and Automatic Control (CCE), IEEE, pp 1–5

  12. Guo L-H, Wang D, Qian Y-Y, Zheng X, Zhao C-K, Li X-L, Bo X-W, Yue W-W, Zhang Q, Shi J et al (2018) A two-stage multi-view learning framework based computer-aided diagnosis of liver tumors with contrast enhanced ultrasound images. Clin Hemorheol Microcirc 69(3):343–354

    Article  Google Scholar 

  13. Hassan Tarek M, Mohammed Elmogy, El-Sayed Sallam (2017) Diagnosis of focal liver diseases based on deep learning technique for ultrasound images. Arab J Sci Eng 42(8):3127–3140

    Article  Google Scholar 

  14. He K, Gkioxari G, Dollár P, Girshick R (2017) Mask r-cnn. In: Proceedings of the IEEE International Conference on Computer Vision, pp 2961–2969,

  15. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 770–778

  16. Huang J, Chan PSF, Lok V, Chen X, Ding H, Jin Y, Yuan J, Lao X, Zheng Z-J, Wong MCS (2021) Global incidence and mortality of breast cancer: a trend analysis. Aging (Albany NY) 13(4):5748

    Article  Google Scholar 

  17. Jamieson AR, Drukker K, Giger ML (2012) Breast image feature learning with adaptive deconvolutional networks. Med Imaging 8315:831506

    Google Scholar 

  18. Joo S, Yang YS, Moon WK, Kim HC (2004) Computer-aided diagnosis of solid breast nodules: Use of an artificial neural network based on multiple sonographic features. IEEE Trans Med Imaging 23(10):1292–1300

    Article  Google Scholar 

  19. Kong X, Tan T, Bao L, Wang G (2018) Classification of breast mass in 3D ultrasound images with annotations based on convolutional neural networks. Chin J Biomed Eng 37(4):414–422

    Google Scholar 

  20. Leng X, Huang G, Yao L, Ma F (2015) Role of multi-mode ultrasound in the diagnosis of level 4 bi-rads breast lesions and logistic regression model. Int J Clin Exp Med 8(9):15889

    Google Scholar 

  21. Li Y, Yuan L, Mengke Z, Guanglei Z, Zhili W, Jianwen L (2019) Radiomics with attribute bagging for breast tumor classification using multimodal ultrasound images. J Ultrasound Med 39:361–371

    Article  Google Scholar 

  22. Miyamoto Y, Ito T, Takada E, Omoto K, Hirai T, Moriyasu F (2014) Efficacy of sonazoid (perflubutane) for contrast-enhanced ultrasound in the differentiation of focal breast lesions: phase 3 multicenter clinical trial. Am J Roentgenol 202(4):W400–W407

    Article  Google Scholar 

  23. Mohammed MA, Al-Khateeb B, Rashid AN, Ibrahim DA, Abd GMK, Mostafa SA (2018) Neural network and multi-fractal dimension features for breast cancer classification from ultrasound images. Comput Electr Eng 70:871–882

    Article  Google Scholar 

  24. Qi X, Lei Z, Yao C, Yong P, Yi C, Qing L, Zhang Y (2018) Automated diagnosis of breast ultrasonography images using deep neural networks. Med Image Anal 52:185–198

    Article  Google Scholar 

  25. Qin L, Yin H, Zhuang H, Luo Y, Liu P, Liu DC (2019) Classification for rectal ceus images based on combining features by transfer learning. In: Proceedings of the Third International Symposium on Image Computing and Digital Medicine, pp 187–191

  26. Qiu Z, Yao T, Mei T (2017) Learning spatio-temporal representation with pseudo-3d residual networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp 5533–5541

  27. Rafailidis V, Sidhu PS (2018) Vascular ultrasound, the potential of integration of multiparametric ultrasound into routine clinical practice. Ultrasound 26(3):136–144

    Article  Google Scholar 

  28. Ronneberger O, Fischer P (2015) And Thomas Brox. Convolutional networks for biomedical image segmentation, U-net

  29. Rundo L, Militello C, Vitabile S, Russo G, Sala E, Gilardi MC (2020) A survey on nature-inspired medical image analysis: a step further in biomedical data integration. Fund Inform 171(1–4):345–365

    MathSciNet  Google Scholar 

  30. Saracco A, Szabo BK, Aspelin P, Leifland K, Tánczos E, Wilczek B, Axelsson R (2015) Contrast-enhanced ultrasound using real-time contrast harmonic imaging in invasive breast cancer: comparison of enhancement dynamics with three different doses of contrast agent. Acta Radiol 56(1):34–41

    Article  Google Scholar 

  31. Selvaraju RR, Cogswell M, Das A, Vedantam R, Parikh D, Batra D (2017) Grad-cam: visual explanations from deep networks via gradient-based localization. In: The IEEE International Conference on Computer Vision (ICCV)

  32. Shen R, Yang C, Luo X, Nian Y, Tang L, Chen Q, Luo J, Lv Z, Cheng Y (2018) Retrospective study of qualitative features of ultrasound contrast-enhanced breast lesions based on multicenter data from china. Chin Med J Imaging 26(12):885–889

    Google Scholar 

  33. Shi J, Zhou S, Liu X, Zhang Q, Minhua L, Wang T (2016) Stacked deep polynomial network based representation learning for tumor classification with small ultrasound image dataset. Neurocomputing 194:87–94

    Article  Google Scholar 

  34. Shin SY, Lee S, Yun ID, Kim SM, Lee KM (2018) Joint weakly and semi-supervised deep learning for localization and classification of masses in breast ultrasound images. IEEE Trans Med Imaging 38(3):762–774

    Article  Google Scholar 

  35. Simonyan K, Zisserman A (2014) Two-stream convolutional networks for action recognition in videos. Advances in neural information processing systems. Springer, Berlin, pp 568–576

    Google Scholar 

  36. Singh BK, Verma K, Thoke AS (2016) Fuzzy cluster based neural network classifier for classifying breast tumors in ultrasound images. Expert Syst Appl 66:114–123

    Article  Google Scholar 

  37. Suckling J, Parker J, Dance DR (1994) Themammographic image analysis society digital mammogram database

  38. Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z (2016) Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2818–2826

  39. Tran D, Bourdev L, Fergus R, Torresani L, Paluri M (2015) Learning spatiotemporal features with 3D convolutional networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp 4489–4497

  40. Tran D, Wang H, Torresani L, Ray J, LeCun Y, Paluri M (2018) A closer look at spatiotemporal convolutions for action recognition. In: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp 6450–6459

  41. Van Esser S, Veldhuis WB, Van Hillegersberg R, Van Diest PJ, Stapper G, ElOuamari M, Borel Rinkes IHM, Th Mali WPM, van den Bosch MAAJ (2007) Accuracy of contrast-enhanced breast ultrasound for pre-operative tumor size assessment in patients diagnosed with invasive ductal carcinoma of the breast. Cancer Imaging 7(1):63

    Article  Google Scholar 

  42. Varghese BA, Lee S, Cen S, Talebi A, Mohd P, Stahl D, Perkins M, Desai B, Duddalwar VA, Larsen LH (2022) Characterizing breast masses using an integrative framework of machine learning and CEUS-based radiomics. J Ultrasound

  43. Wagner J (2006) Pain-free breast biopsy: myth or reality? Decis Imaging Econ

  44. Wang Y, Choi EJ, Choi Y, Zhang H, Jin GY, Ko SB (2020) Breast cancer classification in automated breast ultrasound using multiview convolutional neural network with transfer learning. Ultrasound Med Biol 46(5):1119–32

    Article  Google Scholar 

  45. Wang YM, Fan W, Zhao S, Zhang K, Zhang L, Zhang P, Ma R (2016) Qualitative, quantitative and combination score systems in differential diagnosis of breast lesions by contrast-enhanced ultrasound. Eur J Radiol 85(1):48–54

    Article  Google Scholar 

  46. Wu K, Chen X, Ding M (2014) Deep learning based classification of focal liver lesions with contrast-enhanced ultrasound. Optik Int Jr Light Electron Opt 125(15):4057–4063

    Article  Google Scholar 

  47. Wubulihasimu M, Maimaitusun M, Xu X-L, Liu X-D, Luo B-M (2018) The added value of contrast-enhanced ultrasound to conventional ultrasound in differentiating benign and malignant solid breast lesions: a systematic review and meta-analysis. Clin Radiol 73(11):936–943

    Article  Google Scholar 

  48. Xia H-S, Wang X, Ding H, Wen J-X, Fan P-L, Wang W-P (2014) Papillary breast lesions on contrast-enhanced ultrasound: morphological enhancement patterns and diagnostic strategy. Eur Radiol 24(12):3178–3190

    Article  Google Scholar 

  49. Xiang L-H, Yao M-H, Guang X, Huan P, Liu H, Fang Y, Rong W (2017) Diagnostic value of contrast-enhanced ultrasound and shear-wave elastography for breast lesions of sub-centimeter. Clin Hemorheol Microcirc 67(1):69–80

    Article  Google Scholar 

  50. Xiao T, Liu L, Li K, Qin W, Yu S, Li Z (2018) Comparison of transferred deep neural networks in ultrasonic breast masses discrimination. Biomed Res Int

  51. Xie F, Zhang D, Cheng L, Lei Yu, Yang L, Tong F, Liu H, Wang S, Wang S (2015) Intradermal microbubbles and contrast-enhanced ultrasound (ceus) is a feasible approach for sentinel lymph node identification in early-stage breast cancer. World J Surg Oncol 13(1):319

    Article  Google Scholar 

  52. Xie S, Girshick R, Dollár P, Tu Z, He K (2017) Aggregated residual transformations for deep neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 1492–1500

  53. Yeung M, Sala E, Schönlieb C-B, Rundo L (2022) Unified focal loss: generalising dice and cross entropy-based losses to handle class imbalanced medical image segmentation. Comput Med Imaging Graph 95:102026

    Article  Google Scholar 

  54. Yue-Hei NJ, Hausknecht M, Vijayanarasimhan S, Vinyals O, Monga R, Toderici G (2015) Beyond short snippets: deep networks for video classification. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4694–4702

  55. Zhang M-L, Zhou Z-H (2014) A review on multi-label learning algorithms. Knowl Data Eng 26:1819–1837

    Article  Google Scholar 

  56. Zhang W, Xiao X, Xu X, Liang M, Wu H, Ruan J, Luo B (2018) Non-mass breast lesions on ultrasound: feature exploration and multimode ultrasonic diagnosis. Ultrasound Med Biol 44(8):1703–11

    Article  Google Scholar 

  57. Zhang Y-D, Pan C, Chen X, Wang F (2018) Abnormal breast identification by nine-layer convolutional neural network with parametric rectified linear unit and rank-based stochastic pooling. J Comput Sci 27:57–68

    Article  Google Scholar 

  58. Zhang Y-D, Satapathy SC, Guttery DS, Górriz JM, Wang S-H (2021) Improved breast cancer classification through combining graph convolutional network and convolutional neural network. Inf Process Manag 58(2):102439

    Article  Google Scholar 

  59. Zhang Y, Quan J, Yunxiao Z, Jian C, Zhu H, Liping G (2013) Diagnostic value of contrast-enhanced ultrasound parametric imaging in breast tumors. J Breast Cancer 16(2):208–13

    Article  Google Scholar 

  60. Zhao H, Rong X, Ouyang Q, Chen L, Dong B, Huihua Y (2010) Contrast-enhanced ultrasound is helpful in the differentiation of malignant and benign breast lesions. Eur J Radiol 73(2):288–293

    Article  Google Scholar 

  61. Zhao YX, Shuang L, Hu YB, Ge YY, Lv DM (2017) Diagnostic and prognostic values of contrast-enhanced ultrasound in breast cancer: a retrospective study. Oncotarget Therapy 10:1123–1129

    Article  Google Scholar 

  62. Zhou B, Andonian A, Oliva A, Torralba A (2018) Temporal relational reasoning in videos. In: Proceedings of the European Conference on Computer Vision (ECCV), pp 803–818

  63. Zhou T, Canu S, Ruan S (2021) Automatic COVID-19 CT segmentation using u-net integrated spatial and channel attention mechanism. Int J Imaging Syst Technol 31(1):16–27

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China (61876158), Fundamental Research Funds for the Central Universities (2682021ZTPY030), and Sichuan Science and Technology Program (2019YFS0432).

Author information

Authors and Affiliations

  1. School of Computing and Artificial Intelligence, Southwest Jiaotong University, Chengdu, Sichuan, 611756, China

    Xun Gong, Xu Zhao, Lin Fan & Tianrui Li

  2. Manufacturing Industry Chains Collaboration and Information Support Technology Key Laboratory of Sichuan Province, Chengdu, China

    Xun Gong

  3. North China University of Science and Technology Affiliated Hospital, Tangshan, Hebei, China

    Ying Guo

  4. Sichuan Academy of Medical Sciences, Sichuan Provincial People’s Hospital, Chengdu, Sichuan, China

    Jun Luo

Authors
  1. Xun Gong
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xu Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lin Fan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Tianrui Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ying Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Jun Luo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Xun Gong or Ying Guo.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gong, X., Zhao, X., Fan, L. et al. BUS-net: a bimodal ultrasound network for breast cancer diagnosis. Int. J. Mach. Learn. & Cyber. 13, 3311–3328 (2022). https://doi.org/10.1007/s13042-022-01596-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13042-022-01596-6

Keywords

Search

Navigation