Skip to main content
SpringerLink
Log in

Robust Needle Localization and Enhancement Algorithm for Ultrasound by Deep Learning and Beam Steering Methods

  • Regular Paper
  • Published:
Journal of Computer Science and Technology Aims and scope Submit manuscript

Abstract

Ultrasound (US) imaging is clinically used to guide needle insertions because it is safe, real-time, and low cost. The localization of the needle in the ultrasound image, however, remains a challenging problem due to specular reflection off the smooth surface of the needle, speckle noise, and similar line-like anatomical features. This study presents a novel robust needle localization and enhancement algorithm based on deep learning and beam steering methods with three key innovations. First, we employ beam steering to maximize the reflection intensity of the needle, which can help us to detect and locate the needle precisely. Second, we modify the U-Net which is an end-to-end network commonly used in biomedical segmentation by using two branches instead of one in the last up-sampling layer and adding three layers after the last down-sample layer. Thus, the modified U-Net can real-time segment the needle shaft region, detect the needle tip landmark location and determine whether an image frame contains the needle by one shot. Third, we develop a needle fusion framework that employs the outputs of the multi-task deep learning (MTL) framework to precisely locate the needle tip and enhance needle shaft visualization. Thus, the proposed algorithm can not only greatly reduce the processing time, but also significantly increase the needle localization accuracy and enhance the needle visualization for real-time clinical intervention applications.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Gulsen F, Kantarci F. Ultrasound-guided intervention around the hip joint. American Journal of Roentgenology, 2012, 198(1): W95. https://doi.org/10.2214/AJR.11.7627.

    Article  Google Scholar 

  2. Chin K J, Perlas A, Chan V W S, Brull R. Needle visualization in ultrasound-guided regional anesthesia: Challenges and solutions. Regional Anesthesia and Pain Medicine, 2008, 33(6): 532-544. https://doi.org/10.1016/j.rapm.2008.06.002.

    Article  Google Scholar 

  3. Fevre M C, Vincent C, Picard J, Vighetti A, Chapuis C, Detavernier M, Allenet B, Payen J F, Bosson J L, Albaladejo P. Reduced variability and execution time to reach a target with a needle GPS system: Comparison between physicians, residents and nurse anaesthetists. Anaesthesia Critical Care & Pain Medicine, 2018, 37(1): 55-60. https://doi.org/10.1016/j.accpm.2016.05.008.

  4. Stolka P J, Foroughi P, Rendina M, Weiss C R, Hager G D, Boctor E M. Needle guidance using handheld stereo vision and projection for ultrasound-based interventions. In Proc. the 17th International Conference on Medical Image Computing and Computer-Assisted Intervention, September 2014, pp.684-691. https://doi.org/10.1007/978-3-319-10470-6_85.

  5. Lu H, Li J, Lu Q, Bharat S, Erkamp R, Chen B, Drysdale J, Vignon F, Jain A. A new sensor technology for 2D ultrasound-guided needle tracking. In Proc. the 17th International Conference on Medical Image Computing and Computer-Assisted Intervention, September 2014, pp.389-396. https://doi.org/10.1007/978-3-319-10470-6_49.

  6. Xia W, West S J, Finlay M C, Mari J M, Ourselin S, David A L, Desjardins A E. Looking beyond the imaging plane: 3D needle tracking with a linear array ultrasound probe. Sci. Rep., 2017, 7(1): Article No. 3674. https://doi.org/10.1038/s41598-017-03886-4.

  7. Ding M, Fenster A. Projection-based needle segmentation in 3D ultrasound images. In Proc. the 6th International Conference on Medical Image Computing and Computer-Assisted Intervention, November 2003, pp.319-327. https://doi.org/10.3109/10929080500079321.

  8. Qiu W, Yuchi M, Ding M. Phase grouping-based needle segmentation in 3-D trans-rectal ultrasound-guided prostate trans-perineal therapy. Ultrasound in Medicine & Biology, 2014, 40(4): 804-816. https://doi.org/10.1016/j.ultrasmedbio.2013.11.004.

  9. Beigi P, Rohling R, Salcudean T, Lessoway V A, Ng G C. Needle trajectory and tip localization in real-time 3-D ultrasound using a moving stylus. Ultrasound in Medicine & Biology, 2015, 41(7): 2057-2070. https://doi.org/10.1016/j.ultrasmedbio.2015.03.013.

  10. Zhao Y, Shen Y, Bernard A, Cachard C, Liebgott H. Evaluation and comparison of current biopsy needle localization and tracking methods using 3D ultrasound. Ultrasonics, 2017, 73: 206-220. https://doi.org/10.1016/j.ultras.2016.09.006.

    Article  Google Scholar 

  11. Ayvali E, Desai J P. Optical ow-based tracking of needles and needle-tip localization using circular hough transform in ultrasound images. Annals of Biomedical Engineering, 2015, 43(8): 1828-1840. https://doi.org/10.1007/s10439-014-1208-0.

    Article  Google Scholar 

  12. Uhercik M, Kybic J, Liebgott H, Cachard C. Model fitting using RANSAC for surgical tool localization in 3D ultrasound images. IEEE Transactions on Biomedical Engineering, 2010, 57(8): 1907-1916. https://doi.org/10.1109/TBME.2010.2046416.

    Article  Google Scholar 

  13. Zhao Y, Cachard C, Liebgott H. Automatic needle detection and tracking in 3D ultrasound using an ROI-based RANSAC and Kalman method. Ultrasonic Imaging, 2013, 35(4): 283-306. https://doi.org/10.1177/0161734613502004.

    Article  Google Scholar 

  14. Mwikirize C, Nosher J L, Hacihaliloglu I. Learning needle tip localization from digital subtraction in 2D ultrasound. International Journal of Computer Assisted Radiology and Surgery, 2019, 14(6): 1017-1026. https://doi.org/10.1007/s11548-019-01951-z.

    Article  Google Scholar 

  15. Mwikirize C, Nosher J L, Hacihaliloglu I. Convolution neural networks for real-time needle detection and localization in 2D ultrasound. International Journal of Computer Assisted Radiology and Surgery, 2018, 13(5): 647-657. https://doi.org/10.1007/s11548-018-1721-y.

    Article  Google Scholar 

  16. Beigi P, Rohling R, Salcudean S E, Ng G C. CASPER: Computer-aided segmentation of imperceptible motion—A learning-based tracking of an invisible needle in ultrasound. International Journal of Computer Assisted Radiology and Surgery, 2017, 12(11): 1857-1866. https://doi.org/10.1007/s11548-017-1631-4.

    Article  Google Scholar 

  17. Pourtaherian A, Ghazvinian Zanjani F, Zinger S, Mihajlovic N, Ng G C, Korsten H H M, De With P H N. Robust and semantic needle detection in 3D ultrasound using orthogonal-plane convolutional neural networks. International Journal of Computer Assisted Radiology and Surgery, 2018, 13(9): 1321-1333. https://doi.org/10.1007/s11548-018-1798-3.

    Article  Google Scholar 

  18. Arif M, Moelker A, Van Walsum T. Automatic needle detection and real-time Bi-planar needle visualization during 3D ultrasound scanning of the liver. Medical Image Analysis, 2019, 53: 104-110. https://doi.org/10.1016/j.media.2019.02.002.

    Article  Google Scholar 

  19. Mwikirize C, Nosher J L, Hacihaliloglu I. Single shot needle tip localization in 2D ultrasound. In Proc. the 22nd International Conference on Medical Image Computing and Computer Assisted Intervention, October 2019, pp.637-645. https://doi.org/10.1007/978-3-030-32254-0_71.

  20. Hatt C R, Ng G, Parthasarathy V. Enhanced needle localization in ultrasound using beam steering and learning-based segmentation. Computerized Medical Imaging and Graphics, 2015, 41: 46-54. https://doi.org/10.1016/j.compmedimag.2014.06.016.

    Article  Google Scholar 

  21. Ronneberger O, Fischer P, Brox T. U-Net: Convolutional networks for biomedical image segmentation. In Proc. the 18th International Conference on Medical Image Computing and Computer-Assisted Intervention, October 2015, pp. 234-241. https://doi.org/10.1007/978-3-319-24574-4_28.

  22. Gao J, Liu P, Liu G D, Zhang L. Supplementary material. Technical Report, Sichuan University, 2020. https://github.com/gaojun0821/NLEM/blob/main/doc/-Supplementary_Material.pdf, Dec. 2020.

  23. Jiang B, Struthers A, Sun Z, Feng Z, Zhao X, Zhao K, Dai W, Zhou X, Berens M E, Zhang L. Employing graphics processing unit technology, alternating direction implicit method and domain decomposition to speed up the numerical diffusion solver for the biomedical engineering research. International Journal for Numerical Methods in Biomedical Engineering, 2011, 27(11): 1829-1849. https://doi.org/10.1002/cnm.1444.

    Article  MathSciNet  MATH  Google Scholar 

  24. Jiang B N, Dai W Z, Khaliq A, Carey M, Zhou X B, Zhang L. Novel 3D GPU based numerical parallel diffusion algorithms in cylindrical coordinates for health care simulation. Mathematics and Computers in Simulation, 2015, 109: 1-19. https://doi.org/10.1016/j.matcom.2014.07.003.

    Article  MathSciNet  MATH  Google Scholar 

  25. Zhang L, Jiang B, Wu Y, Strouthos C, Sun P Z, Su J, Zhou X. Developing a multiscale, multi-resolution agent-based brain tumor model by graphics processing units. Theoretical Biology and Medical Modelling, 2011, 8(1): Article No. 46. https://doi.org/10.1186/1742-4682-8-46.

  26. Isola P, Zhu J Y, Zhou T, Efros A. Image-to-image translation with conditional adversarial networks. In Proc. the 2017 IEEE Conference on Computer Vision and Pattern Recognition, July 2017, pp.5967-5976. https://doi.org/10.1109/CVPR.2017.632.

  27. Wu W, Song L, Yang Y, Wang J, Liu H, Zhang L. Exploring the dynamics and interplay of human papillomavirus and cervical tumorigenesis by integrating biological data into a mathematical model. BMC Bioinformatics, 2020, 21(Suppl 7): Article No. 152. https://doi.org/10.1186/s12859-020-3454-5.

  28. Ren S, He K, Girshick R, Sun J. Faster R-CNN: Towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(6): 1137-1149. https://doi.org/10.1109/TPAMI.2016.2577031.

  29. Xu Z, Huo Y, Park J, Landman B, Milkowski A, Grbic S, Zhou S. Less is more: Simultaneous view classification and landmark detection for abdominal ultrasound images. In Proc. the 21st International Conference on Medical Image Computing and Computer-Assisted Intervention, September 2018, pp.711-719. https://doi.org/10.1007/978-3-030-00934-2_79.

  30. Powers D. Evaluation: From precision, recall and F-factor to ROC, informedness, markedness & correlation. arXiv:2010.16061, 2008. https://arxiv.org/abs/2010.16061, Jan. 2021.

  31. Xia Y, Yang C, Hu N, Yang Z, He X, Li T, Zhang L. Exploring the key genes and signaling transduction pathways related to the survival time of glioblastoma multiforme patients by a novel survival analysis model. BMC Genomics, 2017, 18(Suppl 1): Article No. 950. https://doi.org/10.1186/s12864-016-3256-3.

  32. Zhang L, Bai W, Yuan N, Du Z. Comprehensively bench-marking applications for detecting copy number variation. PLoS Computational Biology, 2019, 15(5): Article No. e1007069. https://doi.org/10.1371/journal.pcbi.1007069.

  33. Zhang L, Li J, Yin K, Jiang Z, Li T, Hu R, Yu Z, Feng H, Chen Y. Computed tomography angiography-based analysis of high-risk intracerebral haemorrhage patients by employing a mathematical model. BMC Bioinformatics, 2019, 20(Suppl 7): Article No. 193. https://doi.org/10.1186/s12859-019-2741-5.

  34. Zhang L, Liu Y, Wang M, Wu Z, Li N, Zhang J, Yang C. EZH2-, CHD4-, and IDH-linked epigenetic perturbation and its association with survival in glioma patients. J. Mol. Cell Biol., 2017, 9(6): 477-488. https://doi.org/10.1093/jmcb/mjx056.

    Article  Google Scholar 

  35. Kremkau F W, Taylor K J. Artifacts in ultrasound imaging. Journal of Ultrasound in Medicine, 1986, 5(4): 227-237. https://doi.org/10.7863/jum.1986.5.4.227.

    Article  Google Scholar 

  36. Paul Y, Barthez D, Léveillé R, Peter V, Scrivani D. Side lobes and grating lobes artifacts in ultrasound imaging. Veterinary Radiology & Ultrasound, 1997, 38(5): 387-393. https://doi.org/10.1111/j.1740-8261.1997.tb02104.x.

  37. Matalon T A, Silver B. US guidance of interventional procedures. Radiology, 1990, 174(1): 43-47. https://doi.org/10.1148/radiology.174.1.2403684.

    Article  Google Scholar 

  38. Sofka C, Collins A J, Adler R. Use of ultrasonographic guidance in interventional musculoskeletal procedures: A review from a single institution. Journal of Ultrasound in Medicine: Official Journal of the American Institute of Ultrasound in Medicine, 2001, 20(1): 21-26. https://doi.org/10.7863/jum.2001.20.1.21.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. College of Computer Science, Sichuan University, Chengdu, 610065, China

    Jun Gao & Le Zhang

  2. Stork Healthcare, Chengdu, 610000, China

    Jun Gao & Paul Liu

  3. College of Computer and Information Science, Southwest University, Chongqing, 400715, China

    Guang-Di Liu

  4. West China Biomedical Big Data Center, West China Hospital, Sichuan University, Chengdu, 610065, China

    Le Zhang

  5. Pera Corporation Ltd., Beijing, 100025, China

    Le Zhang

Authors
  1. Jun Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Paul Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Guang-Di Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Le Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Le Zhang.

Supplementary Information

ESM 1

(PDF 155 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gao, J., Liu, P., Liu, GD. et al. Robust Needle Localization and Enhancement Algorithm for Ultrasound by Deep Learning and Beam Steering Methods. J. Comput. Sci. Technol. 36, 334–346 (2021). https://doi.org/10.1007/s11390-021-0861-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11390-021-0861-7

Keywords

Search

Navigation