Skip to main content

Multimodality Biomedical Image Registration Using Free Point Transformer Networks

  • Conference paper
  • First Online:
Medical Ultrasound, and Preterm, Perinatal and Paediatric Image Analysis (ASMUS 2020, PIPPI 2020)

Abstract

We describe a point-set registration algorithm based on a novel free point transformer (FPT) network, designed for points extracted from multimodal biomedical images for registration tasks, such as those frequently encountered in ultrasound-guided interventional procedures. FPT is constructed with a global feature extractor which accepts unordered source and target point-sets of variable size. The extracted features are conditioned by a shared multilayer perceptron point transformer module to predict a displacement vector for each source point, transforming it into the target space. The point transformer module assumes no vicinity or smoothness in predicting spatial transformation and, together with the global feature extractor, is trained in a data-driven fashion with an unsupervised loss function. In a multimodal registration task using prostate MR and sparsely acquired ultrasound images, FPT yields comparable or improved results over other rigid and non-rigid registration methods. This demonstrates the versatility of FPT to learn registration directly from real, clinical training data and to generalize to a challenging task, such as the interventional application presented.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Maintz, J.B., Viergever, M.A.: A survey of medical image registration. Med. Image Anal. 2(5), 1–36 (1998)

    Article  Google Scholar 

  2. Costa, D.N., Pedrosa, I., Donato, F., Roehrborn, C.G., Rofsky, N.M.: MR imaging-transrectal US fusion for targeted prostate biopsies: implication for diagnosis and clinical management. RadioGraphics 35(3), 696–708 (2015)

    Article  Google Scholar 

  3. Puech, P., et al.: Prostate cancer diagnosis: multiparametric MR-targeted biopsy with cognitive and transrectal US-MR fusion guidance versus systematic biopsy – prospective multicenter study. Radiology 268(2), 461–469 (2013)

    Article  Google Scholar 

  4. Lavaerts, M., De Wever, L., Vanhoutte, E., De Keyzer, F., Oyen, R.: TRUS-MR fusion biopsy of the prostate: radiological and histological correlation. J. Belg. Soc. Radiol. 100(1), 1–9 (2016)

    Article  Google Scholar 

  5. Valerio, M., et al.: Detection of clinically significant prostate cancer using magnetic resonance imaging-ultrasound fusion targeted biopsy: a systematic review. Eur. Radiol. 68(1), 8–19 (2015)

    Google Scholar 

  6. Karnik, V.V., et al.: Assessment of image registration accuracy in three-dimensional transrectal ultrasound guided prostate biopsy. Med. Phys. 37(2), 802–813 (2010)

    Article  Google Scholar 

  7. Hu, Y., et al.: MR to ultrasound registration for image-guided prostate interventions. Med. Image Anal. 16(3), 687–703 (2012)

    Article  Google Scholar 

  8. De Silva, T., et al.: 2D-3D rigid registration to compensate for prostate motion during 3D TRUS-guided biopsy. Med. Phys. 40(2), 022904-1–022904-13 (2013)

    Google Scholar 

  9. Hu, Y., et al.: Weakly-supervised convolutional neural networks for multimodal image registration. Med. Image Anal. 49, 1–13 (2018)

    Article  Google Scholar 

  10. Zhang, S., Jiang, S., Yang, Z., Liu, R.: 2D ultrasound and 3D MR image reconstruction of the prostate for brachytherapy surgical navigation. Med. (Balt.) 94(40), e1643 (2015)

    Article  Google Scholar 

  11. Gilles, D.J., Gardi, L., De Silva, T., Zhao, S.R., Fenster, A.: Real-time registration of 3D to 2D ultrasound images for image-guided prostate biopsy. Med. Phys. 44(9), 4708–4723 (2017)

    Article  Google Scholar 

  12. van Sloun, R.J.G., et al.: Deep learning for real-time, automatic, and scanner adapted prostate (zone) segmentation of transrectal ultrasound, for example, magnetic resonance imaging-transrectal ultrasound fusion prostate biopsy. Eur. Urol. Focus. (2019). https://doi.org/10.1016/j.euf.2019.04.009

  13. Ghavami, N., et al.: Integration of spatial information in convolutional neural networks for automatic segmentation of intraoperative transrectal ultrasound images. J. Med. Imaging 6(1), 011003-1, 011003-6 (2018)

    Google Scholar 

  14. Besl, P.J., McKay, N.D.: A method for registration of 3-D shapes. IEEE Trans. Pattern Anal. Mach. Intell. 14(2), 239–256 (1992)

    Article  Google Scholar 

  15. Myronenko, A., Song, X.: Point set registration: coherent point drift. IEEE Trans. Pattern Anal. Mach. Intell. 32(12), 2262–2275 (2010)

    Article  Google Scholar 

  16. Jian, B., Vemuri, B.C.: Robust point set registration using gaussian mixture models. IEEE Trans. Pattern Anal. Mach. Intell. 33(8), 1633–1645 (2010)

    Article  Google Scholar 

  17. Chui, H., Rangarajan, A.: A new point matching algorithm for non-rigid registration. Comput. Vis. Image Underst. 89(2–3), 114–141 (2003)

    Article  Google Scholar 

  18. Aoki, Y., Goforth, H., Srivatsan, R.A., Lucey, S.: PointNetLK: robust and efficient point cloud registration using PointNet. In: The IEEE Conference on Computer Vision and Pattern Recognition, pp. 7163–7172. IEEE (2019)

    Google Scholar 

  19. Qi, C.R., Su, H., Mo, K., Guibas, L.J.: PointNet: deep learning on point sets for 3D classification and segmentation. In: The IEEE Conference on Computer Vision and Pattern Recognition, pp. 652–660. IEEE (2017)

    Google Scholar 

  20. Fan, H., Su, H., Guibas, L.J.: A point set generation network for 3D object reconstruction from a single image. In: The IEEE Conference on Computer Vision and Pattern Recognition, pp. 605–613. IEEE (2017)

    Google Scholar 

  21. Wu, Z., et al.: 3D ShapeNets: a deep representation for volumetric shapes. In: The IEEE Conference on Computer Vision and Pattern Recognition, pp. 1912–1920. IEEE (2015)

    Google Scholar 

  22. Pan, S.J., Yang, Q.: A survey on transfer learning. IEEE Trans. Knowl. Data Eng. 22(10), 1345–1359 (2010)

    Article  Google Scholar 

  23. Donaldson, I., et al.: MP33-20 the smart target biopsy trial: a prospective paired blinded trial with randomization to compare visual-estimation and image-fusion targeted prosate biopsies. J. Urol. 197(4), e425 (2017)

    Google Scholar 

  24. Wein, W., Brunke, S., Khamene, A., Callstrom, M.R., Navab, N.: Automatic ct-ultrasound registration for diagnostic imaging and image-guided intervention. Med. Image Anal. 12(5), 577–585 (2008)

    Article  Google Scholar 

  25. Gueziri, H.-E., Drouin, S., Yan, C.X.B., Collins, D.L.: Toward real-time rigid registration of intra-operative ultrasound with preoperative ct images for lumbar spinal fusion surgery. Int. J. Comput. Assist. Radiol. Surg. 14(11), 1933–1943 (2019)

    Article  Google Scholar 

  26. Wein, W., Roper, B., Navab, N.: Automatic registration and fusion of ultrasound with ct for radiotherapy. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 303–311 (2005)

    Google Scholar 

Download references

Acknowledgments

Z. Baum is supported by the Natural Sciences and Engineering Research Council of Canada Postgraduate Scholarships-Doctoral Program, the University College London Overseas and Graduate Research Scholarships. This work is also supported by the Wellcome/EPSRC Centre for Interventional and Surgical Sciences (203145Z/16/Z).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Zachary M. C. Baum .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Baum, Z.M.C., Hu, Y., Barratt, D.C. (2020). Multimodality Biomedical Image Registration Using Free Point Transformer Networks. In: Hu, Y., et al. Medical Ultrasound, and Preterm, Perinatal and Paediatric Image Analysis. ASMUS PIPPI 2020 2020. Lecture Notes in Computer Science(), vol 12437. Springer, Cham. https://doi.org/10.1007/978-3-030-60334-2_12

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-60334-2_12

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-60333-5

  • Online ISBN: 978-3-030-60334-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics