Abstract
We describe a point-set registration algorithm based on a novel free point transformer (FPT) network, designed for points extracted from multimodal biomedical images for registration tasks, such as those frequently encountered in ultrasound-guided interventional procedures. FPT is constructed with a global feature extractor which accepts unordered source and target point-sets of variable size. The extracted features are conditioned by a shared multilayer perceptron point transformer module to predict a displacement vector for each source point, transforming it into the target space. The point transformer module assumes no vicinity or smoothness in predicting spatial transformation and, together with the global feature extractor, is trained in a data-driven fashion with an unsupervised loss function. In a multimodal registration task using prostate MR and sparsely acquired ultrasound images, FPT yields comparable or improved results over other rigid and non-rigid registration methods. This demonstrates the versatility of FPT to learn registration directly from real, clinical training data and to generalize to a challenging task, such as the interventional application presented.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Maintz, J.B., Viergever, M.A.: A survey of medical image registration. Med. Image Anal. 2(5), 1–36 (1998)
Costa, D.N., Pedrosa, I., Donato, F., Roehrborn, C.G., Rofsky, N.M.: MR imaging-transrectal US fusion for targeted prostate biopsies: implication for diagnosis and clinical management. RadioGraphics 35(3), 696–708 (2015)
Puech, P., et al.: Prostate cancer diagnosis: multiparametric MR-targeted biopsy with cognitive and transrectal US-MR fusion guidance versus systematic biopsy – prospective multicenter study. Radiology 268(2), 461–469 (2013)
Lavaerts, M., De Wever, L., Vanhoutte, E., De Keyzer, F., Oyen, R.: TRUS-MR fusion biopsy of the prostate: radiological and histological correlation. J. Belg. Soc. Radiol. 100(1), 1–9 (2016)
Valerio, M., et al.: Detection of clinically significant prostate cancer using magnetic resonance imaging-ultrasound fusion targeted biopsy: a systematic review. Eur. Radiol. 68(1), 8–19 (2015)
Karnik, V.V., et al.: Assessment of image registration accuracy in three-dimensional transrectal ultrasound guided prostate biopsy. Med. Phys. 37(2), 802–813 (2010)
Hu, Y., et al.: MR to ultrasound registration for image-guided prostate interventions. Med. Image Anal. 16(3), 687–703 (2012)
De Silva, T., et al.: 2D-3D rigid registration to compensate for prostate motion during 3D TRUS-guided biopsy. Med. Phys. 40(2), 022904-1–022904-13 (2013)
Hu, Y., et al.: Weakly-supervised convolutional neural networks for multimodal image registration. Med. Image Anal. 49, 1–13 (2018)
Zhang, S., Jiang, S., Yang, Z., Liu, R.: 2D ultrasound and 3D MR image reconstruction of the prostate for brachytherapy surgical navigation. Med. (Balt.) 94(40), e1643 (2015)
Gilles, D.J., Gardi, L., De Silva, T., Zhao, S.R., Fenster, A.: Real-time registration of 3D to 2D ultrasound images for image-guided prostate biopsy. Med. Phys. 44(9), 4708–4723 (2017)
van Sloun, R.J.G., et al.: Deep learning for real-time, automatic, and scanner adapted prostate (zone) segmentation of transrectal ultrasound, for example, magnetic resonance imaging-transrectal ultrasound fusion prostate biopsy. Eur. Urol. Focus. (2019). https://doi.org/10.1016/j.euf.2019.04.009
Ghavami, N., et al.: Integration of spatial information in convolutional neural networks for automatic segmentation of intraoperative transrectal ultrasound images. J. Med. Imaging 6(1), 011003-1, 011003-6 (2018)
Besl, P.J., McKay, N.D.: A method for registration of 3-D shapes. IEEE Trans. Pattern Anal. Mach. Intell. 14(2), 239–256 (1992)
Myronenko, A., Song, X.: Point set registration: coherent point drift. IEEE Trans. Pattern Anal. Mach. Intell. 32(12), 2262–2275 (2010)
Jian, B., Vemuri, B.C.: Robust point set registration using gaussian mixture models. IEEE Trans. Pattern Anal. Mach. Intell. 33(8), 1633–1645 (2010)
Chui, H., Rangarajan, A.: A new point matching algorithm for non-rigid registration. Comput. Vis. Image Underst. 89(2–3), 114–141 (2003)
Aoki, Y., Goforth, H., Srivatsan, R.A., Lucey, S.: PointNetLK: robust and efficient point cloud registration using PointNet. In: The IEEE Conference on Computer Vision and Pattern Recognition, pp. 7163–7172. IEEE (2019)
Qi, C.R., Su, H., Mo, K., Guibas, L.J.: PointNet: deep learning on point sets for 3D classification and segmentation. In: The IEEE Conference on Computer Vision and Pattern Recognition, pp. 652–660. IEEE (2017)
Fan, H., Su, H., Guibas, L.J.: A point set generation network for 3D object reconstruction from a single image. In: The IEEE Conference on Computer Vision and Pattern Recognition, pp. 605–613. IEEE (2017)
Wu, Z., et al.: 3D ShapeNets: a deep representation for volumetric shapes. In: The IEEE Conference on Computer Vision and Pattern Recognition, pp. 1912–1920. IEEE (2015)
Pan, S.J., Yang, Q.: A survey on transfer learning. IEEE Trans. Knowl. Data Eng. 22(10), 1345–1359 (2010)
Donaldson, I., et al.: MP33-20 the smart target biopsy trial: a prospective paired blinded trial with randomization to compare visual-estimation and image-fusion targeted prosate biopsies. J. Urol. 197(4), e425 (2017)
Wein, W., Brunke, S., Khamene, A., Callstrom, M.R., Navab, N.: Automatic ct-ultrasound registration for diagnostic imaging and image-guided intervention. Med. Image Anal. 12(5), 577–585 (2008)
Gueziri, H.-E., Drouin, S., Yan, C.X.B., Collins, D.L.: Toward real-time rigid registration of intra-operative ultrasound with preoperative ct images for lumbar spinal fusion surgery. Int. J. Comput. Assist. Radiol. Surg. 14(11), 1933–1943 (2019)
Wein, W., Roper, B., Navab, N.: Automatic registration and fusion of ultrasound with ct for radiotherapy. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 303–311 (2005)
Acknowledgments
Z. Baum is supported by the Natural Sciences and Engineering Research Council of Canada Postgraduate Scholarships-Doctoral Program, the University College London Overseas and Graduate Research Scholarships. This work is also supported by the Wellcome/EPSRC Centre for Interventional and Surgical Sciences (203145Z/16/Z).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Baum, Z.M.C., Hu, Y., Barratt, D.C. (2020). Multimodality Biomedical Image Registration Using Free Point Transformer Networks. In: Hu, Y., et al. Medical Ultrasound, and Preterm, Perinatal and Paediatric Image Analysis. ASMUS PIPPI 2020 2020. Lecture Notes in Computer Science(), vol 12437. Springer, Cham. https://doi.org/10.1007/978-3-030-60334-2_12
Download citation
DOI: https://doi.org/10.1007/978-3-030-60334-2_12
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-60333-5
Online ISBN: 978-3-030-60334-2
eBook Packages: Computer ScienceComputer Science (R0)