Abstract
Current registration evaluations typically compute target registration error using manually annotated datasets. As a result, the quality of landmark annotations is crucial for unbiased comparisons because registration algorithms are trained and tested using these landmarks. Even though some data providers claim to have mitigated the inter-observer variability by having multiple raters, quality control such as a third-party screening can still be reassuring for intended users. Examining the landmark quality for neurosurgical datasets (RESECT and BITE) poses specific challenges. In this study, we applied the variogram, which is a tool extensively used in geostatistics, to convert 3D landmark distributions into an intuitive 2D representation. This allowed us to identify potential problematic cases efficiently so that they could be examined by experienced radiologists. In both the RESECT and BITE datasets, we identified and confirmed a small number of landmarks with potential localization errors and found that, in some cases, the landmark distribution was not ideal for an unbiased assessment of non-rigid registration errors. Under discussion, we provide some constructive suggestions for improving the utility of publicly available annotated data.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Bardosi, Z., Freysinger, W.: Estimating FLE distributions of manual fiducial localization in CT images. Int. J. Comput. Assist. Radiol. Surg. 11, 1043–1049 (2016)
Beatriz Garcia, C., Solter, J., Bossa, Matias Husch, A.: On the composition and limitations of publicly availabble covid-19 x-ray imaging datasets. arXiv (2020)
Borovec, J.: BIRL: benchmark on image registration methods with landmark validation. arXiv (2020)
Borovec, J., Munoz-Barrutia, A.: ANHIR: automatic non-rigid histological image registration challenge. IEEE Trans. Med. Imaging 39(10), 3042–3052 (2020)
Christensen, G.E., Geng, X., Kuhl, J.G., Bruss, J., Grabowski, T.J., Pirwani, I.A., Vannier, M.W., Allen, J.S., Damasio, H.: Introduction to the non-rigid image registration evaluation project (NIREP). In: Pluim, J.P.W., Likar, B., Gerritsen, F.A. (eds.) WBIR 2006. LNCS, vol. 4057, pp. 128–135. Springer, Heidelberg (2006). https://doi.org/10.1007/11784012_16
Cressie, N.: Statistics for Spatial Data. Wiley, New York (1991)
Fedorov, A., et al.: 3D slicer as an image computing platform for the quantitative imaging network. Mag. Resonan. Imaging 30(9), 1323–1341 (2012)
Fitzpatrick, J.: The retrospective image registration evaluation project. Insight-Journal (2007)
Fitzpatrick, J.: Fiducial registration error and target registration error are uncorrelated. In: Proceedings of SPIE Medical Imaging. p. 726102G. SPIE (2009)
Hellier, P., et al.: Retrospective evaluation of inter-subject brain registration. IEEE Trans. Med. Imaging 22(9), 1120–1130 (2003)
Hering, A., Murphy, K., van Ginneken, B.: Learn2reg challenge: CT lung registration. In: MICCAI (2021)
Kabus, S., Klinder, T., Murphy, K., van Ginneken, B., Lorenz, C., Pluim, J.P.W.: Evaluation of 4D-CT lung registration. In: Yang, G.-Z., Hawkes, D., Rueckert, D., Noble, A., Taylor, C. (eds.) MICCAI 2009. LNCS, vol. 5761, pp. 747–754. Springer, Heidelberg (2009). https://doi.org/10.1007/978-3-642-04268-3_92
Klein, A., et al.: Evaluation of 14 nonlinear deformation algorithms applied to human brain MRI registration. Neuroimage 46(3), 786–802 (2009)
Klein, A., et al.: Evaluation of volume-based and surface-based brain image registration methods. Neuroimage 51(1), 214–220 (2010)
Luo, J., et al.: A Feature-driven active framework for ultrasound-based brain shift compensation. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11073, pp. 30–38. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00937-3_4
Luo, J., Frisken, S., Wang, D., Golby, A., Sugiyama, M., Wells III, W.: Are registration uncertainty and error monotonically associated? In: MICCAI 2020 (2020)
Machado, I., et al.: Deformable MRI-ultrasound registration using correlation-based attribute matching for brain shift correction: accuracy and generality in multi-site data. Neuroimage 202(15), (2019)
Mercier, L., Del Maestro, R., Petrecca, K., Araujo, D., Haegelen, C., Collins, D.: Online database of clinical MR and ultrasound images of brain tumors. Med. Phys. 39(6), 3253–3261 (2012)
Murphy, K., et al.: Evaluation of registration methods on thoracic CT: the empire10 challenge. IEEE Trans. Med. Imaging 30(11), 1901–1920 (2011)
Oakden-Rayner, L.: Exploring large scale public medical image datasets. arXiv (2019)
Ou, Y., Akbari, H., Bilello, M., Da, X., Davatzikos, C.: Comparative evaluation of registration algorithms in different brain databases with varying difficulty. IEEE Trans. Med. Imaging 33(10), 2039–2065 (2014)
Reinertsen, I., Collins, D., Drouin, S.: The essential role of open data and software for the future of ultrasound-based neuronavigation. Front. Oncol. 10 (2020)
Rohlfing, T.: Image similarity and tissue overlaps as surrogates for image registration accuracy: widely used but unreliable. IEEE Trans. Med. Imaging 31(2), 153–16 (2012)
dos Santos, T., et al.: Pose-independent surface matching for intra-operative soft-tissue marker-less registration. Med. Image Anal. 18(7), 1101–1114 (2014)
Shamir, R.R., Joskowicz, L., Shoshan, Y.: Fiducial optimization for minimal target registration error in image-guided neurosurgery. IEEE Trans. Med. Imaging 31(3), 725–737 (2012)
Song, J.: Methods for evaluating image registration. Dissertation, University of Iowa (2017)
Sonka, M.: Handbook of medical imaging: medical image processing and analysis. In: SPIE (2000)
Sotiras, A., Davatzikos, C., Paragios, N.: Deformable medical image registration: A survey. IEEE Trans. Med. Imaging 32(7), 1153–1190 (2013)
Thompson, S., Penney, G., Dasgupta, P., Hawkes, D.: Improved modelling of tool tracking errors by modelling dependent marker errors. IEEE Trans. Med. Imaging 32(2), 165–177 (2013)
Wang, J., Zhang, M.: Deepflash: An efficient network for learning-based medical image registration. In: CVPR 2020, pp. 4444–4452. IEEE (2020)
West, J., Fitzpatrick, J., Wang, M., Woods, R.: Comparison and evaluation of retrospective intermodality brain image registration techniques. J. Comput. Assist. Tomogr. 21(4), 554–566 (1997)
Xiao, Y., Fortin, M., Unsgard, G., Rivaz, H., Reinertsen, I.: Retrospective evaluation of cerebral tumors (resect): a clinical database of pre-operative MRI and intra-operative ultrasound in low-grade glioma surgeries. Med. Phys. 44(7), 3875–3882 (2017)
Xiao, Y., Rivaz, H., Chabanas, M.: Evaluation of MRI to ultrasound registration methods for brain shift correction: the curious2018 challenge. IEEE Trans. Med. Imaging 39(3), 777–786 (2020)
Xu, Z., et al.: Evaluation of six registration methods for the human abdomen on clinically acquired CT. IEEE Trans. Biomed. Eng. 63(8), 1563–1572 (2016)
Yassa, M., Stark, C.E.: A quantitative evaluation of cross-participant registration techniques for MRI studies of the medial temporal lobe. Neuroimage 442, 319–327 (2009)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Luo, J. et al. (2022). On the Dataset Quality Control for Image Registration Evaluation. In: Wang, L., Dou, Q., Fletcher, P.T., Speidel, S., Li, S. (eds) Medical Image Computing and Computer Assisted Intervention – MICCAI 2022. MICCAI 2022. Lecture Notes in Computer Science, vol 13436. Springer, Cham. https://doi.org/10.1007/978-3-031-16446-0_4
Download citation
DOI: https://doi.org/10.1007/978-3-031-16446-0_4
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-16445-3
Online ISBN: 978-3-031-16446-0
eBook Packages: Computer ScienceComputer Science (R0)
