Abstract
Quantitative performance metrics play a pivotal role in medical imaging by offering critical insights into method performance and facilitating objective method comparison. Recently, platforms providing recommendations for metrics selection as well as resources for evaluating methods through computational challenges and online benchmarking have emerged, with an inherent assumption that metrics implementations are consistent across studies and equivalent throughout the community. In this study, we question this assumption by reviewing five different open-source implementations for computing the Hausdorff distance (HD), a boundary-based metric commonly used for assessing the performance of semantic segmentation. Despite sharing a single generally accepted mathematical definition, our experiments reveal notable systematic differences in the HD and its 95th percentile variant across implementations when applied to clinical segmentations with varying voxel sizes, which fundamentally impacts and constrains the ability to objectively compare results across different studies. Our findings should encourage the medical imaging community towards standardizing the implementation of the HD computation, so as to foster objective, reproducible and consistent comparisons when reporting performance results.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
- 2.
- 3.
- 4.
- 5.
Meshing is based on the marching cubes algorithm [14] that takes a 3D mask A and generates a 3D mesh \(\mathcal {M}_{\!A}\) consisting of vertices and surfels (i.e. triangular faces).
- 6.
Voxelization (3D analogue of rasterization) is based on vtkPolyDataToImageStencil from the Visualization Toolkit (VTK) [22] that allows completely bijective transforms between the mesh and image space, i.e. voxelization of \(\mathcal {M}_{\!A}\), obtained from A by meshing without smoothing, results again in A.
- 7.
Based on vtkImplicitPolyDataDistance in the VTK [22].
- 8.
https://github.com/google-deepmind/surface-distance, v0.1, sha1-hash: ee651c8.
- 9.
https://github.com/Project-MONAI/MetricsReloaded, v0.1.0, sha1-hash: b3a3715.
- 10.
https://github.com/Project-MONAI/MONAI, v1.3.0, sha1-hash: 865972f.
- 11.
https://gitlab.com/plastimatch/plastimatch, v1.9.4, sha1-hash: 581c7692.
- 12.
https://github.com/Jingnan-Jia/segmentation_metrics, v1.6.1, sha1-hash: fed1852.
- 13.
This is indeed the case for Plastimatch that does not implement \(H\!D_{100}\) as a 100-percentile but uses a separate calculation pipeline based on max of all distances.
References
Alt, H., Guibas, L.J.: Discrete geometric shapes: matching, interpolation, and approximation. In: Handbook of Computational Geometry, chap. 3, pp. 121–153. Elsevier (2000). https://doi.org/10.1016/B978-0-444-82537-7.X5000-1
Aydin, O.U., Taha, A.A., Hilbert, A., et al.: On the usage of average Hausdorff distance for segmentation performance assessment: hidden error when used for ranking. Eur. Radiol. Exp. 5, 4 (2021). https://doi.org/10.1186/s41747-020-00200-2
Cardoso, M.J., Li, W., Brown, R., et al.: MONAI: an open-source framework for deep learning in healthcare. arXiv:2211.02701 (2022). https://doi.org/10.48550/arXiv.2211.02701
Duprez, M., Bordas, S.P.A., Bucki, M., et al.: Quantifying discretization errors for soft tissue simulation in computer assisted surgery: a preliminary study. Appl. Math. Model. 77, 709–723 (2020). https://doi.org/10.1016/j.apm.2019.07.055
Fedorov, A., Beichel, R., Kalpathy-Cramer, J., et al.: 3D Slicer as an image computing platform for the Quantitative Imaging Network. Magn. Reson. Imaging 30, 1323–1341 (2012). https://doi.org/10.1016/j.mri.2012.05.001
Felzenszwalb, P.F., Huttenlocher, D.P.: Distance transforms of sampled functions. Theory Comput. 8, 415–428 (2012). https://doi.org/10.4086/toc.2012.v008a019
Hausdorff, F.: Grundzüge der Mengenlehre [Basics of Set Theory]. Leipzig Viet, Leipzig, Germany (1914). https://archive.org/details/grundzgedermen00hausuoft/
Huttenlocher, D.P., Klanderman, G.A., Rucklidge, W.J.: Comparing images using the Hausdorff distance. IEEE Trans. Pattern Anal. Mach. Intell. 15, 850–863 (1993). https://doi.org/10.1109/34.232073
Jia, J., Staring, M., Stoel, B.C.: Seg-metrics: a Python package to compute segmentation metrics. medRxiv (2024). https://doi.org/10.1101/2024.02.22.24303215
Jungeblut, P., Kleist, L., Miltzow, T.: The complexity of the Hausdorff distance. Discret. Comput. Geom. 71, 177–213 (2024). https://doi.org/10.1007/s00454-023-00562-5
Karimi, D., Salcudean, S.E.: Reducing the Hausdorff distance in medical image segmentation with convolutional neural networks. IEEE Trans. Med. Imaging 39, 499–513 (2020). https://doi.org/10.1109/TMI.2019.2930068
Klette, R., Rosenfeld, A.: Metrics. In: Digital Geometry: Geometric Methods for Digital Picture Analysis, chap. 3, pp. 77–116. Elsevier (2004). https://doi.org/10.1016/B978-1-55860-861-0.X5000-7
Li, W., Liang, Z., Ma, P., Wang, R., Cui, X., Chen, P.: Hausdorff GAN: improving GAN generation quality with Hausdorff metric. IEEE Trans. Cybern. 52, 10407–10419 (2022). https://doi.org/10.1109/TCYB.2021.3062396
Lorensen, W.E., Cline, H.E.: Marching cubes: a high resolution 3D surface construction algorithm. In: 14th Conference on Computer Graphics and Interactive Techniques - SIGGRAPH 1987, pp. 163–169. ACM (1987). https://doi.org/10.1145/37401.37422
Mackay, K., Bernstein, D., Glocker, B., Kamnitsas, K., Taylor, A.: A review of the metrics used to assess auto-contouring systems in radiotherapy. Clin. Oncol. 35, 354–369 (2023). https://doi.org/10.1016/j.clon.2023.01.016
Maier-Hein, L., et al.: Metrics reloaded: recommendations for image analysis validation. Nat. Methods 21, 195–212 (2024). https://doi.org/10.1038/s41592-023-02151-z
Nikolov, S., Blackwell, S., Zverovitch, A., et al.: Clinically applicable segmentation of head and neck anatomy for radiotherapy: deep learning algorithm development and validation study. J. Med. Internet Res. 23, e26151 (2021). https://doi.org/10.2196/26151
Podobnik, G., Ibragimov, B., Strojan, P., Peterlin, P., Vrtovec, T.: vOARiability: interobserver and intermodality variability analysis in OAR contouring from head and neck CT and MR images. Med. Phys. 51, 2175–2186 (2024). https://doi.org/10.1002/mp.16924
Reinke, A., et al.: Understanding metric-related pitfalls in image analysis validation. Nat. Methods 21, 182–194 (2024). https://doi.org/10.1038/s41592-023-02150-0
Ryu, J., Kamata, S.: An efficient computational algorithm for Hausdorff distance based on points-ruling-out and systematic random sampling. Pattern Recogn. 114, 107857 (2021). https://doi.org/10.1016/j.patcog.2021.107857
Sangineto, E.: Pose and expression independent facial landmark localization using Dense-SURF and the Hausdorff distance. IEEE Trans. Pattern Anal. Mach. Intell. 35, 624–638 (2013). https://doi.org/10.1109/TPAMI.2012.87
Schroeder, W., Martin, K., Lorensen, B.: The Visualization Toolkit: An Object-Oriented Approach to 3D Graphics, 4th edn. Kitware (2006). https://isbndb.com/book/9781930934191
Sim, D.G., Kwon, O.K., Park, R.H.: Object matching algorithms using robust Hausdorff distance measures. IEEE Trans. Image Process. 8, 425–429 (1999). https://doi.org/10.1109/83.748897
Sim, D.G., Park, R.H.: Two-dimensional object alignment based on the robust oriented Hausdorff similarity measure. IEEE Trans. Image Process. 10, 475–483 (2001). https://doi.org/10.1109/83.908541
Taha, A.A., Hanbury, A.: An efficient algorithm for calculating the exact Hausdorff distance. IEEE Trans. Pattern Anal. Mach. Intell. 37, 2153–2163 (2015). https://doi.org/10.1109/TPAMI.2015.2408351
Taha, A.A., Hanbury, A.: Metrics for evaluating 3D medical image segmentation: analysis, selection, and tool. BMC Med. Imaging 15, 29 (2015). https://doi.org/10.1186/s12880-015-0068-x
Vrtovec, T., Močnik, D., Strojan, P., Pernuš, F., Ibragimov, B.: Auto-segmentation of organs at risk for head and neck radiotherapy planning: from atlas-based to deep learning methods. Med. Phys. 47, e929–e950 (2020). https://doi.org/10.1002/mp.14320
Zaffino, P., Raudaschl, P., Fritscher, K., Sharp, G.C., Spadea, M.F.: Technical note: plastimatch mabs, an open source tool for automatic image segmentation. Med. Phys. 43, 5155 (2016). https://doi.org/10.1118/1.4961121
Acknowledgments
This study was supported by the Slovenian Research and Innovation Agency (ARIS) under projects No. J2-4453, J2-50067 and P2-0232, and by the European Union Horizon project ARTILLERY under grant agreement No. 101080983.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Ethics declarations
Disclosure of Interests
The authors have no competing interests to declare that are relevant to the content of this article.
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Podobnik, G., Vrtovec, T. (2024). HDilemma: Are Open-Source Hausdorff Distance Implementations Equivalent?. In: Linguraru, M.G., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2024. MICCAI 2024. Lecture Notes in Computer Science, vol 15009. Springer, Cham. https://doi.org/10.1007/978-3-031-72114-4_30
Download citation
DOI: https://doi.org/10.1007/978-3-031-72114-4_30
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-72113-7
Online ISBN: 978-3-031-72114-4
eBook Packages: Computer ScienceComputer Science (R0)