Skip to main content

HDilemma: Are Open-Source Hausdorff Distance Implementations Equivalent?

  • Conference paper
  • First Online:
Medical Image Computing and Computer Assisted Intervention – MICCAI 2024 (MICCAI 2024)

Abstract

Quantitative performance metrics play a pivotal role in medical imaging by offering critical insights into method performance and facilitating objective method comparison. Recently, platforms providing recommendations for metrics selection as well as resources for evaluating methods through computational challenges and online benchmarking have emerged, with an inherent assumption that metrics implementations are consistent across studies and equivalent throughout the community. In this study, we question this assumption by reviewing five different open-source implementations for computing the Hausdorff distance (HD), a boundary-based metric commonly used for assessing the performance of semantic segmentation. Despite sharing a single generally accepted mathematical definition, our experiments reveal notable systematic differences in the HD and its 95th percentile variant across implementations when applied to clinical segmentations with varying voxel sizes, which fundamentally impacts and constrains the ability to objectively compare results across different studies. Our findings should encourage the medical imaging community towards standardizing the implementation of the HD computation, so as to foster objective, reproducible and consistent comparisons when reporting performance results.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    https://miccai.org/index.php/special-interest-groups/challenges/.

  2. 2.

    https://huggingface.co/.

  3. 3.

    https://paperswithcode.com/.

  4. 4.

    https://metrics-reloaded.dkfz.de/.

  5. 5.

    Meshing is based on the marching cubes algorithm [14] that takes a 3D mask A and generates a 3D mesh \(\mathcal {M}_{\!A}\) consisting of vertices and surfels (i.e. triangular faces).

  6. 6.

    Voxelization (3D analogue of rasterization) is based on vtkPolyDataToImageStencil from the Visualization Toolkit (VTK) [22] that allows completely bijective transforms between the mesh and image space, i.e. voxelization of \(\mathcal {M}_{\!A}\), obtained from A by meshing without smoothing, results again in A.

  7. 7.

    Based on vtkImplicitPolyDataDistance in the VTK [22].

  8. 8.

    https://github.com/google-deepmind/surface-distance, v0.1, sha1-hash: ee651c8.

  9. 9.

    https://github.com/Project-MONAI/MetricsReloaded, v0.1.0, sha1-hash: b3a3715.

  10. 10.

    https://github.com/Project-MONAI/MONAI, v1.3.0, sha1-hash: 865972f.

  11. 11.

    https://gitlab.com/plastimatch/plastimatch, v1.9.4, sha1-hash: 581c7692.

  12. 12.

    https://github.com/Jingnan-Jia/segmentation_metrics, v1.6.1, sha1-hash: fed1852.

  13. 13.

    This is indeed the case for Plastimatch that does not implement \(H\!D_{100}\) as a 100-percentile but uses a separate calculation pipeline based on max of all distances.

References

  1. Alt, H., Guibas, L.J.: Discrete geometric shapes: matching, interpolation, and approximation. In: Handbook of Computational Geometry, chap. 3, pp. 121–153. Elsevier (2000). https://doi.org/10.1016/B978-0-444-82537-7.X5000-1

  2. Aydin, O.U., Taha, A.A., Hilbert, A., et al.: On the usage of average Hausdorff distance for segmentation performance assessment: hidden error when used for ranking. Eur. Radiol. Exp. 5, 4 (2021). https://doi.org/10.1186/s41747-020-00200-2

    Article  Google Scholar 

  3. Cardoso, M.J., Li, W., Brown, R., et al.: MONAI: an open-source framework for deep learning in healthcare. arXiv:2211.02701 (2022). https://doi.org/10.48550/arXiv.2211.02701

  4. Duprez, M., Bordas, S.P.A., Bucki, M., et al.: Quantifying discretization errors for soft tissue simulation in computer assisted surgery: a preliminary study. Appl. Math. Model. 77, 709–723 (2020). https://doi.org/10.1016/j.apm.2019.07.055

    Article  MathSciNet  Google Scholar 

  5. Fedorov, A., Beichel, R., Kalpathy-Cramer, J., et al.: 3D Slicer as an image computing platform for the Quantitative Imaging Network. Magn. Reson. Imaging 30, 1323–1341 (2012). https://doi.org/10.1016/j.mri.2012.05.001

    Article  Google Scholar 

  6. Felzenszwalb, P.F., Huttenlocher, D.P.: Distance transforms of sampled functions. Theory Comput. 8, 415–428 (2012). https://doi.org/10.4086/toc.2012.v008a019

    Article  MathSciNet  Google Scholar 

  7. Hausdorff, F.: Grundzüge der Mengenlehre [Basics of Set Theory]. Leipzig Viet, Leipzig, Germany (1914). https://archive.org/details/grundzgedermen00hausuoft/

  8. Huttenlocher, D.P., Klanderman, G.A., Rucklidge, W.J.: Comparing images using the Hausdorff distance. IEEE Trans. Pattern Anal. Mach. Intell. 15, 850–863 (1993). https://doi.org/10.1109/34.232073

    Article  Google Scholar 

  9. Jia, J., Staring, M., Stoel, B.C.: Seg-metrics: a Python package to compute segmentation metrics. medRxiv (2024). https://doi.org/10.1101/2024.02.22.24303215

  10. Jungeblut, P., Kleist, L., Miltzow, T.: The complexity of the Hausdorff distance. Discret. Comput. Geom. 71, 177–213 (2024). https://doi.org/10.1007/s00454-023-00562-5

    Article  MathSciNet  Google Scholar 

  11. Karimi, D., Salcudean, S.E.: Reducing the Hausdorff distance in medical image segmentation with convolutional neural networks. IEEE Trans. Med. Imaging 39, 499–513 (2020). https://doi.org/10.1109/TMI.2019.2930068

    Article  Google Scholar 

  12. Klette, R., Rosenfeld, A.: Metrics. In: Digital Geometry: Geometric Methods for Digital Picture Analysis, chap. 3, pp. 77–116. Elsevier (2004). https://doi.org/10.1016/B978-1-55860-861-0.X5000-7

  13. Li, W., Liang, Z., Ma, P., Wang, R., Cui, X., Chen, P.: Hausdorff GAN: improving GAN generation quality with Hausdorff metric. IEEE Trans. Cybern. 52, 10407–10419 (2022). https://doi.org/10.1109/TCYB.2021.3062396

    Article  Google Scholar 

  14. Lorensen, W.E., Cline, H.E.: Marching cubes: a high resolution 3D surface construction algorithm. In: 14th Conference on Computer Graphics and Interactive Techniques - SIGGRAPH 1987, pp. 163–169. ACM (1987). https://doi.org/10.1145/37401.37422

  15. Mackay, K., Bernstein, D., Glocker, B., Kamnitsas, K., Taylor, A.: A review of the metrics used to assess auto-contouring systems in radiotherapy. Clin. Oncol. 35, 354–369 (2023). https://doi.org/10.1016/j.clon.2023.01.016

    Article  Google Scholar 

  16. Maier-Hein, L., et al.: Metrics reloaded: recommendations for image analysis validation. Nat. Methods 21, 195–212 (2024). https://doi.org/10.1038/s41592-023-02151-z

  17. Nikolov, S., Blackwell, S., Zverovitch, A., et al.: Clinically applicable segmentation of head and neck anatomy for radiotherapy: deep learning algorithm development and validation study. J. Med. Internet Res. 23, e26151 (2021). https://doi.org/10.2196/26151

    Article  Google Scholar 

  18. Podobnik, G., Ibragimov, B., Strojan, P., Peterlin, P., Vrtovec, T.: vOARiability: interobserver and intermodality variability analysis in OAR contouring from head and neck CT and MR images. Med. Phys. 51, 2175–2186 (2024). https://doi.org/10.1002/mp.16924

    Article  Google Scholar 

  19. Reinke, A., et al.: Understanding metric-related pitfalls in image analysis validation. Nat. Methods 21, 182–194 (2024). https://doi.org/10.1038/s41592-023-02150-0

  20. Ryu, J., Kamata, S.: An efficient computational algorithm for Hausdorff distance based on points-ruling-out and systematic random sampling. Pattern Recogn. 114, 107857 (2021). https://doi.org/10.1016/j.patcog.2021.107857

    Article  Google Scholar 

  21. Sangineto, E.: Pose and expression independent facial landmark localization using Dense-SURF and the Hausdorff distance. IEEE Trans. Pattern Anal. Mach. Intell. 35, 624–638 (2013). https://doi.org/10.1109/TPAMI.2012.87

    Article  Google Scholar 

  22. Schroeder, W., Martin, K., Lorensen, B.: The Visualization Toolkit: An Object-Oriented Approach to 3D Graphics, 4th edn. Kitware (2006). https://isbndb.com/book/9781930934191

  23. Sim, D.G., Kwon, O.K., Park, R.H.: Object matching algorithms using robust Hausdorff distance measures. IEEE Trans. Image Process. 8, 425–429 (1999). https://doi.org/10.1109/83.748897

    Article  Google Scholar 

  24. Sim, D.G., Park, R.H.: Two-dimensional object alignment based on the robust oriented Hausdorff similarity measure. IEEE Trans. Image Process. 10, 475–483 (2001). https://doi.org/10.1109/83.908541

    Article  Google Scholar 

  25. Taha, A.A., Hanbury, A.: An efficient algorithm for calculating the exact Hausdorff distance. IEEE Trans. Pattern Anal. Mach. Intell. 37, 2153–2163 (2015). https://doi.org/10.1109/TPAMI.2015.2408351

    Article  Google Scholar 

  26. Taha, A.A., Hanbury, A.: Metrics for evaluating 3D medical image segmentation: analysis, selection, and tool. BMC Med. Imaging 15, 29 (2015). https://doi.org/10.1186/s12880-015-0068-x

    Article  Google Scholar 

  27. Vrtovec, T., Močnik, D., Strojan, P., Pernuš, F., Ibragimov, B.: Auto-segmentation of organs at risk for head and neck radiotherapy planning: from atlas-based to deep learning methods. Med. Phys. 47, e929–e950 (2020). https://doi.org/10.1002/mp.14320

  28. Zaffino, P., Raudaschl, P., Fritscher, K., Sharp, G.C., Spadea, M.F.: Technical note: plastimatch mabs, an open source tool for automatic image segmentation. Med. Phys. 43, 5155 (2016). https://doi.org/10.1118/1.4961121

    Article  Google Scholar 

Download references

Acknowledgments

This study was supported by the Slovenian Research and Innovation Agency (ARIS) under projects No. J2-4453, J2-50067 and P2-0232, and by the European Union Horizon project ARTILLERY under grant agreement No. 101080983.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Gašper Podobnik .

Editor information

Editors and Affiliations

Ethics declarations

Disclosure of Interests

The authors have no competing interests to declare that are relevant to the content of this article.

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Podobnik, G., Vrtovec, T. (2024). HDilemma: Are Open-Source Hausdorff Distance Implementations Equivalent?. In: Linguraru, M.G., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2024. MICCAI 2024. Lecture Notes in Computer Science, vol 15009. Springer, Cham. https://doi.org/10.1007/978-3-031-72114-4_30

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-72114-4_30

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-72113-7

  • Online ISBN: 978-3-031-72114-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics