Abstract
Medical image segmentation is a monotonous, time-consuming, and costly task performed by highly skilled medical annotators. Despite adequate training, the intra- and inter-annotator variations results in significantly differing segmentations. If the variations arise from the uncertainty of the segmentation task, due to poor image contrast, lack of expert consensus, etc., then the algorithms for automatic segmentation should learn to capture the annotator (dis)agreements. In our approach we modeled the annotator (dis)agreement by aggregating the multi-annotator segmentations to reflect the uncertainty of the segmentation task and formulated the segmentation as multi-class pixel classification problem within an open source convolutional neural architecture nnU-Net. Validation was carried out for a wide range of imaging modalities and segmentation tasks as provided by the 2020 and 2021 QUBIQ (Quantification of Uncertainties in Biomedical Image Quantification) challenges. We achieved high quality segmentation results, despite a small set of training samples, and at time of this writing achieved an overall third and sixth best result on the respective QUBIQ 2020 and 2021 challenge leaderboards.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
September 9, 2021.
References
Hu, S., Worrall, D., Knegt, S., Veeling, B., Huisman, H., Welling, M.: Supervised uncertainty quantification for segmentation with multiple annotations. arXiv:1907.01949 [cs, stat], July 2019. http://arxiv.org/abs/1907.01949
Isensee, F., Jäger, P.F., Kohl, S.A.A., Petersen, J., Maier-Hein, K.H.: Automated design of deep learning methods for biomedical image segmentation. Nat Methods 18(2), 203–211 (2021). https://doi.org/10.1038/s41592-020-01008-z, http://arxiv.org/abs/1904.08128, arXiv: 1904.08128 version: 2
Isensee, F., et al.: nnU-Net: self-adapting framework for U-net-based medical image segmentation. arXiv:1809.10486 [cs], September 2018. http://arxiv.org/abs/1809.10486
Jungo, A., Meier, R., Ermis, E., Herrmann, E., Reyes, M.: Uncertainty-driven sanity check: application to postoperative brain tumor cavity segmentation. arXiv:1806.03106 [cs], June 2018. http://arxiv.org/abs/1806.03106
Karimi, D., Dou, H., Warfield, S.K., Gholipour, A.: Deep learning with noisy labels: exploring techniques and remedies in medical image analysis. Med. Image Anal. 65, 101759 (2020). https://doi.org/10.1016/j.media.2020.101759
Kohl, S.A.A., et al.: A probabilistic U-net for segmentation of ambiguous images. arXiv:1806.05034 [cs, stat], January 2019. http://arxiv.org/abs/1806.05034
Lampert, T.A., Stumpf, A., Gançarski, P.: An empirical study into annotator agreement, ground truth estimation, and algorithm evaluation. IEEE Trans. Image Process. 25(6), 2557–2572 (2016). https://doi.org/10.1109/TIP.2016.2544703, http://arxiv.org/abs/1307.0426
Langerak, T.R., van der Heide, U.A., Kotte, A.N.T.J., Viergever, M.A., van Vulpen, M., Pluim, J.P.W.: Label fusion in atlas-based segmentation using a selective and iterative method for performance level estimation (SIMPLE). IEEE Trans. Med. Imaging 29(12), 2000–2008 (2010). https://doi.org/10.1109/TMI.2010.2057442
Litjens, G., Debats, O., van de Ven, W., Karssemeijer, N., Huisman, H.: A pattern recognition approach to zonal segmentation of the prostate on MRI. In: Ayache, N., Delingette, H., Golland, P., Mori, K. (eds.) MICCAI 2012. LNCS, vol. 7511, pp. 413–420. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33418-4_51
Quantification of Uncertainties in Biomedical Image Quantification Challenge 2021. https://qubiq21.grand-challenge.org/. Accessed 11 Aug 2021
Ronneberger, O., Fischer, P., Brox, T.: U-net: convolutional networks for biomedical image segmentation. arXiv:1505.04597 [cs], May 2015. http://arxiv.org/abs/1505.04597
Tanno, R., Saeedi, A., Sankaranarayanan, S., Alexander, D.C., Silberman, N.: Learning from noisy labels by regularized estimation of annotator confusion. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019
Warfield, S.K., Zou, K.H., Wells, W.M.: Simultaneous truth and performance level estimation (STAPLE): an algorithm for the validation of image segmentation. IEEE Trans. Med. Imaging 23(7), 903–921 (2004). https://doi.org/10.1109/TMI.2004.828354, https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1283110/
Zhang, Y., et al.: U-net-and-a-half: convolutional network for biomedical image segmentation using multiple expert-driven annotations. arXiv:2108.04658 [cs], August 2021. http://arxiv.org/abs/2108.04658
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Žukovec, M., Dular, L., Špiclin, Ž. (2022). Modeling Multi-annotator Uncertainty as Multi-class Segmentation Problem. In: Crimi, A., Bakas, S. (eds) Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries. BrainLes 2021. Lecture Notes in Computer Science, vol 12962. Springer, Cham. https://doi.org/10.1007/978-3-031-08999-2_9
Download citation
DOI: https://doi.org/10.1007/978-3-031-08999-2_9
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-08998-5
Online ISBN: 978-3-031-08999-2
eBook Packages: Computer ScienceComputer Science (R0)
