Abstract
Deep reinforcement learning (DRL) is a promising technique for anatomical landmark detection in 3D medical images and a useful first step in automated medical imaging pathology detection. However, deployment of landmark detection in a pathology detection pipeline requires a self-assessment process to identify out-of-distribution images for manual review. We therefore propose a novel method derived from the full-width-half-maxima of q-value probability distributions for estimating the uncertainty of a distributional deep q-learning (dist-DQN) landmark detection agent. We trained two dist-DQN models targeting the locations of knee fibular styloid and intercondylar eminence of the tibia, using 1552 MR sequences (Sagittal PD, PDFS and T2FS) with an approximate 75%, 5%, 20% training, validation, and test split. Error for the two landmarks was 3.25 ± 0.12 mm and 3.06 ± 0.10 mm respectively (mean ± standard error). Mean error for the two landmarks was 28% lower than a non-distributional DQN baseline (3.16 ± 0.11 mm vs 4.36 ± 0.27 mm). Additionally, we demonstrate that the dist-DQN derived uncertainty metric has an AUC of 0.91 for predicting out-of-distribution images with a specificity of 0.77 at sensitivity 0.90, illustrating the double benefit of improved error rate and the ability to defer reviews to experts.
Keywords
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Alansary, A., et al.: Evaluating reinforcement learning agents for anatomical landmark detection. Med. Image Anal. 53, 156–164 (2019)
Bellemare, M.G., Dabney, W., Munos, R.: A distributional perspective on reinforcement learning. In: International Conference on Machine Learning, pp. 449–458. PMLR (2017)
Bellman, R.E.: Dynamic Programming. Princeton University Press, Princeton (1957)
Brockman, G., et al.: OpenAI Gym. arXiv:1606.01540 [cs] (2016)
Chua, K., Calandra, R., McAllister, R., Levine, S.: Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In: Advances in Neural Information Processing Systems 31 (2018)
Clements, W.R., Van Delft, B., Robaglia, B.-M., Slaoui, R.B., Toth, S.: Estimating risk and uncertainty in deep reinforcement learning. arXiv:1905.09638 [cs, stat] (2020)
Fortunato, M., et al.: Noisy networks for exploration. arXiv:1706.10295 [cs, stat] (2019)
Ghesu, F., et al.: Multi-scale deep reinforcement learning for real-time 3D-landmark detection in CT scans. IEEE Trans. Pattern Anal. Mach. Intell. 41, 176–189 (2019)
van Hasselt, H., Guez, A., Silver, D.: Deep reinforcement learning with double Q-learning. In: Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence 2094–2100. AAAI Press (2016)
Horgan, D., et al.: Distributed prioritized experience replay. In: International Conference on Learning Representations. ICLR (2018)
Kahn, G., Villaflor, A., Pong, V., Abbeel, P., Levine, S.: Uncertainty-aware reinforcement learning for collision avoidance. arXiv:1702.01182 [cs] (2017)
Mnih, V., et al.: Human-level control through deep reinforcement learning. Nature 518, 529–533 (2015)
Moritz, P., et al.: Ray: a distributed framework for emerging AI applications. arXiv:1712.05889 [cs, stat] (2018)
Nikolov, N., Kirschner, J., Berkenkamp, F., Krause, A.: Information-directed exploration for deep reinforcement learning. In: International Conference on Learning Representations. ICLR (2018)
Virtanen, P., et al.: SciPy 1.0: fundamental algorithms for scientific computing in Python. Nat. Methods 17, 261–272 (2020)
Vlontzos, A., Alansary, A., Kamnitsas, K., Rueckert, D., Kainz, B.: Multiple landmark detection using multi-agent reinforcement learning. arXiv:1907.00318 [cs] (2019)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Browning, J. et al. (2021). Uncertainty Aware Deep Reinforcement Learning for Anatomical Landmark Detection in Medical Images. In: de Bruijne, M., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2021. MICCAI 2021. Lecture Notes in Computer Science(), vol 12903. Springer, Cham. https://doi.org/10.1007/978-3-030-87199-4_60
Download citation
DOI: https://doi.org/10.1007/978-3-030-87199-4_60
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-87198-7
Online ISBN: 978-3-030-87199-4
eBook Packages: Computer ScienceComputer Science (R0)