Abstract
Eye trackers can provide visual guidance to sonographers during ultrasound (US) scanning. Such guidance is potentially valuable for less experienced operators to improve their scanning skills on how to manipulate the probe to achieve the desired plane. In this paper, a multimodal guidance approach (Multimodal-GuideNet) is proposed to capture the stepwise dependency between a real-world US video signal, synchronized gaze, and probe motion within a unified framework. To understand the causal relationship between gaze movement and probe motion, our model exploits multitask learning to jointly learn two related tasks: predicting gaze movements and probe signals that an experienced sonographer would perform in routine obstetric scanning. The two tasks are associated by a modality-aware spatial graph to detect the co-occurrence among the multi-modality inputs and share useful cross-modal information. Instead of a deterministic scanning path, Multimodal-GuideNet allows for scanning diversity by estimating the probability distribution of real scans. Experiments performed with three typical obstetric scanning examinations show that the new approach outperforms single-task learning for both probe motion guidance and gaze movement prediction. The prediction can also provide a visual guidance signal with an error rate of less than 10 pixels for a 224 \(\times \) 288 US image.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Baumgartner, C.F., et al.: SonoNet: real-time detection and localisation of fetal standard scan planes in freehand ultrasound. IEEE Trans. Med. Imaging 36(11), 2204–2215 (2017)
Cai, Y., Sharma, H., Chatelain, P., Noble, J.A.: Multi-task SonoEyeNet: detection of fetal standardized planes assisted by generated sonographer attention maps. In: International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI), pp. 871–879 (2018)
Cai, Y., Sharma, H., Chatelain, P., Noble, J.A.: SonoEyeNet: standardized fetal ultrasound plane detection informed by eye tracking. In: IEEE International Symposium on Biomedical Imaging (ISBI), pp. 1475–1478 (2018)
Cho, K., van Merrienboer, B., Gulcehre, C., Bougares, F., Schwenk, H., Bengio, Y.: Learning phrase representations using RNN encoder-decoder for statistical machine translation. In: Conference on Empirical Methods in Natural Language Processing (EMNLP) (2014)
Droste, R., et al.: Ultrasound image representation learning by modeling sonographer visual attention. In: International Conference on Information Processing in Medical Imaging, pp. 592–604 (2019)
Droste, R., Drukker, L., Papageorghiou, A.T., Noble, J.A.: Automatic probe movement guidance for freehand obstetric ultrasound. In: International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI), pp. 583–592 (2020)
Drukker, L., et al.: Transforming obstetric ultrasound into data science using eye tracking, voice recording, transducer motion and ultrasound video. Sci. Rep. 11(1), 1–12 (2021)
Graves, A.: Generating sequences with recurrent neural networks. arXiv:1308.0850 (2013)
Gupta, A., Johnson, J., Fei-Fei, L., Savarese, S., Alahi, A.: Social GAN: socially acceptable trajectories with generative adversarial networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2255–2264 (2018)
Housden, R.J., Treece, G.M., Gee, A.H., Prager, R.W.: Calibration of an orientation sensor for freehand 3D ultrasound and its use in a hybrid acquisition system. Biomed. Eng. Online 7(1), 1–13 (2008)
Li, K., et al.: Autonomous navigation of an ultrasound probe towards standard scan planes with deep reinforcement learning. In: IEEE International Conference on Robotics and Automation (ICRA), pp. 8302–8308 (2021)
Li, Y., Zemel, R., Brockschmidt, M., Tarlow, D.: Gated graph sequence neural networks. In: International Conference on Learning Representations (ICLR) (2016)
Mustafa, A.S.B., et al.: Development of robotic system for autonomous liver screening using ultrasound scanning device. In: IEEE International Conference on Robotics and Biomimetics (ROBIO), pp. 804–809 (2013)
Prevost, R., et al.: 3D freehand ultrasound without external tracking using deep learning. Med. Image Anal. 48, 187–202 (2018)
Salomon, L.J., et al.: Practice guidelines for performance of the routine mid-trimester fetal ultrasound scan. Ultrasound Obstet. Gynecol. 37(1), 116–126 (2011)
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L.C.: MobileNetV2: inverted residuals and linear bottlenecks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4510–4520 (2018)
Teng, C., Sharma, H., Drukker, L., Papageorghiou, A.T., Noble, J.A.: Towards scale and position invariant task classification using normalised visual scanpaths in clinical fetal ultrasound. In: International Workshop on Advances in Simplifying Medical Ultrasound, pp. 129–138 (2021)
Toporek, G., Wang, H., Balicki, M., Xie, H.: Autonomous image-based ultrasound probe positioning via deep learning. In: Hamlyn Symposium on Medical Robotics (2018)
Wang, S., et al.: Robotic-assisted ultrasound for fetal imaging: evolution from single-arm to dual-arm system. In: Annual Conference Towards Autonomous Robotic Systems, pp. 27–38 (2019)
Yan, S., Xiong, Y., Lin, D.: Spatial temporal graph convolutional networks for skeleton-based action recognition. In: Thirty-Second AAAI Conference on Artificial Intelligence (2018)
Zhang, P., Lan, C., Zeng, W., Xing, J., Xue, J., Zheng, N.: Semantics-guided neural networks for efficient skeleton-based human action recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1112–1121 (2020)
Zhao, C., Droste, R., Drukker, L., Papageorghiou, A.T., Noble, J.A.: Visual-assisted probe movement guidance for obstetric ultrasound scanning using landmark retrieval. In: International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI), pp. 670–679 (2021)
Acknowledgements
We acknowledge the ERC (ERC-ADG-2015 694581, project PULSE), the EPSRC (EP/MO13774/1, EP/R013853/1), and the NIHR Oxford Biomedical Research Centre.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Men, Q., Teng, C., Drukker, L., Papageorghiou, A.T., Noble, J.A. (2022). Multimodal-GuideNet: Gaze-Probe Bidirectional Guidance in Obstetric Ultrasound Scanning. In: Wang, L., Dou, Q., Fletcher, P.T., Speidel, S., Li, S. (eds) Medical Image Computing and Computer Assisted Intervention – MICCAI 2022. MICCAI 2022. Lecture Notes in Computer Science, vol 13437. Springer, Cham. https://doi.org/10.1007/978-3-031-16449-1_10
Download citation
DOI: https://doi.org/10.1007/978-3-031-16449-1_10
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-16448-4
Online ISBN: 978-3-031-16449-1
eBook Packages: Computer ScienceComputer Science (R0)
