Abstract
Hyperspectral imaging (HSI) captures a greater level of spectral detail than traditional optical imaging, making it a potentially valuable intraoperative tool when precise tissue differentiation is essential. Hardware limitations of current optical systems used for handheld real-time video HSI result in a limited focal depth, thereby posing usability issues for integration of the technology into the operating room. This work integrates a focus-tunable liquid lens into a video HSI exoscope, and proposes novel video autofocusing methods based on deep reinforcement learning. A first-of-its-kind robotic focal-time scan was performed to create a realistic and reproducible testing dataset. We benchmarked our proposed autofocus algorithm against traditional policies, and found our novel approach to perform significantly (\(p<0.05\)) better than traditional techniques (\(0.070\pm .098\) mean absolute focal error compared to \(0.146\pm .148\)). In addition, we performed a blinded usability trial by having two neurosurgeons compare the system with different autofocus policies, and found our novel approach to be the most favourable, making our system a desirable addition for intraoperative HSI.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Anikina, A., Rogov, O.Y., Dylov, D.V.: Dasha: decentralized autofocusing system with hierarchical agents. arXiv preprint arXiv:2108.12842 (2021)
Bogaards, A., et al.: Increased brain tumor resection using fluorescence image guidance in a preclinical model. Lasers Surg. Med. 35(3), 181–190 (2004)
Budd, C., Herrera, L.C.G.P., Huber, M., Ourselin, S., Vercauteren, T.: Rapid and robust endoscopic content area estimation: a lean gpu-based pipeline and curated benchmark dataset. In: Computer Methods in Biomechanics and Biomedical Engineering: Imaging & Visualization, pp. 1–10 (2023)
Ebner, M., et al.: Intraoperative hyperspectral label-free imaging: from system design to first-in-patient translation. J. Phys. D Appl. Phys. 54(29) (2021). https://doi.org/10.1088/1361-6463/abfbf6. https://www.scopus.com/inward/record.url?scp=85107008535 &partnerID=8YFLogxK
Halicek, M., Fabelo, H., Ortega, S., Callico, G.M., Fei, B.: In-vivo and ex-vivo tissue analysis through hyperspectral imaging techniques: revealing the invisible features of cancer. Cancers 11(6) (2019). https://doi.org/10.3390/cancers11060756. https://www.mdpi.com/2072-6694/11/6/756
Herrmann, C., et al.: Learning to autofocus. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020)
Ivanov, T., Kumar, A., Sharoukhov, D., Ortega, F., Putman, M.: DeepFocus: a deep learning model for focusing microscope systems. In: Zelinski, M.E., Taha, T.M., Howe, J., Awwal, A.A.S., Iftekharuddin, K.M. (eds.) Applications of Machine Learning 2020, vol. 11511, p. 1151103. International Society for Optics and Photonics, SPIE (2020). https://doi.org/10.1117/12.2568990
Jia, D., Zhang, C., Wu, N., Zhou, J., Guo, Z.: Autofocus algorithm using optimized laplace evaluation function and enhanced mountain climbing search algorithm. Multimedia Tools Appl. 81(7), 10299–10311 (2022)
Liu, Y.Q., Du, X., Shen, H.L., Chen, S.J.: Estimating generalized gaussian blur kernels for out-of-focus image deblurring. IEEE Trans. Circ. Syst. Video Technol. 31(3), 829–843 (2021). https://doi.org/10.1109/TCSVT.2020.2990623
Mnih, V., et al.: Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 (2013)
Pichette, J.: Intraoperative video-rate hemodynamic response assessment in human cortex using snapshot hyperspectral optical imaging. Neurophotonics 3(4), 045003 (2016)
Shapey, J., et al.: Intraoperative multispectral and hyperspectral label-free imaging: a systematic review of in vivo clinical studies. J. Biophoton. 12(9), e201800455 (2019)
Twinanda, A.P., Shehata, S., Mutter, D., Marescaux, J., de Mathelin, M., Padoy, N.: Endonet: a deep architecture for recognition tasks on laparoscopic videos. CoRR abs/1602.03012 (2016). https://arxiv.org/abs/1602.03012
Wang, C., Huang, Q., Cheng, M., Ma, Z., Brady, D.J.: Deep learning for camera autofocus. IEEE Trans. Comput. Imaging 7, 258–271 (2021). https://doi.org/10.1109/TCI.2021.3059497
Yu, X., Yu, R., Yang, J., Duan, X.: A robotic auto-focus system based on deep reinforcement learning. In: 2018 15th International Conference on Control, Automation, Robotics and Vision (ICARCV), pp. 204–209. IEEE (2018)
Acknowledgements
This study/project is funded by the NIHR [NIHR202114]. This work was supported by core funding from the Wellcome/EPSRC [WT203148/Z/16/Z; NS/A000049/1]. This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 101016985 (FAROS project). TV is supported by a Medtronic/RAEng Research Chair [
]. For the purpose of open access, the authors have applied a CC BY public copyright licence to any Author Accepted Manuscript version arising from this submission. TV is a co-founder and shareholder of Hypervision Surgical.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Budd, C. et al. (2023). Deep Reinforcement Learning Based System for Intraoperative Hyperspectral Video Autofocusing. In: Greenspan, H., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2023. MICCAI 2023. Lecture Notes in Computer Science, vol 14228. Springer, Cham. https://doi.org/10.1007/978-3-031-43996-4_63
Download citation
DOI: https://doi.org/10.1007/978-3-031-43996-4_63
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-43995-7
Online ISBN: 978-3-031-43996-4
eBook Packages: Computer ScienceComputer Science (R0)
