Abstract
Instrument-tissue interaction detection in surgical videos is a fundamental problem for surgical scene understanding which is of great significance to computer-assisted surgery. However, few works focus on this fine-grained surgical activity representation. In this paper, we propose to represent instrument-tissue interaction as \(\langle \)instrument bounding box, tissue bounding box, instrument class, tissue class, action class\(\rangle \) quintuples. We present a novel quintuple detection network (QDNet) for the instrument-tissue interaction quintuple detection task in cataract surgery videos. Specifically, a spatiotemporal attention layer (STAL) is proposed to aggregate spatial and temporal information of the regions of interest between adjacent frames. We also propose a graph-based quintuple prediction layer (GQPL) to reason the relationship between instruments and tissues. Our method achieves an \(\textrm{mAP}\) of 42.24% on a cataract surgery video dataset, significantly outperforming other methods.
W. Lin and Y. Hu—Co-first authors.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Chao, Y.W., Zhan, W., He, Y., Wang, J., Jia, D.: Hico: A benchmark for recognizing human-object interactions in images. In: IEEE International Conference on Computer Vision (2015)
Chen, K., et al.: Application of computer-assisted virtual surgical procedures and three-dimensional printing of patient-specific pre-contoured plates in bicolumnar acetabular fracture fixation. Orthop. Traumatol. Surg. Res. 105, 877–884 (2019). https://doi.org/10.1016/j.otsr.2019.05.011
Chen, Y.W., Hanak, B.W., Yang, T.C., Wilson, T.A., Nagatomo, K.J.: Computer-assisted surgery in medical and dental applications. Expert Rev. Med. Devices 18(7), 669–696 (2021)
DiPietro, R.S., et al.: Recognizing surgical activities with recurrent neural networks. In: MICCAI (2016)
Gao, C., Zou, Y., Huang, J.B.: iCAN: instance-centric attention network for human-object interaction detection. In: British Machine Vision Conference (2018)
Gilmer, J., Schoenholz, S.S., Riley, P.F., Vinyals, O., Dahl, G.E.: Neural message passing for quantum chemistry. In: International Conference on Machine Learning, pp. 1263–1272. PMLR (2017)
Gupta, S., Malik, J.: Visual semantic role labeling. arXiv preprint arXiv:1505.04474 (2015)
Hashimoto, D.A., et al.: Computer vision analysis of intraoperative video: automated recognition of operative steps in laparoscopic sleeve gastrectomy. Ann. Surg. 270(3), 414 (2019)
He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask r-CNN. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2961–2969 (2017)
He, Y., Huang, T., Zhang, Y., An, J., He, L.H.: Application of a computer-assisted surgical navigation system in temporomandibular joint ankylosis surgery: a retrospective study. Int. J. Oral Maxillofac. Surg. 46, 189–197 (2016). https://doi.org/10.1016/j.ijom.2016.10.006
Islam, M., Lalithkumar, S., Ming, L.C., Ren, H.: Learning and reasoning with the graph structure representation in robotic surgery. CoRR arXiv preprint arXiv:2007.03357 (2020)
Jin, Y., et al.: SV-RCNet: workflow recognition from surgical videos using recurrent convolutional network. IEEE Trans. Med. Imaging 37(5), 1114–1126 (2018). https://doi.org/10.1109/TMI.2017.2787657
Jin, Y., et al.: Multi-task recurrent convolutional network with correlation loss for surgical video analysis. Med. Image Anal. 59, 101572 (2020)
Khatibi, Toktam, Dezyani, Parastoo: Proposing novel methods for gynecologic surgical action recognition on laparoscopic videos. Multimed. Tools Appl. 79(41), 30111–30133 (2020). https://doi.org/10.1007/s11042-020-09540-y
Lalys, F., Jannin, P.: Surgical process modelling: a review. Int. J. Comput. Assist. Radiol. Surg. 9(3), 495–511 (2013). https://doi.org/10.1007/s11548-013-0940-5
Lin, T.Y., Dollár, P., Girshick, R., He, K., Hariharan, B., Belongie, S.: Feature pyramid networks for object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2117–2125 (2017)
Nwoye, C.I., et al.: Recognition of instrument-tissue interactions in endoscopic videos via action triplets. In: Martel, A.L., et al. (eds.) MICCAI 2020. LNCS, vol. 12263, pp. 364–374. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59716-0_35
Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. In: Advances in Neural Information Processing Systems, vol.28 (2015)
Seenivasan, L., Mitheran, S., Islam, M., Ren, H.: Global-reasoned multi-task learning model for surgical scene understanding. IEEE Robot. Autom. Lett. 7(2), 3858–3865 (2022). https://doi.org/10.1109/LRA.2022.3146544
Twinanda, A.P., Shehata, S., Mutter, D., Marescaux, J., De Mathelin, M., Padoy, N.: Endonet: a deep architecture for recognition tasks on laparoscopic videos. IEEE Trans. Med. Imaging 36(1), 86–97 (2016)
Ulutan, O., Iftekhar, A., Manjunath, B.S.: Vsgnet: Spatial attention network for detecting human object interactions using graph convolutions. arXiv preprint arXiv:2003.05541 (2020)
Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems, vol.30 (2017)
Xu, M., Islam, M., Ming Lim, C., Ren, H.: Learning domain adaptation with model calibration for surgical report generation in robotic surgery. In: 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 12350–12356 (2021). https://doi.org/10.1109/ICRA48506.2021.9561569
Zhang, F.Z., Campbell, D., Gould, S.: Spatially conditioned graphs for detecting human-object interactions (2020)
Acknowledgement
This work was supported in part by The National Natural Science Foundation of China(8210072776), Guangdong Provincial Department of Education(2020ZDZX 3043), Guangdong Basic and Applied Basic Research Foundation(2021A1515012195), Shenzhen Natural Science Fund (JCYJ20200109140820699), the Stable Support Plan Program (20200925174052004), and AME Programmatic Fund (A20H4b0141).
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Lin, W. et al. (2022). Instrument-tissue Interaction Quintuple Detection in Surgery Videos. In: Wang, L., Dou, Q., Fletcher, P.T., Speidel, S., Li, S. (eds) Medical Image Computing and Computer Assisted Intervention – MICCAI 2022. MICCAI 2022. Lecture Notes in Computer Science, vol 13437. Springer, Cham. https://doi.org/10.1007/978-3-031-16449-1_38
Download citation
DOI: https://doi.org/10.1007/978-3-031-16449-1_38
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-16448-4
Online ISBN: 978-3-031-16449-1
eBook Packages: Computer ScienceComputer Science (R0)
