Abstract
Surgical instrument detection is a fundamental task of a robotic scrub nurse. For this, image-based deep learning techniques are effective but usually demand large amounts of annotated data, whose creation is expensive and time-consuming. In this work, we propose a strategy based on the copy-paste technique for the generation of reliable synthetic image training data with a minimal amount of annotation effort. Our approach enables the efficient in situ creation of datasets for specific surgeries and contexts. We study the amount of employed manually annotated data and training set sizes on our model’s performance, as well as different blending techniques for improved training data. We achieve 91.9 box mAP and 91.6 mask mAP, training solely on synthetic data, in a real-world scenario. Our evaluation relies on an annotated image dataset of the wisdom teeth extraction surgery set, created in an actual operating room. This dataset, the corresponding code, and further data are made publicly available (https://github.com/Jorebs/Modular-Label-Efficient-Dataset-Generation-for-Instrument-Detection-for-Robotic-Scrub-Nurses).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Marć, M., Bartosiewicz, A., Burzyńska, J., Chmiel, Z., Januszewicz, P.: A nursing shortage-a prospect of global and local policies. Int. Nurs. Rev. 66(1), 9–16 (2019)
Haczyński, J., Skrzypczak, Z., Winter, M.: Nurses in Poland-immediate action needed. Eng. Manag. Prod. Serv. 9(2), 97–104 (2017)
Lowman, G.H., Harms, P.D.: Addressing the nurse workforce crisis: a call for greater integration of the organizational behavior, human resource management and nursing literatures. J. Manag. Psychol. 37(3), 294–303 (2022)
Harms, P.D.: Nursing: a critical profession in a perilous time. Ind. Organ. Psychol. 14(1–2), 264–266 (2021)
Zemmar, A., Lozano, A.M., Nelson, B.J.: The rise of robots in surgical environments during COVID-19. Nat. Mach. Intell. 2(10), 566–572 (2020)
Kyrarini, M., et al.: A survey of robots in healthcare. Technologies 9(1), 8 (2021)
Appendectomy Set, New Med Instruments. https://new-medinstruments.com/appendectomy-set.html. Accessed 26 May 2023
Glaucoma Surgical Instrument Set, New Med Instruments. https://new-medinstruments.com/surgery-sets/general-surgery-instruments-sets.html/glaucoma-surgical-instrument-set.html. Accessed 26 May 2023
Major General Surgery Set, New Med Instruments. https://new-medinstruments.com/surgery-sets/general-surgery-instruments-sets.html/general-surgery-set.html. Accessed 26 May 2023
AlHajj, H., Lamard, M., Conze, P.H., et al.: Challenge on automatic tool annotation for cataract surgery: cataracts. Med. Image Anal. 52, 24–41 (2019). https://doi.org/10.1016/j.media.2018.11.00
Allan, M., Shvets, A., Kurmann, T., et al.: 2017 robotic instrument segmentation challenge. ArXiv arXiv:1902:06426 (2019)
Ross, T., Reinke, A., Full, P.M., et al.: Robust medical instrument segmentation challenge. ArXiv preprint (2019)
Twinanda, A.P., Shehata, S., Mutter, D., et al.: EndoNet: a deep architecture for recognition tasks on laparoscopic videos. IEEE Trans. Med. Imaging 36, 86–97 (2017). https://doi.org/10.1109/TMI.2016.2593957
Rodrigues, M., Mayo, M., Patros, P.: Evaluation of deep learning techniques on a novel hierarchical surgical tool dataset. In: Long, G., Yu, X., Wang, S. (eds.) AI 2021. LNCS, vol. 13151, pp. 169–180. Springer, Cham (2022). https://doi.org/10.1007/978-3-030-97546-3_14
Badilla-Solórzano, J., Spindeldreier, S., Ihler, S., Gellrich, N.C., Spalthoff, S.: Deep-learning-based instrument detection for intra-operative robotic assistance. Int. J. Comput. Assist. Radiol. Surg. 17(9), 1685–1695 (2022)
Peng, H., et al.: Reducing annotating load: active learning with synthetic images in surgical instrument segmentation. arXiv preprint arXiv:2108.03534 (2021)
Dwibedi, D., Misra, I., Hebert, M.: Cut, paste and learn: surprisingly easy synthesis for instance detection. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1301–1310 (2017)
Dvornik, N., Mairal, J., Schmid, C.: Modeling visual context is key to augmenting object detection datasets. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 364–380 (2018)
Ghiasi, G., et al.: Simple copy-paste is a strong data augmentation method for instance segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2918–2928 (2021)
Wang, A., Islam, M., Xu, M., Ren, H.: Rethinking surgical instrument segmentation: a background image can be all you need. In: Wang, L., Dou, Q., Fletcher, P.T., Speidel, S., Li, S. (eds.) MICCAI 2022. LNCS, vol. 13437, pp. 355–364. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-16449-1_34
Garcia-Peraza-Herrera, L.C., Fidon, L., D’Ettorre, C., Stoyanov, D., Vercauteren, T., Ourselin, S.: Image compositing for segmentation of surgical tools without manual annotations. IEEE Trans. Med. Imaging 40(5), 1450–1460 (2021)
Hasty: Adaptive Automation for Vision AI. 2023 Hasty GmbH. https://app.hasty.ai. Accessed 28 May 2023
He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask R-CNN. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2961–2969 (2017)
Lin, T.Y., et al.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10602-1_48
Pérez, P., Gangnet, M., Blake, A.: Poisson image editing. In: ACM SIGGRAPH 2003 Papers, pp. 313–318 (2003)
Jocher, G., et al.: ultralytics/yolov5: v7. 0-YOLOv5 SOTA realtime instance segmentation. Zenodo (2022)
Jocher, G., Chaurasia, A., Qiu, J.: YOLO by Ultralytics (version 8.0.0) [computer software]. https://github.com/ultralytics/ultralytics. Accessed 1 June 2023
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End-to-end object detection with transformers. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.M. (eds.) ECCV 2020. LNCS, vol. 12346, pp. 213–229. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58452-8_13
Grammatikopoulou, M., et al.: CaDIS: cataract dataset for surgical RGB-image segmentation. Med. Image Anal. 71, 102053 (2021)
Jiang, W., Xia, T., Wang, Z., Jia, F.: Semi-supervised surgical tool detection based on highly confident pseudo labeling and strong augmentation driven consistency. In: Engelhardt, S., et al. (eds.) DGM4MICCAI DALI 2021. LNCS, vol. 13003, pp. 154–162. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-88210-5_14
Acknowledgements
The authors are deeply thankful for the support provided by the University of Costa Rica, which enabled the creation of this document.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Badilla-Solórzano, J., Gellrich, NC., Seel, T., Ihler, S. (2024). Modular, Label-Efficient Dataset Generation for Instrument Detection for Robotic Scrub Nurses. In: Xue, Y., Chen, C., Chen, C., Zuo, L., Liu, Y. (eds) Data Augmentation, Labelling, and Imperfections. MICCAI 2023. Lecture Notes in Computer Science, vol 14379. Springer, Cham. https://doi.org/10.1007/978-3-031-58171-7_10
Download citation
DOI: https://doi.org/10.1007/978-3-031-58171-7_10
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-58170-0
Online ISBN: 978-3-031-58171-7
eBook Packages: Computer ScienceComputer Science (R0)
