Skip to main content
Springer Nature Link
Log in

Improving situation recognition using endoscopic videos and navigation information for endoscopic sinus surgery

  • Original Article
  • Published:
International Journal of Computer Assisted Radiology and Surgery Aims and scope Submit manuscript

Abstract

Purpose

Endoscopic sinus surgery (ESS) is widely used to treat chronic sinusitis. However, it involves the use of surgical instruments in a narrow surgical field in close proximity to vital organs, such as the brain and eyes. Thus, an advanced level of surgical skill is expected of surgeons performing this surgery. In a previous study, endoscopic images and surgical navigation information were used to develop an automatic situation recognition method in ESS. In this study, we aimed to develop a more accurate automatic surgical situation recognition method for ESS by improving the method proposed in our previous study and adding post-processing to remove incorrect recognition.

Method

We examined the training model parameters and the number of long short-term memory (LSTM) units, modified the input data augmentation method, and added post-processing. We also evaluated the modified method using clinical data.

Result

The proposed improvements improved the overall scene recognition accuracy compared with the previous study. However, phase recognition did not exhibit significant improvement. In addition, the application of the one-dimensional median filter significantly reduced short-time false recognition compared with the time series results. Furthermore, post-processing was required to set constraints on the transition of the scene to further improve recognition accuracy.

Conclusion

We suggested that the scene recognition could be improved by considering the model parameter, adding the one-dimensional filter and post-processing. However, the scene recognition accuracy remained unsatisfactory. Thus, a more accurate scene recognition and appropriate post-processing method is required.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Sireci F, Lorusso F, Martines F, Salvago P, Immordino A, Dispenza F, Gallina S, Canevari FR (2021) Guide to the management of complications in endoscopic sinus surgery (ESS). In: Duncan LT (ed) Advances in Health and Disease, vol 37. Nova Science Publishers, New York, pp 159–176

    Google Scholar 

  2. Sireci F, Gazia F, Dispenza F, Galletti B (2021) The role of computer assisted navigation in the endoscopic sinus surgery for the chronic rhinosinusitis. In: Duncan LT (ed) Advances in health and disease, vol 37. Nova Science Publishers, New York, pp 209–216

    Google Scholar 

  3. Dalgorf DM, Sacks R, Wormald PJ, Naidoo Y, Panizza B, Uren B, Brown C, Curotta J, Snidvongs K, Harvey RJ (2013) Image-guided surgery influences perioperative morbidity from endoscopic sinus surgery: a systematic review and meta-analysis. Otolaryngol Head Neck Surg 149(1):17–29. https://doi.org/10.1177/0194599813488519

    Article  Google Scholar 

  4. Sugino T, Nakamura R, Kuboki A, Honda O, Yamamoto M, Ohtori N (2019) Comparative analysis of surgical processes for image-guided endoscopic sinus surgery. Int J Comput Assist Radiol Surg 14(1):93–104. https://doi.org/10.1007/s11548-018-1855-y

    Article  Google Scholar 

  5. Yamaguchi T, Nakamura R, Kuboki A, Sawano Y, Ebata R, Otori N (2020) Skill assessment and visualization system for endoscopic sinus surgery: A clinical study. J Jpn Soc Comput Aid Surg 22(1):21–32. https://doi.org/10.5759/jscas.22.21

    Article  Google Scholar 

  6. Pernek I, Ferscha A (2017) A survey of context recognition in surgery. Med Biol Eng Comput 55(10):1719–1734. https://doi.org/10.1007/s11517-017-1670-6

    Article  Google Scholar 

  7. Aksamentov I, Twinanda AP, Mutter D, Marescaux J, Padoy N (2017) Deep neural networks predict remaining surgery duration from cholecystectomy videos. Medical image computing and computer-assisted intervention − MICCAI 2017. Springer, Cham, pp 586–593

    Chapter  Google Scholar 

  8. Padoy N, Blum T, Ahmadi SA, Feussner H, Berger MO, Navab N (2012) Statistical modeling and recognition of surgical workflow. Med Image Anal 16(3):632–641. https://doi.org/10.1016/j.media.2010.10.001

    Article  Google Scholar 

  9. Bardram JE, Doryab A, Jensen RM, Lange PM, Nielsen KLG, Petersen ST (2011) Phase recognition during surgical procedures using embedded and body-worn sensors. IEEE Int Conf Pervasive Comput Commun (PerCom) 2011:45–53. https://doi.org/10.1109/PERCOM.2011.5767594

    Article  Google Scholar 

  10. Thiemjarus S, James A, Yang G (2012) An eye-hand data fusion framework for pervasive sensing of surgical activities. Pattern Recognit 45(8):2855–2867. https://doi.org/10.1016/j.patcog.2012.01.008

    Article  Google Scholar 

  11. Jin Y, Dou Q, Chen H, Yu L, Qin J, Fu CW, Heng PA (2018) SV-RCNet: Workflow recognition from surgical videos using recurrent convolutional network. IEEE Trans Med Imag 37(5):1114–1126. https://doi.org/10.1109/TMI.2017.2787657

    Article  Google Scholar 

  12. Yu F, Silva Croso GS, Kim TS, Song Z, Parker F, Hager GD, Reiter A, Vedula SS, Ali H, Sikder S (2019) Assessment of automated identification of phases in videos of cataract surgery using machine learning and deep learning techniques. JAMA Netw Open 2(4):e191860. https://doi.org/10.1001/jamanetworkopen.2019.1860

    Article  Google Scholar 

  13. Jin Y, Li H, Dou Q, Chen H, Qin J, Fu CW, Heng PA (2020) Multi-task recurrent convolutional network with correlation loss for surgical video analysis. Med Image Anal 59(9):101572. https://doi.org/10.1016/j.media.2019.101572

    Article  Google Scholar 

  14. Mondal SS, Sathish R, Sheet D (2018) Multitask learning of temporal connectionism in convolutional networks using a joint distribution loss function to simultaneously identify tools and phase in surgical videos. 5th MedImage Workshop of 11th Indian Conference on Computer Vision, Graphics and Image Processing, India. https://doi.org/10.48550/arXiv.1905.08315

  15. Sarikaya D, Guru KA, Corso JJ (2018) Joint surgical gesture and task classification with multi-task and multimodal learning. Comput Science Arxiv. https://doi.org/10.48550/arXiv.1805.00721

  16. Liu P, Qiu X, Huang X (2017) Adversarial multi-task learning for text classification. Proceedings of the 55th annual meeting of the Association for Computational Linguistics. Canadian. https://doi.org/10.18653/v1/P17-1001

  17. Zhu G, Zhang L, Shen P, Song J (2017) Multimodal gesture recognition using 3-D convolution and convolutional LSTM. IEEE Access 5:4517–4524. https://doi.org/10.1109/ACCESS.2017.2684186

    Article  Google Scholar 

  18. Ebata R, Nakamura R, Kawamura K, Otori N (2021) Automatic situation recognition using endoscopic images and navigation information in endoscopic sinus surgery. J Jpn Soc Comput Aid Surg 23(2):48–55. https://doi.org/10.5759/jscas.23.48

    Article  Google Scholar 

  19. Zia A, Hung A, Essa I, Jarc A (2018) Surgical activity recognition in robot-assisted radical prostatectomy using deep learning. Medical image computing and computer assisted intervention (MICCAI 2018). Lect Notes Comput Sci 11073:273–280. https://doi.org/10.1007/978-3-030-00937-3_32

    Article  Google Scholar 

  20. Zia A, Guo L, Zhou L, Essa I, Jarc A (2019) Novel evaluation of surgical activity recognition models using task-based efficiency metrics. Int J Comput Assist Radiol Surg 14(12):2155–2163. https://doi.org/10.1007/s11548-019-02025-w

    Article  Google Scholar 

  21. Klank U, Padoy N, Feussner H, Navab N (2008) Automatic feature generation in endoscopic images. Int J Comput Assist Radiol Surg 3:331–339. https://doi.org/10.1007/s11548-008-0223-8

    Article  Google Scholar 

Download references

Acknowledgements

This study received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors. This study was supported through management expenses grants from Chiba University.

Author information

Authors and Affiliations

  1. Center for Frontier Medical Engineering, Chiba University, Chiba, Japan

    Kazuya Kawamura & Ryoichi Nakamura

  2. Graduate School of Science and Engineering, Chiba University, Chiba, Japan

    Ryu Ebata

  3. Institute of Biomaterials and Bioengineering, Tokyo Medical and Dental University, Tokyo, Japan

    Ryoichi Nakamura

  4. Department of Otorhinolaryngology, The Jikei University School of Medicine, Tokyo, Japan

    Nobuyoshi Otori

Authors
  1. Kazuya Kawamura
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Ryu Ebata
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Ryoichi Nakamura
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Nobuyoshi Otori
    View author publications

    You can also search for this author inPubMed Google Scholar

Contributions

KK contributed to the analysis and interpretation of experimental data and wrote the initial draft of the manuscript. RE, RN, and NO designed the study, contributed to the data collection and interpretation, and critically reviewed the manuscript. All authors approved the final version of the manuscript and agreed to be accountable for all aspects of the work, ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Corresponding author

Correspondence to Kazuya Kawamura.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Consent to participate

Informed consent was obtained from all individual participants included in the study.

Consent to publish

The participant has consented to the submission of the case report to the journal.

Ethical approval

All procedures performed in studies involving human participants were conducted in accordance with the ethical standards of the Institutional Research Committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards. In this study, measurement data from a clinical case of ESS were used. This study was approved by the ethics committee of the Jikei University School of Medicine (27–131(8016)).

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kawamura, K., Ebata, R., Nakamura, R. et al. Improving situation recognition using endoscopic videos and navigation information for endoscopic sinus surgery. Int J CARS 18, 9–16 (2023). https://doi.org/10.1007/s11548-022-02754-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11548-022-02754-5

Keywords