Skip to main content

Advertisement

SpringerLink
Log in

Co-occurrence balanced time series classification for the semi-supervised recognition of surgical smoke

  • Original Article
  • Published:
International Journal of Computer Assisted Radiology and Surgery Aims and scope Submit manuscript

Abstract

Purpose

Automatic recognition and removal of smoke in surgical procedures can reduce risks to the patient by supporting the surgeon. Surgical smoke changes its visibility over time, impacting the vision depending on its amount and the volume of the body cavity. While modern deep learning algorithms for computer vision require large amounts of data, annotations for training are scarce. This paper investigates the use of unlabeled training data with a modern time-based deep learning algorithm.

Methods

We propose to improve the state of the art in smoke recognition by enhancing a image classifier based on convolutional neural networks with a recurrent architecture thereby providing temporal context to the algorithm. We enrich the training with unlabeled recordings from similar procedures. The influence of surgical tools on the smoke recognition task is studied to reduce a possible bias.

Results

The evaluations show that smoke recognition benefits from the additional temporal information during training. The use of unlabeled data from the same domain in a semi-supervised training procedure shows additional improvements reaching an accuracy of 86.8%. The proposed balancing policy is shown to have a positive impact on learning the discrimination of co-occurring surgical tools.

Conclusions

This study presents, to the best of our knowledge, the first use of a time series algorithm for the recognition of surgical smoke and the first use of this algorithm in the described semi-supervised setting. We show that the performance improvements with unlabeled data can be enhanced by integrating temporal context. We also show that adaption of the data distribution is beneficial to avoid learning biases.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

References

  1. Bar O, Neimark D, Zohar M, Hager GD, Girshick R, Fried GM, Wolf T, Asselmann D (2020) Impact of data on generalization of ai for surgical intelligence applications. Sci Rep 10(1):1–12

    Article  Google Scholar 

  2. Chen L, Tang W, John NW, Wan TR, Zhang JJ (2020) De-smokegcn: Generative cooperative networks for joint surgical smoke detection and removal. IEEE Trans Med Imag 39(5):1615–1625. https://doi.org/10.1109/TMI.2019.2953717

    Article  Google Scholar 

  3. Czempiel T, Paschali M, Keicher M, Simson W, Feussner H, Kim ST, Navab N (2020) Tecno: Surgical phase recognition with multi-stage temporal convolutional networks. In: Martel AL, Abolmaesumi P, Stoyanov D, Mateus D, Zuluaga MA, Zhou SK, Racoceanu D, Joskowicz L (eds) Medical image computing and computer assisted intervention—MICCAI 2020. Springer International Publishing, Cambridge, pp 343–352

    Chapter  Google Scholar 

  4. Funke I, Jenke A, Mees ST, Weitz J, Speidel S, Bodenstedt S (2018) Temporal Coherence-based Self-supervised Learning for Laparoscopic Workflow Analysis. In: OR 2.0 Context-Aware Operating Theaters, Computer Assisted Robotic Endoscopy, Clinical Image-Based Procedures, and Skin Image Analysis, vol. 11041, pp 85–93, Springer International Publishing, Cambridge

  5. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780

    Article  CAS  Google Scholar 

  6. Huang G, Liu Z, van der Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition

  7. Jin Y, Dou Q, Chen H, Yu L, Qin J, Fu C, Heng P (2018) Sv-rcnet: Workflow recognition from surgical videos using recurrent convolutional network. IEEE Trans Med Imag 37(5):1114–1126. https://doi.org/10.1109/TMI.2017.2787657

    Article  Google Scholar 

  8. Johnson JM, Khoshgoftaar TM (2019) Survey on deep learning with class imbalance. J Big Data 6(1):27

    Article  Google Scholar 

  9. Laine S, Aila T (2017) Temporal ensembling for semi-supervised learning. In: 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24–26, 2017, Conference Track Proceedings. OpenReview.net. https://openreview.net/forum?id=BJ6oOfqge

  10. Leibetseder A, Primus MJ, Petscharnig S, Schoeffmann K (2017) Image-based smoke detection in laparoscopic videos. In: Computer assisted and robotic endoscopy and clinical image-based procedures. Springer, Berlin, pp 70–87

  11. Leibetseder A, Primus MJ, Petscharnig S, Schoeffmann K (2017) Real-time image-based smoke detection in endoscopic videos. Proceedings of the on Thematic Workshops of ACM Multimedia 2017:296–304

    Article  Google Scholar 

  12. Maier-Hein L, Eisenmann M, Sarikaya D, März K, Collins T, Malpani A, Fallert J, Feussner H, Giannarou S, Mascagni P, Nakawala H, Park A, Pugh C, Stoyanov D, Vedula SS, Müller BP, Cleary K, Fichtinger G, Forestier G, Gibaud B, Grantcharov T, Hashizume M, Kenngott H, Kikinis R, Mündermann L, Navab N, Onogur S, Sznitman R, Taylor R, Dietlinde Tizabi M, Wagner M, Hager GD, Neumuth T, Padoy N, Jannin P, Speidel S (2020) Surgical Data Science—from Concepts to Clinical Translation. arXiv e-prints arXiv:2011.02284

  13. Reiter W (2020) Improving endoscopic smoke detection with semi-supervised noisy student models. Curr Direct Biomed Eng 6(1):26

    Article  Google Scholar 

  14. Ross T, Zimmerer D, Vemuri A, Isensee F, Wiesenfarth M, Bodenstedt S, Both F, Kessler P, Wagner M, Müller B, Kenngott H, Speidel S, Kopp-Schneider A, Maier-Hein K, Maier-Hein L (2018) Exploiting the potential of unlabeled endoscopic video data with self-supervised learning. Int J CARS 13(6):925–933. https://doi.org/10.1007/s11548-018-1772-0

    Article  Google Scholar 

  15. Takahashi H, Yamasaki M, Hirota M, Miyazaki Y, Moon JH, Souma Y, Mori M, Doki Y, Nakajima K (2013) Automatic smoke evacuation in laparoscopic surgery: a simplified method for objective evaluation. Surg Endosc 27(8):2980–2987

    Article  Google Scholar 

  16. Tarvainen A, Valpola H (2017) Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In: I. Guyon, U. von Luxburg, S. Bengio, H.M. Wallach, R. Fergus, S.V.N. Vishwanathan, R. Garnett (eds.) Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4–9, 2017, Long Beach, CA, USA, pp. 1195–1204. https://proceedings.neurips.cc/paper/2017/hash/68053af2923e00204c3ca7c6a3150cf7-Abstract.html

  17. Twinanda A, Mutter D, Marescaux J, Mathelin M, Padoy N (2016) Single and multi-task architectures for surgical workflow challenge. In: Proceedings of workshop and challenges on modeling and monitoring of computer assisted interventions (M2CAI) at Medical Image Computing and Computer-Assisted Intervention (MICCAI)

  18. Twinanda AP, Shehata S, Mutter D, Marescaux J, de Mathelin M, Padoy N (2017) EndoNet: a deep architecture for recognition tasks on laparoscopic videos. IEEE Trans Med Imag 36(1):86–97

    Article  Google Scholar 

  19. Wu Z, Guo Y, Lin W, Yu S, Ji Y (2018) A weighted deep representation learning model for imbalanced fault diagnosis in cyber-physical systems. Sensors 18(4):1096

    Article  Google Scholar 

  20. Xie Q, Luong MT, Hovy E, Le QV (2020) Self-training with noisy student improves imagenet classification. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10687–10698

  21. Yengera G, Mutter D, Marescaux J, Padoy N (2018) Less is more: Surgical phase recognition with less annotations through self-supervised pre-training of cnn-lstm networks. arXiv preprint arXiv:1805.08569

Download references

Author information

Authors and Affiliations

  1. Wintegral GmbH, München, Germany

    Wolfgang Reiter

Authors
  1. Wolfgang Reiter
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Wolfgang Reiter.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest

Ethical approval

For this type of study, formal consent is not required

Informed consent

This article contains patient data from publically available datasets. Informed consent was obtained from patients for non-public recordings.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Reiter, W. Co-occurrence balanced time series classification for the semi-supervised recognition of surgical smoke. Int J CARS 16, 2021–2027 (2021). https://doi.org/10.1007/s11548-021-02411-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11548-021-02411-3

Keywords

Search

Navigation