Skip to main content
SpringerLink
Log in

Patch-based classification of gallbladder wall vascularity from laparoscopic images using deep learning

  • Original Article
  • Published:
International Journal of Computer Assisted Radiology and Surgery Aims and scope Submit manuscript

Abstract

Purpose

In this study, we propose a deep learning approach for assessment of gallbladder (GB) wall vascularity from images of laparoscopic cholecystectomy (LC). Difficulty in the visualization of GB wall vessels may be the result of fatty infiltration or increased thickening of the GB wall, potentially as a result of cholecystitis or other diseases.

Methods

The dataset included 800 patches and 181 region outlines of the GB wall extracted from 53 operations of the Cholec80 video collection. The GB regions and patches were annotated by two expert surgeons using two labeling schemes: 3 classes (low, medium and high vascularity) and 2 classes (low vs. high). Two convolutional neural network (CNN) architectures were investigated. Preprocessing (vessel enhancement) and post-processing (late fusion of CNN output) techniques were applied.

Results

The best model yielded accuracy 94.48% and 83.77% for patch classification into 2 and 3 classes, respectively. For the GB wall regions, the best model yielded accuracy 91.16% (2 classes) and 80.66% (3 classes). The inter-observer agreement was 91.71% (2 classes) and 78.45% (3 classes). Late fusion analysis allowed the computation of spatial probability maps, which provided a visual representation of the probability for each vascularity class across the GB wall region.

Conclusions

This study is the first significant step forward to assess the vascularity of the GB wall from intraoperative images based on computer vision and deep learning techniques. The classification performance of the CNNs was comparable to the agreement of two expert surgeons. The approach may be used for various applications such as for classification of LC operations and context-aware assistance in surgical education and practice.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Notes

  1. https://mpl-en.med.uoa.gr/as/datasets.

References

  1. Zeng C, Xing W, Wu Z, Huang H, Huang W (2016) A combination of three-dimensional printing and computer-assisted virtual surgical procedure for preoperative planning of acetabular fracture reduction. Injury 47:2223–2227

    Article  Google Scholar 

  2. Loukas C, Nikiteas N, Kanakis M, Georgiou E (2011) The contribution of simulation training in enhancing key components of laparoscopic competence. Am Surg 77:708–715

    Article  Google Scholar 

  3. Vedula SS, Ishii M, Hager GD (2017) Objective assessment of surgical technical skill and competency in the operating room. Annu Rev Biomed Eng 19:301–325

    Article  CAS  Google Scholar 

  4. Maier-Hein L, Vedula SS, Speidel S, Navab N, Kikinis R, Park A, Eisenmann M, Feussner H, Forestier G, Giannarou S, Hashizume M, Katic D, Kenngott H, Kranzfelder M, Malpani A, März K, Neumuth T, Padoy N, Pugh C, Schoch N, Stoyanov D, Taylor R, Wagner M, Hager GD, Jannin P (2017) Surgical data science for next-generation interventions. Nat Biomed Eng 1:691–696

    Article  Google Scholar 

  5. Bodenstedt S, Wagner M, Mündermann L, Kenngott H, Müller-Stich B, Breucha M, Mees ST, Weitz J, Speidel S (2019) Prediction of laparoscopic procedure duration using unlabeled, multimodal sensor data. Int J Comput Assist Radiol Surg 14:1089–1095

    Article  Google Scholar 

  6. Loukas C (2018) Video content analysis of surgical procedures. Surg Endosc 32:553–568

    Article  Google Scholar 

  7. Loukas C, Georgiou E (2013) Surgical workflow analysis with Gaussian mixture multivariate autoregressive (GMMAR) models: a simulation study. Comput Aided Surg 18:47–62

    Article  Google Scholar 

  8. Twinanda AP, Yengera G, Mutter D, Marescaux J, Padoy N (2019) RSDNet: learning to predict remaining surgery duration from laparoscopic videos without manual annotations. IEEE Trans Med Imaging 38:1069–1078

    Article  Google Scholar 

  9. Loukas C, Georgiou E (2015) Smoke detection in endoscopic surgery videos: a first step towards retrieval of semantic events. Int J Med Robot Comput Assist Surg 11:80–94

    Article  Google Scholar 

  10. Funke I, Mees ST, Weitz J, Speidel S (2019) Video-based surgical skill assessment using 3D convolutional neural networks. Int J Comput Assist Radiol Surg 14:1217–1225

    Article  Google Scholar 

  11. Loukas C, Varytimidis C, Rapantzikos K, Kanakis MA (2018) Keyframe extraction from laparoscopic videos based on visual saliency detection. Comput Methods Progr Biomed 165:13–23

    Article  Google Scholar 

  12. Lalys F, Bouget D, Riffaud L, Jannin P (2013) Automatic knowledge-based recognition of low-level tasks in ophthalmological procedures. Int J Comput Assist Radiol Surg 8:39–49

    Article  Google Scholar 

  13. Loukas C, Sgouros NP (2020) Multi-instance multi-label learning for surgical image annotation. Int J Med Robot Comput Assist Surg 16:1–12

    Article  Google Scholar 

  14. Zadeh SM, Francois T, Calvet L, Chauvet P, Canis M, Bartoli A, Bourdel N (2020) SurgAI: deep learning for computerized laparoscopic image understanding in gynaecology. Surg Endosc (in press)

  15. Baghdadi A, Hussein AA, Ahmed Y, Cavuoto LA, Guru KA (2019) A computer vision technique for automated assessment of surgical performance using surgeons’ console-feed videos. Int J Comput Assist Radiol Surg 14:697–707

    Article  Google Scholar 

  16. Derathé A, Reche F, Moreau-Gaudry A, Jannin P, Gibaud B, Voros S (2020) Predicting the quality of surgical exposure using spatial and procedural features from laparoscopic videos. Int J Comput Assist Radiol Surg 15:59–67

    Article  Google Scholar 

  17. Garcia-Martinez A, Vicente-Samper JM, Sabater-Navarro JM (2017) Automatic detection of surgical haemorrhage using computer vision. Artif Intell Med 78:55–60

    Article  Google Scholar 

  18. Petscharnig S, Schöffmann K (2018) Binary convolutional neural network features off-the-shelf for image to video linking in endoscopic multimedia databases. Multimed Tools Appl 77:28817–28842

    Article  Google Scholar 

  19. Iwashita Y, Hibi T, Ohyama T, Honda G, Yoshida M, Miura F, Takada T, Han H-S, Hwang T-L, Shinya S, Suzuki K, Umezawa A, Yoon Y-S, Choi I-S, Huang WS-W, Chen K-H, Watanabe M, Abe Y, Misawa T, Nagakawa Y, Yoon D-S, Jang J-Y, Yu HC, Ahn KS, Kim SC, Song IS, Kim JH, Yun SS, Choi SH, Jan Y-Y, Shan Y-S, Ker C-G, Chan D-C, Wu C-C, Lee K-T, Toyota N, Higuchi R, Nakamura Y, Mizuguchi Y, Takeda Y, Ito M, Norimizu S, Yamada S, Matsumura N, Shindoh J, Sunagawa H, Gocho T, Hasegawa H, Rikiyama T, Sata N, Kano N, Kitano S, Tokumura H, Yamashita Y, Watanabe G, Nakagawa K, Kimura T, Yamakawa T, Wakabayashi G, Mori R, Endo I, Miyazaki M, Yamamoto M (2017) An opportunity in difficulty: Japan–Korea–Taiwan expert Delphi consensus on surgical difficulty during laparoscopic cholecystectomy. J Hepatobiliary Pancreat Sci 24:191–198

    Article  Google Scholar 

  20. Hayama S, Ohtaka K, Shoji Y (2016) Risk factors for difficult laparoscopic cholecystectomy in acute cholecystitis. J Soc Laparoendosc Surg 20:1–8

    Article  Google Scholar 

  21. Van Breda Vriesman AC, Engelbrecht MR, Smithuis RHM, Puylaert JBCM (2007) Diffuse gallbladder wall thickening: differential diagnosis. Am J Roentgenol 188:495–501

    Article  Google Scholar 

  22. Loukas C, Schizas D (2020) Assessment of gallbladder wall vascularity from laparoscopic images using deep learning. In: Proceedings of 13th international joint conference biomedical engineering systems and technologies, vol. 2 Bioimaging. Valletta, Malta, pp 28–36

  23. Twinanda AP, Shehata S, Mutter D, Marescaux J, de Mathelin M, Padoy N (2017) EndoNet: A deep architecture for recognition tasks on laparoscopic videos. IEEE Trans Med Imaging 36:86–97

    Article  Google Scholar 

  24. Lux M, Marques O (2013) Visual information retrieval using Java and LIRE. Synth Lect Inf Concepts Retr Serv. https://doi.org/10.2200/S00468ED1V01Y201301ICR025

    Article  Google Scholar 

  25. Vallières M, Freeman CR, Skamene SR, El Naqa I (2015) A radiomics model from joint FDG-PET and MRI texture features for the prediction of lung metastases in soft-tissue sarcomas of the extremities. Phys Med Biol 60:5471–5496

    Article  Google Scholar 

  26. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. https://arxiv.org/abs/1409.1556

  27. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: IEEE conference on computer vision and pattern recognition. IEEE, Las Vegas, USA, pp 770–778

  28. Schols RM, Bouvy ND, van Dam RM, Masclee AAM, Dejong CHC, Stassen LPS (2013) Combined vascular and biliary fluorescence imaging in laparoscopic cholecystectomy. Surg Endosc 27:4511–4517

    Article  Google Scholar 

  29. Uggowitzer M, Kugler C, Schramayer G, Kammerhuber F, Gröll R, Hausegger KA, Ratschek M, Quehenberger F (1997) Sonography of acute cholecystitis: comparison of color and power Doppler sonography in detecting a hypervascularized gallbladder wall. Am J Roentgenol 168:707–712

    Article  CAS  Google Scholar 

  30. Kim J, Candemir S, Chew EY, Thoma GR (2018) Region of interest detection in fundus images using deep learning and blood vessel information. In: International symposium on computer-based medical systems. IEEE, Karlstad, Sweden, pp 357–362

  31. Jin Y, Dou Q, Chen H, Yu L, Qin J, Fu C-W, Heng P-A (2018) SV-RCNet: Workflow recognition from surgical videos using recurrent convolutional network. IEEE Trans Med Imaging 37:1114–1126

    Article  Google Scholar 

Download references

Funding

None.

Author information

Authors and Affiliations

  1. Laboratory of Medical Physics, Medical School, National and Kapodistrian University of Athens, Athens, Greece

    Constantinos Loukas

  2. 1st Propaedeutic Department of Surgery, Hippocration General Hospital, Medical School, National and Kapodistrian University of Athens, Athens, Greece

    Maximos Frountzas

  3. 1st Department of Surgery, Laikon General Hospital, Medical School, National and Kapodistrian University of Athens, Athens, Greece

    Dimitrios Schizas

Authors
  1. Constantinos Loukas
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Maximos Frountzas
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Dimitrios Schizas
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Constantinos Loukas.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical approval

This article does not contain any studies with human participants or animals performed by any of the authors.

Informed consent

For this type of study, formal consent is not required. Used data were anonymously available through the Cholec80 video dataset.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Loukas, C., Frountzas, M. & Schizas, D. Patch-based classification of gallbladder wall vascularity from laparoscopic images using deep learning. Int J CARS 16, 103–113 (2021). https://doi.org/10.1007/s11548-020-02285-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11548-020-02285-x

Keywords

Search

Navigation